Re: Building a Docker image for GitLab-CI

[Simon Josefsson]

All,

I am trying to get a Guix container usable in GitLab, and thought I'd
share my status.  I have established working networking in the resulting
Guix container, which seems like progress (whoohoo!).  tl;dr:

https://gitlab.com/debdistutils/guix/container/-/jobs/8652014833

The problem seems to be that GitLab (I suppose they use docker?) picks
the wrong container file system from the resulting image.  It doesn't
use the same file system as a local 'podman run', see below.  Could this
be a 'guix pack' OCI container bug/problem?  Maybe 'guix pack' produces
too many blobs and GitLab/docker stops looking after while?  Or the
container configuration is incorrect?  Some bad interaction with 'podman
load -i'?  My OCI-fu is too weak here, help appreciated.

Here is my work:

https://gitlab.com/debdistutils/guix/container/
https://gitlab.com/debdistutils/guix/container/-/blob/main/.gitlab-ci.yml

(The system-* jobs are inspired by Cayetano's work, they aren't
necessary unless there is something wrong with the 'guix pack' approach
below...)

The idea is:

1) First job 'debian-with-guix' creates a Debian trixie container with
Guix and podman, and a 'guix pull'.  This is for caching, it takes 25
minutes to run on a large compute node, using a GitLab-local mirror of
Guix because Savannah access makes it even slower.

https://gitlab.com/debdistutils/guix/container/-/blob/main/debian-with-guix/Containerfile?ref_type=heads

https://gitlab.com/debdistutils/guix/container/-/jobs/8649009536

Output container (quite useful on its own):

registry.gitlab.com/debdistutils/guix/container:debian-with-guix

2) Second job 'pack' uses the 'debian-with-guix' image, runs a 'guix
pack' and creates a container using 'podman' and uploads it into:

registry.gitlab.com/debdistutils/guix/container:pack

I suspect the problem is happening in the 'guix pack' or 'podman load'
commands here.  Output from last successful run:

https://gitlab.com/debdistutils/guix/container/-/jobs/8649183646

3) Third job 'pack-test' tries to use the pack image in a GitLab job, as
a normal GitLab CI/CD build would work.

Job output is here:

https://gitlab.com/debdistutils/guix/container/-/jobs/8652014833

It fails with networking errors just like Ludo's earlier e-mail:

fping: icmp: unknown protocol

What is really weird is this root directory:

Using docker image 
sha256:57160f1c13ce56799d6e3e83dd97da4c929993ac008404ac38c67317cded25d1 for 
registry.gitlab.com/debdistutils/guix/container:pack with digest 
registry.gitlab.com/debdistutils/guix/container@sha256:be1ad3a7af69669cf3d138c6ec2b1201a64294aad33320246212c6689a1e5c9d
 ...
...
$ ls -la /etc
total 20
drwxr-xr-x 2 0 0 4096 Dec 16 10:15 .
drwxr-xr-x 1 0 0 4096 Dec 16 10:15 ..
-rw-r--r-- 1 0 0   46 Dec 16 10:15 hostname
-rw-r--r-- 1 0 0  283 Dec 16 10:15 hosts
lrwxrwxrwx 1 0 0   12 Dec 16 10:15 mtab -> /proc/mounts
-rw-r--r-- 1 0 0  841 Dec 16 10:15 resolv.conf

There is no /etc/protocols!  No wonder things doesn't work.

However if I run the same container locally, it looks different:

jas@kaka:~/src/guix-container$ podman images|grep pack
registry.gitlab.com/debdistutils/guix/container                   pack          
        57160f1c13ce  54 years ago   1.23 GB
jas@kaka:~/src/guix-container$ podman run --entrypoint /bin/bash -it --rm 
registry.gitlab.com/debdistutils/guix/container:pack
bash-5.1# ls -la /etc
lrwxrwxrwx 1 0 0 55 Jan  1  1970 /etc -> 
/gnu/store/kcpr09dgqnr8q29d167cjk3wsmgizzkb-profile/etc
bash-5.1# ls -la /etc/
total 92
...
lrwxrwxrwx 1 0 0   70 Jan  1  1970 protocols -> 
/gnu/store/bfp25w47fxn8z0fdwj45prx2609sx59j-net-base-5.3/etc/protocols
...
bash-5.1# 

What is going on here?  Seems like GitLab docker is finding another
container file system compared to my local podman?  The hashes are the
same, so I should be working on the same file.

Does anyone have docker installed?  What does the equivalent of this
command do with docker?  Does it work like podman or GitLab?

podman run registry.gitlab.com/debdistutils/guix/container:pack ls -l /etc/

Networking seems to be set up fine, except the missing /etc/protocols:

$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group 
default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: eth0@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state 
UP group default 
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd76:5338:4685:1:0:242:ac11:3/64 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:3/64 scope link tentative 
       valid_lft forever preferred_lft forever
$ ip -4 route
default via 172.17.0.1 dev eth0 
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.3 
$ ip -6 route
fd76:5338:4685:1::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fd76:5338:4685:1::1 dev eth0 metric 1024 pref medium
...
$ fping -c3 -D 130.237.72.201
fping: icmp: unknown protocol

Adding a small /etc/protocols makes networking work:

$ printf 'icmp\t1\tICMP\nipv6-icmp\t58\tIPv6-ICMP\n' > /etc/protocols
$ fping -c3 -D 130.237.72.201
[1734345646.83678] 130.237.72.201 : [0], 64 bytes, 112 ms (112 avg, 0% loss)
[1734345647.83701] 130.237.72.201 : [1], 64 bytes, 111 ms (112 avg, 0% loss)
[1734345648.83694] 130.237.72.201 : [2], 64 bytes, 111 ms (112 avg, 0% loss)
130.237.72.201 : xmt/rcv/%loss = 3/3/0%, min/avg/max = 111/112/112
$ fping -c3 -D gnu.org
[1734345648.90485] gnu.org : [0], 64 bytes, 25.1 ms (25.1 avg, 0% loss)
[1734345649.86358] gnu.org : [1], 64 bytes, 24.2 ms (24.6 avg, 0% loss)
[1734345650.86371] gnu.org : [2], 64 bytes, 24.3 ms (24.5 avg, 0% loss)
gnu.org : xmt/rcv/%loss = 3/3/0%, min/avg/max = 24.2/24.5/25.1

/Simon

signature.asc
Description: PGP signature