Re: [Dragora-users] Distribution of ARM rootfs tarballs

[Matias Fonzo]

El 2020-01-31 09:38, Kevin "The Nuclear" Bloom escribió:
> Matias Fonzo writes:
>
> > El 2020-01-31 00:06, Kevin "The Nuclear" Bloom escribió:
> > > Thanks for the quick reply, Matias. See my comments below:
> > >
> > > > El 2020-01-29 16:50, Kevin "The Nuclear" Bloom escribió:
> > > > > Hi,
> > > >
> > > > Hello Kevin.  :-)
> > > >
> > > > > Those of us who have a C201 know that installation on this device 
> > > > > is
> > > > > quite nontraditional. Instead of booting off of a USB stick and 
> > > > > running
> > > > > an installer, one must do it manually by loading an sd card (or usb
> > > > > stick) with a special kernel partition and a special root
> > > > > partition. What this means is that creating an ISO for this machine 
> > > > > is
> > > > > pointless. Due to that, most distros that support the machine have 
> > > > > a
> > > > > rootfs tarball that you unpack into the root partition and, 
> > > > > normally,
> > > > > inside of /boot there is a linux.kpart or something that gets 
> > > > > written to
> > > > > the kernel partition using `dd`.
> > > >
> > > > Okay.  Question: what format would be appropriate for create the 
> > > > rootfs?.
> > >
> > > Arch-arm uses tar.gz and we probably should stick to that because 
> > > some
> > > people might be unpacking it from ChromeOS which doesn't come with 
> > > lzip
> > > installed. It can, however, unpack gzip.

Okay, tar+gzip then.

> > > > > That being said, I'm curious as to how we wish to handle the
> > > > > distribution of Dragora 3 rootfs tarballs for this machine. Most
> > > > > distros' tarball is quite small and only contains the core system 
> > > > > with
> > > > > simple network tools such as wpa-supplicant for connecting the 
> > > > > machine
> > > > > to the internet (there is no Ethernet port, so wpa will be
> > > > > required). Once the core system is booted the user is expected to
> > > > > install the rest of the system via their package manager. Since 
> > > > > Dragora
> > > > > doesn't have a package repo that contains precompiled binaries 
> > > > > (that I'm
> > > > > aware of), I'm not sure how we want to do this.
> > > >
> > > > Here we could say that Dragora's "kernel" includes everything needed 
> > > > to boot
> > > > the
> > > > system, as well as the network part, including the wpa_supplicant 
> > > > currently.
> > > > As
> > > > for the packages, we can say that the official packages are provided 
> > > > and
> > > > distributed after each release[1].  In this sense, it is not a high 
> > > > priority
> > > > (for me) to provide updates to pre-compiled packages like any other
> > > > pre-compiled
> > > > package, since the distribution has to be finished, or at least 
> > > > until it
> > > > reaches
> > > > the stable one.
> > > >
> > > > [1] http://rsync.dragora.org/v3/packages/
> > >
> > > I think that is a good idea. Would take the stress away from trying 
> > > to
> > > keep every package up-to-date all the time. I'm still curious about 
> > > how
> > > we should manage downloading the binaries and then installing them in
> > > the correct order. Any ideas how to do this? (i.e. `wget -i
> > > BINARY-LIST.txt | qi -i` or something)
> >
> > Qi can read from standard input, for example if the file currently 
> > contains the
> > full (local) path of one or more packages, it can install them, e.g: 
> > qi -i - <
> > pkglist.txt
> >
> > What you want is to read, download and install.  Currently Qi has the 
> > code to
> > download and generate the .sha256 on the source side.  As a pending 
> > issue, we
> > could use or adapt this code (as it declares the General Network 
> > Downloader) to
> > tell Qi to download the packages when using the -i option and if 
> > "http(s)://" is
> > specified on the command line.
> >
> > Of course, this has to be studied to make it as reliable as possible 
> > (.sha256,
> > signatures...).
>
> That would be quite handy! If this would be valuable to the other D3
> archs then I think it would be great addition, otherwise, we may want 
> to
> just have a shell script that does this using wget+qi or
> something.

In theory this was going to be part of 'jul', unfortunately Jul doesn't 
continue[1].  I prefer a separate program to do the whole remote package 
thing...

[1] http://git.savannah.nongnu.org/cgit/dragora.git/tree/testing/jul

> > > > > My idea is this: we do the same thing that other distros do, for 
> > > > > the
> > > > > most part. Keep the tarball small and use just the core system with 
> > > > > some
> > > > > networking programs. The kernel will be in /boot under a name like
> > > > > kernel.kpart or something. Inside of the root home directory there 
> > > > > will
> > > > > be a few different text files that contain urls to pre-compiled 
> > > > > binary
> > > > > packages. Each file will have names that match up with the .order 
> > > > > files
> > > > > when building D3: editors.txt, sound.txt, xorg.txt, etc. They will 
> > > > > have
> > > > > all the programs in the orders that they need to be in to insure a 
> > > > > safe
> > > > > installation. Then, the user uses a few commands to download and 
> > > > > install
> > > > > each package (probably something with wget that passes the binary 
> > > > > into a
> > > > > qi command). Once they've installed all the stuff they need, 
> > > > > they'll be
> > > > > good to go!
> > > >
> > > > What I see here is that it is possible that the kernel configuration 
> > > > needs to
> > > > be
> > > > adjusted[2], in addition to testing it (very important), I do not 
> > > > own such a
> > > > computer, and if I did, I would not have enough time now to focus 
> > > > exclusively
> > > > on
> > > > this, considering all that needs to be done. I keep thinking about 
> > > > how these
> > > > lists will facilitate the installation of the packages (how to 
> > > > produce them
> > > > from
> > > > Qi), for the moment you can compile the core[3] and produce the 
> > > > rootfs, then
> > > > compile the rest to get the packages...
> > > >
> > > > [2]
> > > > http://git.savannah.nongnu.org/cgit/dragora.git/plain/archive/kernel/config-c201-v7
> > > > [3]
> > > > http://git.savannah.nongnu.org/cgit/dragora.git/plain/recipes/00-core.order
> > >
> > > Yes, I just completed the core build with the current master
> > > branch. Everything went smoothly except for meson, which has always 
> > > been
> > > a problem child on the C201. I will be creating the signed kernel and
> > > attempting booting tomorrow, if time permits.
> > >
> > > > > Let me know if this is a good idea or if it need tweaked at all! 
> > > > > This is
> > > > > quite a lot of work for only 1 machine but it's the only way I can 
> > > > > think
> > > > > of other than just having all that stuff in the tarball but that 
> > > > > would
> > > > > make it very large.
> > > >
> > > > I will try to assist you and provide you with what you need.
> >
> > What I can think of is that we can create a new scenario for the 
> > bootstrapping
> > process.  This would be a minimal system to boot and log in to, from 
> > there you
> > could install whatever you want, reusing the minimal system tools.   
> > This will
> > allow you:
> >
> > - Check and test the kernel configuration.
> > - Save time instead of building the stage1, the whole core, etc.
> > - Accessible via enter-chroot.
> > - Have the rootfs small.
> > - Ready to boot.
> >
> > For example, you would set the cross compiler in motion:
> >
> > ./boostrap -s0 -a armv7_hf
> >
> > Then you would produce the minimum system using the cross compiler for 
> > your
> > machine:
> >
> > ./bootstrap -s201 -a armv7_hf
> >
> > If you already have the cross-compiler in place, you would use the 
> > "201"
> > scenario/stage as many times as necessary (related changes, kernel
> > configuration, busybox, settings, etc.)
> >
> > In time, the new produced rootfs will be adjusted to what is "just and
> > necessary".
> >
> > ... Cum on feel the noize! ;-)
>
> This would be great! Would it be possible to just have another special
> .order file called `base.order` or `minimal.order`? Which would just
> build the essentials and some network stuff. Using the bootstrap 
> command
> you mention would work too!

What I can think of is that we can simplify the structure of the Dragora 
series.  For example, we currently have the output sorted for packages 
as (default):

    /var/cache/qi/packages/<arch>/<series_name>

Instead of having <series_name>, we can try to have "hashtags" for the 
packages, so we would categorize in the recipes "essential", 
"networking", etc.

This will be reflected in the package name and the destination directory 
($destdir), for example:

    Package name: util-linux-2.34-x86_64+1#essential.tlz
    It will be installed as: util-linux-2.34-x86_64+1#essential

This would make it easier to find the essential packages and other 
series to be installed or removed.

The "essential" label would deal with the minimum or essential packages 
to run the system.

What do you think?.