Re: [Dragora-users] Distribution of ARM rootfs tarballs

[Matias Fonzo]

El 2020-02-07 09:40, Kevin "The Nuclear" Bloom escribió:
> Matias Fonzo writes:
>
> > El 2020-01-31 09:38, Kevin "The Nuclear" Bloom escribió:
> > > Matias Fonzo writes:
> > >
> > > > El 2020-01-31 00:06, Kevin "The Nuclear" Bloom escribió:
> > > > > Thanks for the quick reply, Matias. See my comments below:
> > > > >
> > > > > > El 2020-01-29 16:50, Kevin "The Nuclear" Bloom escribió:
> > > > > > > Hi,
> > > > > >
> > > > > > Hello Kevin.  :-)
> > > > > >
> > > > > > > Those of us who have a C201 know that installation on this device 
> > > > > > > is
> > > > > > > quite nontraditional. Instead of booting off of a USB stick and 
> > > > > > > running
> > > > > > > an installer, one must do it manually by loading an sd card (or 
> > > > > > > usb
> > > > > > > stick) with a special kernel partition and a special root
> > > > > > > partition. What this means is that creating an ISO for this 
> > > > > > > machine is
> > > > > > > pointless. Due to that, most distros that support the machine 
> > > > > > > have a
> > > > > > > rootfs tarball that you unpack into the root partition and, 
> > > > > > > normally,
> > > > > > > inside of /boot there is a linux.kpart or something that gets 
> > > > > > > written to
> > > > > > > the kernel partition using `dd`.
> > > > > >
> > > > > > Okay.  Question: what format would be appropriate for create the 
> > > > > > rootfs?.
> > > > >
> > > > > Arch-arm uses tar.gz and we probably should stick to that because 
> > > > > some
> > > > > people might be unpacking it from ChromeOS which doesn't come with 
> > > > > lzip
> > > > > installed. It can, however, unpack gzip.
> >
> > Okay, tar+gzip then.
> >
> > > > > > > That being said, I'm curious as to how we wish to handle the
> > > > > > > distribution of Dragora 3 rootfs tarballs for this machine. Most
> > > > > > > distros' tarball is quite small and only contains the core system 
> > > > > > > with
> > > > > > > simple network tools such as wpa-supplicant for connecting the 
> > > > > > > machine
> > > > > > > to the internet (there is no Ethernet port, so wpa will be
> > > > > > > required). Once the core system is booted the user is expected to
> > > > > > > install the rest of the system via their package manager. Since 
> > > > > > > Dragora
> > > > > > > doesn't have a package repo that contains precompiled binaries 
> > > > > > > (that I'm
> > > > > > > aware of), I'm not sure how we want to do this.
> > > > > >
> > > > > > Here we could say that Dragora's "kernel" includes everything 
> > > > > > needed to
> > > > > > boot
> > > > > > the
> > > > > > system, as well as the network part, including the wpa_supplicant
> > > > > > currently.
> > > > > > As
> > > > > > for the packages, we can say that the official packages are 
> > > > > > provided and
> > > > > > distributed after each release[1].  In this sense, it is not a 
> > > > > > high
> > > > > > priority
> > > > > > (for me) to provide updates to pre-compiled packages like any 
> > > > > > other
> > > > > > pre-compiled
> > > > > > package, since the distribution has to be finished, or at least 
> > > > > > until it
> > > > > > reaches
> > > > > > the stable one.
> > > > > >
> > > > > > [1] http://rsync.dragora.org/v3/packages/
> > > > >
> > > > > I think that is a good idea. Would take the stress away from trying 
> > > > > to
> > > > > keep every package up-to-date all the time. I'm still curious about 
> > > > > how
> > > > > we should manage downloading the binaries and then installing them 
> > > > > in
> > > > > the correct order. Any ideas how to do this? (i.e. `wget -i
> > > > > BINARY-LIST.txt | qi -i` or something)
> > > >
> > > > Qi can read from standard input, for example if the file currently 
> > > > contains
> > > > the
> > > > full (local) path of one or more packages, it can install them, e.g: 
> > > > qi -i -
> > > > <
> > > > pkglist.txt
> > > >
> > > > What you want is to read, download and install.  Currently Qi has 
> > > > the code to
> > > > download and generate the .sha256 on the source side.  As a pending 
> > > > issue, we
> > > > could use or adapt this code (as it declares the General Network 
> > > > Downloader)
> > > > to
> > > > tell Qi to download the packages when using the -i option and if 
> > > > "http(s)://"
> > > > is
> > > > specified on the command line.
> > > >
> > > > Of course, this has to be studied to make it as reliable as possible
> > > > (.sha256,
> > > > signatures...).
> > >
> > > That would be quite handy! If this would be valuable to the other D3
> > > archs then I think it would be great addition, otherwise, we may want 
> > > to
> > > just have a shell script that does this using wget+qi or
> > > something.
> >
> > In theory this was going to be part of 'jul', unfortunately Jul 
> > doesn't
> > continue[1].  I prefer a separate program to do the whole remote 
> > package
> > thing...
> >
> > [1] http://git.savannah.nongnu.org/cgit/dragora.git/tree/testing/jul
>
> Yes, I agree that it should be separate. I created an emacs package
> called `jul-mode` back when jul was still around - it only requires jul
> for the actual downloading/installing part and parses the HTML to see
> what packages were available. I'm sure I could write another program
> that could do the same sort of thing but use `qi` to install. We could
> come up with a better way to check for packages rather than parsing 
> HTML
> (which could be slow if we had a lot of stuff), maybe an sqlite db or
> something. If there is a faster way that you know of, let me know. I
> would probably write it in guile scheme since I don't know anything
> about tcl.
>
> Ideas?

If the database is binary, we are lost, this breaks with a principle of 
software development in Unix-like systems.  Specifically what would you 
use the database for, is the sqlite database compatible with other 
system tools?.

> > > > > > > My idea is this: we do the same thing that other distros do, for 
> > > > > > > the
> > > > > > > most part. Keep the tarball small and use just the core system 
> > > > > > > with some
> > > > > > > networking programs. The kernel will be in /boot under a name 
> > > > > > > like
> > > > > > > kernel.kpart or something. Inside of the root home directory 
> > > > > > > there will
> > > > > > > be a few different text files that contain urls to pre-compiled 
> > > > > > > binary
> > > > > > > packages. Each file will have names that match up with the .order 
> > > > > > > files
> > > > > > > when building D3: editors.txt, sound.txt, xorg.txt, etc. They 
> > > > > > > will have
> > > > > > > all the programs in the orders that they need to be in to insure 
> > > > > > > a safe
> > > > > > > installation. Then, the user uses a few commands to download and 
> > > > > > > install
> > > > > > > each package (probably something with wget that passes the binary 
> > > > > > > into a
> > > > > > > qi command). Once they've installed all the stuff they need, 
> > > > > > > they'll be
> > > > > > > good to go!
> > > > > >
> > > > > > What I see here is that it is possible that the kernel 
> > > > > > configuration needs
> > > > > > to
> > > > > > be
> > > > > > adjusted[2], in addition to testing it (very important), I do not 
> > > > > > own such
> > > > > > a
> > > > > > computer, and if I did, I would not have enough time now to focus
> > > > > > exclusively
> > > > > > on
> > > > > > this, considering all that needs to be done. I keep thinking about 
> > > > > > how
> > > > > > these
> > > > > > lists will facilitate the installation of the packages (how to 
> > > > > > produce them
> > > > > > from
> > > > > > Qi), for the moment you can compile the core[3] and produce the 
> > > > > > rootfs,
> > > > > > then
> > > > > > compile the rest to get the packages...
> > > > > >
> > > > > > [2]
> > > > > > http://git.savannah.nongnu.org/cgit/dragora.git/plain/archive/kernel/config-c201-v7
> > > > > > [3]
> > > > > > http://git.savannah.nongnu.org/cgit/dragora.git/plain/recipes/00-core.order
> > > > >
> > > > > Yes, I just completed the core build with the current master
> > > > > branch. Everything went smoothly except for meson, which has always 
> > > > > been
> > > > > a problem child on the C201. I will be creating the signed kernel 
> > > > > and
> > > > > attempting booting tomorrow, if time permits.
> > > > >
> > > > > > > Let me know if this is a good idea or if it need tweaked at all! 
> > > > > > > This is
> > > > > > > quite a lot of work for only 1 machine but it's the only way I 
> > > > > > > can think
> > > > > > > of other than just having all that stuff in the tarball but that 
> > > > > > > would
> > > > > > > make it very large.
> > > > > >
> > > > > > I will try to assist you and provide you with what you need.
> > > >
> > > > What I can think of is that we can create a new scenario for the
> > > > bootstrapping
> > > > process.  This would be a minimal system to boot and log in to, from 
> > > > there
> > > > you
> > > > could install whatever you want, reusing the minimal system tools.   
> > > > This
> > > > will
> > > > allow you:
> > > >
> > > > - Check and test the kernel configuration.
> > > > - Save time instead of building the stage1, the whole core, etc.
> > > > - Accessible via enter-chroot.
> > > > - Have the rootfs small.
> > > > - Ready to boot.
> > > >
> > > > For example, you would set the cross compiler in motion:
> > > >
> > > > ./boostrap -s0 -a armv7_hf
> > > >
> > > > Then you would produce the minimum system using the cross compiler 
> > > > for your
> > > > machine:
> > > >
> > > > ./bootstrap -s201 -a armv7_hf
> > > >
> > > > If you already have the cross-compiler in place, you would use the 
> > > > "201"
> > > > scenario/stage as many times as necessary (related changes, kernel
> > > > configuration, busybox, settings, etc.)
> > > >
> > > > In time, the new produced rootfs will be adjusted to what is "just 
> > > > and
> > > > necessary".
> > > >
> > > > ... Cum on feel the noize! ;-)
> > >
> > > This would be great! Would it be possible to just have another 
> > > special
> > > .order file called `base.order` or `minimal.order`? Which would just
> > > build the essentials and some network stuff. Using the bootstrap 
> > > command
> > > you mention would work too!
> >
> > What I can think of is that we can simplify the structure of the 
> > Dragora series.
> > For example, we currently have the output sorted for packages as 
> > (default):
> >
> >     /var/cache/qi/packages/<arch>/<series_name>
> >
> > Instead of having <series_name>, we can try to have "hashtags" for the 
> > packages,
> > so we would categorize in the recipes "essential", "networking", etc.
> >
> > This will be reflected in the package name and the destination 
> > directory
> > ($destdir), for example:
> >
> >     Package name: util-linux-2.34-x86_64+1#essential.tlz
> >     It will be installed as: util-linux-2.34-x86_64+1#essential
> >
> > This would make it easier to find the essential packages and other 
> > series to be
> > installed or removed.
> >
> > The "essential" label would deal with the minimum or essential 
> > packages to run
> > the system.
> >
> > What do you think?.
>
> Hmm, how would it work if there was a lib that was required by another
> program in another category? For example, you wish to build emacs and 
> it
> requires XML support. `libxml` would probably be under a `#lib` or 
> maybe
> `#networking` but emacs would be in `#editors` or something. How would
> we know that you need libxml before emacs?

The package "hash tag" should not be used to determine dependencies, 
instead it can be a complement to the package name, version, 
architecture.

> (disclaimer: emacs doesn't _require_ xml, it is optional. Just for
> argument's sake.)

Metadata must be generated, the metadata that Qi generates for the 
package is in text format.  As you can see here:

http://rsync.dragora.org/v3/packages/x86_64/compressors/xz-5.2.4-x86_64+2.tlz.txt

Some dependencies are required before building a package, some are 
optional, and some are required at runtime.  The simplest thing is to 
list what the package requires to work, that is, what it depends on, 
commonly all dependencies are related to dynamic linking.  The simplest 
thing would be to list these requirements (for example, in the metadata 
file) and have the new program analyze them, I guess.

What I saw years ago that some distributions based on Slackware do, is 
that they add or list the packages that the package requires to work, 
then the program in charge of verifying this, checks if it is installed 
or if it doesn't try to download it, install it.