[Dragora-users] Jul design document

[Lucas Sköldqvist]

Hello everyone,

I've written a design document for jul that I will include below. It is
also available in the fossil repository on dragora.org. We need to
revise the document but I wanted to send out a `draft' so people can
check it out, here it goes:

1 INTRODUCTION
==============

Dragora does have a system for managing packages already (pkgsystem[1]
in Dragora version 2 and qi[2] in version 3). Both of them can build,
install, remove, and upgrade packages, but with limited flexibility;
they only work with local packages on the system. There is no way to
search for available packages, check for upgrades, or fetch packages.

This is where jul comes in. Jul uses the existing package managers and
adds aditional functionality on top. There is a working yet incomplete
version available for use today on Dragora 2.

One can think of pkgsystem/qi as dpkg and jul as apt-get (in Debian)
but with a simpler design.

[1]
http://www.dragora.org/repo.fsl/doc/trunk/www/dragora2/guides/pkgmanager.md
[2] http://www.dragora.org/repo.fsl/dir?name=qi

2 DESIGN
========

Jul is written in Tcl[1] and has a command line interface. The
interface is modeled after Fossil[2] but commands are named similar to
that of pkgsystem and qi. Some examples:

        $ jul search foo

        $ jul help add

        $ jul add -h

[1] https://tcl.tk/
[2] http://fossil-scm.org/

2.1 COMMANDS
============

The following commands are needed:

        1) add/install

        2) list

        3) search

        4) sync/update

        5) upgrade

        6) help

2.1.1 ADD/INSTALL
=================

These commands do the same thing; adds a package to system. Jul will do
a search for the given package(s) and list the matches. One can then
choose from the list of packages which package(s) to install.

In jul 0.5 you can only add one package at a time, and a list of
matches makes sense. But it is not clear how this would work when
multiple packages are specified on the command line.

2.1.2 LIST
==========

This command lists installed packages. It can have options to only list
packages from certain repositories (see section 3) or match against a
search term.

2.1.3 SEARCH
============

`search' takes on or more search-terms and looks for packages. By
default it searches in all enabled repositories but it could take an
option to only search in one or more repositories.

Currently searches in a local copy of the database of available
packages. One might want to do an update simultaneously (see section 4).

2.1.4 SYNC/UPDATE
=================

These commands fetch the latest changes from the repositories.

2.1.5 UPGRADE
=============

`upgrade' compares the installed packages with the ones in the
database, and if they are newer in the database, they can be upgraded.

Some useful options to this commands could be:

        1) --list/-l       only list available upgrade, don't prompt to
        upgrade

        2) --onlysec/-o    only list/upgrade packages that are
        vulnerable

2.1.6 HELP
==========

The `help' command is used to display help screens. It shows the same
help that the `-h' option does, so these command lines are equivalent:

        $ jul help

        $ jul -h

        $ jul help add

        $ jul add -h

3 REPOSITORIES
==============

A repository is a SQLite[1] database with a few tables. Currently (see
section 3.2) it has support for basic package information plus
descriptions.

Packages are added to the database and then the database is downloaded
by jul when doing a sync/update. The copy is stored in $HOME/.jul and
used by most commands.

Up until now I've manually (or via scripts) added package information
to the database. It is a lot of work and we need a better and simpler
way...

[1] https://sqlite.org/

3.1 GENERATING DATABASES
========================

Qi introduces `.meta' files, which contain most of the information
needed for an entry in the database. A meta file could look something
like this:

        name="foo"
        version="1.3"
        desc="Simple program"
        license="GPLv3+"
        order="bar zab foo"

If each package has one of these files a database can easily be
generated by parsing these files.

3.2 SQLITE SCHEMA
=================

CREATE TABLE repository (
    name text,
    url text,
    PRIMARY KEY(name)
);
CREATE TABLE description (
    name text,
    lang text,
    desc text,
    PRIMARY KEY(name)
);
CREATE TABLE location (
    name text,
    url text,
    PRIMARY KEY(name)
);
CREATE TABLE package (
    repo text,
    name text,
    version text,
    arch text,
    build integer,
    FOREIGN KEY(repo) REFERENCES repository,
    FOREIGN KEY(name) REFERENCES description
);

3.3 HOSTING AND SYNCHRONISING
=============================

One hosts a database simply by serving it via HTTP. The database on
gungre.ch is available at: https://gungre.ch/jul/gungre.db

(the databases contain one or more repositories)

Curl[1] is used for fetching so other protocols are supported as well.

Jul supports multiple databases so you could point at dragora.org for
the default repository, gungre.ch for community repositories, and so on.

It would be better if each host could synchronise the other hosts
databases and packages so if one goes down there are other mirrors
available.

So each host has a directory tree with packages. The host then runs a
script (see section 3.4) each time packages are added (or using a cron
job, or something) to generate the database.

The hosts will then, maybe twice a day, or every hour, use rsync[2] to
fetch the new version of the database and the new packages.

[1] https://curl.haxx.se/
[2] https://rsync.samba.org/

3.4 VERSION COMPARISON
======================

The script that generates the databases should only add the latest
packages to the database. To achive this the different versions of a
package need to be compared.

To compare versions we need a specification how to version packages. For
example the author of a package might use the version 1.2-alpha1 and
another author might use the version 2.0a. We need to decide which
style to use, to be able to compare them.

All package managers that does version comparison does this, Gentoo's
portage is not an exception[1].

Besides the specification, some algorithms are needed. I'm currently
translating the algorithms from portage[2].

When the code is written it will be used for jul's `upgrade' command as
well.

[1] https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-300003.2
[2] https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-310003.3

4 CONFIGURATION FILES
=====================

The configuration file is just a Tcl script where you can set some
variables.

There is a system wide configuration file in `/etc' and a local
configuration file in the users home directory. Like most programs.

Examples of settings:

        1) which packages to show (archs)

        2) which repositories to use

        3) always run the sync/update command before most of the other
           commands

Most, if not all, of the settings should be modified via the command
line as options.

5 DISCUSSION
============

A limited version of jul is available[1,2] today, and a few people in
the community including me use it a lot even if the most exiting
features aren't complete or even missing.

The most pressing thing at the moment is the code for version
comparison. I'm working on it but we need a version specification for
it to work. I'm using portage's version spec. temporarily.

This document was written in somewhat of a hurry and I'm sure things
were left out.

Please discuss on the mailing list and IRC so we have something ready
for the long awaited Dragora 3. :-)

[1] https://gungre.ch/dragora/repo/frusen/stable/jul/
[2] http://dragora.org/repo.fsl/dir?name=jul