hello everyone
nice to have you here at Guix Days 2022
I would like to talk about Python build system
and why we need to modernize it
python-build-system is the Guix component
that turns the source distribution of any Python package out there
into an installable image
so, basically, a directory under the GNU store
let's have a look at tomli a quite simple Python package
which has no external dependencies
so, if we import it using `guix import`, here
add a few imports to the resulting file
and then try to build it
it should work out of the box, right?
The problem is, it does not!
and instead we see at the bottom
an error message saying "no setup.py found"
and, indeed, if we look at the source distribution
there is no `setup.py` to be found anywhere
So, what is going on?
Fortunately, this package is already available in Guix
as `python-tomli`
so we can have a look at its definition
to see how it is currently built
let's just do that
looking at the build system's arguments
we see the phases `build` here and `install` here
which are usually provided by Python build system
replaced with custom code
I'm only showing the interesting parts here
the actual commands are actually much longer
first the build phase
uses a Python module called `build`
to build the wheel, as we can see here
the wheel is basically a distribution format in the Python world
then in the install phase
we simply use a well known tool called Pip
to install the wheel that we just built
into the output which would be somewhere around the GNU store
so how does the build module knows what to do
what to build?
it follows PEP 517
PEPs are kind of the RFCs of the Python world
and this PEP basically splits building wheels into two parts
a frontend and a backend
the frontend is the user facing part
for example the `build` we just saw here
this is the user facing part of the build process
and then a backend
the frontend is supposed to read a file called `pyproject.toml`
this is what we are seeing here
and in that TOML file, a section called `build-system`
this one here
declares which backend will actually build our wheel
and, in this case, another package called `flit_core`
its requirements a build time dependency of tomli
and its module `flit_core.buildapi`
is providing us with the build entrypoint.
The file also contains standardize metadata
and tool related configuration data
which I'm not showing here
A PEP 517* compatible build backend
provides a standard function entrypoint
called `build_wheel`
in the module I just referenced here in the top
and, if we call it, it will just do its magic
and it will produce a wheel file
and its first argument it's the wheel directory
that wheel is basically a zip file
with a predefined structure
that we can extract into our store
and we are almost done
and this is what Pip does in the install phase here
and that's basically the entire build process
as specified by PEP 517
there is no `setup.py` involved any more
we don't have to call it
we don't have to create it as a package provider
so the reason why the error message I showed, showed up earlier
will keep on poping up more and more
is simple: we are late!
we are really really late, actually
because PEP 517 was originally created in 2015
and that it gained provisional acceptance in 2017
and just last year,
after being basically being the *de facto* successor of `setup.py` for some time
it has been finalized and fully accepted
and more importantly, flit which you remember from the previous slide
is also able to create source distributions
and upload them to PyPI
Python public package repository basically
so far, it has been generating a `setup.py`
and does nobody really noticed
but since version 3.5, which was released in November 2021
flit stop doing that by default
and thus we are seeing more and more packages without `setup.py`
in their source distributions
and so we are basically unable to build this projects right now
or this Python modules
a look at the Guix's repository in late January
and back then only 11 packages actually used Pip
or PyPA build as we've seen
but I think our ecosystem is quite old
about half the packages not being the latest versions available upstream
according to `guix refresh`
so it's possible that more packages actually require support for this `pyproject.toml`
and we simply have not updated them yet for whatever reason
maybe because it's too difficult or nobody poped to do it yet
they are also more issues with our current Python build system
for instance `setup.py` test ???? has been deprecated for quite some time
since 2019 actually
and the Python build system's default check phase relies on that
even though it doesn't work for a lot of packages right now
and thus, almost every package in our repository
needs to replace that check phase with a call to Pytest
which is the *de facto* standard right now for executing tests
???? tox does something similar but not quite the same
another long standing issue is wrapping of Python path
or Guix Python path now
because it includes native inputs
and thus propagates way too many packages
and I'm sure we can find more issues with the current Python build system
thus, for almost a year now, I've been working on a modernized Python build system
that addresses some of this issues
unfortunately I haven't had much time lately to improve it for others
I hope to pick it up again
and get it into shape
and get it to master at some point
you can read about it here
in Guix issue #46848
or you can just pull a branch called `wip-python-pep517`
it's also build by the CI by the way
so you don't have to do the whole world rebuild by yourself
and it's using exactly the method I described above
the part written in Guile ends up reading the project's `pyproject.toml`
and then calls the build system's entrypoint `build_wheel` as we can see here
and if there's no `pyproject.toml` it's also not a problem actually
because setuptools, which is still provided by default,
and still included in the build system by default,
because setuptools still provides a PEP 517 build backend
which simply uses the existing `setup.py` behind the curtains
so we don't have to, like provide extra logic for that
they were actually other options for this build phase
in how to do it basically
for example, we could also use the build module
that we have seen above
and some packages use currently
however that raises questions about bootstraping
how do you build the build package if it essentially requires itself
because the build module is part of the Python build system
and of of it's dependency
tomli for example is one of the dependency of build
and what do we do with tomli
like how do we install it without having build
so this is tricky, and I tried it in the beginning
and it works
but we need to do ???? manual copying
and that's not really pretty to be honest
we could also use Pip
which is usually distributed with Python
also our python package bundles Pip and setuptools
this has caused a few issues in the past
because we cannot update either of them
for provide alternative newer versions
without having a full world rebuild basically
and also Pip being a massive project
it bundles quite a few of it's dependencies
and I feel "we" as a Guix project
should try to untangle that mess whenever we can
and in this case, we actually can
so, in conclusion, I think running this 4 lines of Python code
that you can see here
is a much simpler solution
that trying to bootstrap the build module,
tomli and all the modules required to get a Python based builder basically
we can just do it in 4 lines of code
there are a few changes for packagers
which aime to reduce the need for custom phases
first, it is possible to override the build backend detected from the `pyproject.toml`
especially early adopters of PEP 517
use modules which have been renamed in the meantime
I think both Poetry and Flit did that at some point
the split their build logic into a separate project
and renamed the backend
so, for some packages, it's necessary to override this detected build backend
and you can do that simply by specifying the `#:build-backend` option
and also if the detection is wrong at some point
we could override it without causing too much damage
secondly, we have `#:configure-flags`
they had existed before, but where applied only during install
but not during the build phase
which, I believe, is a ????, we'll see
and this are arguments to `setup.py`
but their syntax depends on the build backend unfortunately
sometimes you can pass the argument directly
sometimes you need to pass them as an option to another option
like we can see here
that's the setuptools specific way to do it
so you have an option
and then you have to pass the actual options as an argument to that option
and, in theory, they could also look completly different
like no relationship to any command line flags
it's just legacy code right now
and it's not pretty
and it's one of the things that have been criticized about PEP 517
this configure flags are not really specified how they are supposed to look
but this solution exists, so we have to swim with the flow, kind of
and, finally, they are `#:test-flags`
my proposal was to auto detect test binary based on the package's native inputs
so if pytest is available, it will just use pytest
if nose is available, it will use nose
and so on
now, test flags override the default arguments passed to this test command
and, hopefully, this can reduce the number of custom phases required
to exclude tests mostly
in this case, we are simply don't want to run the `test_legacy_callbacks` function
and so we can just exclude it with the flags
and the package will build just fine, probably, hopefully
so, where do we go from here?
the first step would be to finalize the work on the Python build system
and get the branch ready into a state where it can be merged
unfortunately, right now, quite a few packages fail to build there
because their previously disabled test suite fails
not necessarily disabled, but it was never ran
because the test runner was buggy and the `setup.py test` command simply did nothing
and didn't detect edge cases
we could also go a different route and add a new build system
without touching the old one
and just slowly migrate every Python package we have over
instead of doing the big switch at once
that was also one of the suggestions that was certainly possible
I haven't explored that it would works
as I mentioned earlier, currently the native inputs end up in wrapped binaries
and I think this is one thing we should address while doing the whole world rebuild anyway
but I haven't done any work on that yet, unfortunately
and finally, the pypi import needs support to read `pyproject.toml`
not so much for the runtime dependencies,
this is fine, we have another metadata file which contains the runtime dependencies
but for build time and test dependencies
because they are not included in the wheel file,
there's a file called metadata, and it doesn't have this data
so right now, if you import a `pyproject.toml` based project
you will lack the build dependencies basically
for example flit will not be in the native input right now
and that's it for me for now
I hope to answer some of your questions during the live Q and A
and in the PDF version of this slides, you'll find some clickable links
to different talks and to the standards I have referenced
so, I hope I'll see you soon!