{29}{43}hello everyone
{44}{109}nice to have you here at Guix Days 2022
{110}{161}I would like to talk about Python build system
{162}{202}and why we need to modernize it
{203}{251}python-build-system is the Guix component
{252}{318}that turns the source distribution of any Python package out there
{319}{347}into an installable image
{348}{405}so, basically, a directory under the GNU store
{406}{492}let's have a look at tomli a quite simple Python package
{493}{526}which has no external dependencies
{527}{608}so, if we import it using `guix import`, here
{609}{668}add a few imports to the resulting file
{669}{697}and then try to build it
{698}{753}it should work out of the box, right?
{754}{782}The problem is, it does not!
{783}{840}and instead we see at the bottom
{841}{927}an error message saying "no setup.py found"
{928}{985}and, indeed, if we look at the source distribution
{986}{1043}there is no `setup.py` to be found anywhere
{1044}{1074}So, what is going on?
{1075}{1142}Fortunately, this package is already available in Guix
{1143}{1171}as `python-tomli`
{1172}{1209}so we can have a look at its definition
{1210}{1239}to see how it is currently built
{1240}{1268}let's just do that
{1269}{1327}looking at the build system's arguments
{1328}{1395}we see the phases `build` here and `install` here
{1421}{1486}which are usually provided by Python build system
{1487}{1531}replaced with custom code
{1532}{1582}I'm only showing the interesting parts here
{1583}{1634}the actual commands are actually much longer
{1635}{1681}first the build phase
{1682}{1739}uses a Python module called `build`
{1740}{1801}to build the wheel, as we can see here
{1802}{1926}the wheel is basically a distribution format in the Python world
{1927}{1995}then in the install phase
{1996}{2052}we simply use a well known tool called Pip
{2053}{2116}to install the wheel that we just built
{2117}{2203}into the output which would be somewhere around the GNU store
{2204}{2261}so how does the build module knows what to do
{2262}{2291}what to build?
{2292}{2355}it follows PEP 517
{2356}{2406}PEPs are kind of the RFCs of the Python world
{2407}{2522}and this PEP basically splits building wheels into two parts
{2523}{2551}a frontend and a backend
{2552}{2585}the frontend is the user facing part
{2586}{2638}for example the `build` we just saw here
{2639}{2696}this is the user facing part of the build process
{2697}{2725}and then a backend
{2726}{2870}the frontend is supposed to read a file called `pyproject.toml`
{2871}{2899}this is what we are seeing here
{2900}{2993}and in that TOML file, a section called `build-system`
{2994}{3015}this one here
{3016}{3075}declares which backend will actually build our wheel
{3076}{3160}and, in this case, another package called `flit_core`
{3161}{3247}its requirements a build time dependency of tomli
{3248}{3305}and its module `flit_core.buildapi`
{3306}{3361}is providing us with the build entrypoints.
{3362}{3421}The file also contains standardize metadata
{3422}{3460}and tool related configuration data
{3461}{3489}which I'm not showing here
{3490}{3595}A PEP 517* compatible build backend
{3596}{3653}provides a standard function entrypoint
{3654}{3682}called `build_wheel`
{3683}{3740}in the module I just referenced here in the top
{3741}{3802}and, if we call it, it will just do its magic
{3803}{3847}and it will produce a wheel file
{3848}{3911}and its first argument it's the wheel directory
{3912}{3968}that wheel is basically a zip file
{3969}{3990}with a predefined structure
{3991}{4032}that we can extract into our store
{4033}{4081}and we are almost done
{4082}{4146}and this is what Pip does in the install phase here
{4234}{4291}and that's basically the entire build process
{4292}{4326}as specified by PEP 517
{4327}{4378}there is no `setup.py` involved any more
{4379}{4387}we don't have to call it
{4388}{4465}we don't have to create it as a package provider
{4491}{4575}so the reason why the error message I showed, showed up earlier
{4576}{4610}will keep on poping up more and more
{4611}{4645}is pretty simple we are late!
{4646}{4697}we are really really late, actually
{4698}{4784}because PEP 517 was originally created in 2015
{4785}{4869}and that it gained provisional acceptance in 2017
{4870}{4898}and just last year,
{4899}{4987}after being basically being the *de facto* successor of `setup.py` for some time
{4988}{5045}it has been finalized and fully accepted
{5046}{5137}and more importantly, flit which you remember from the previous slide
{5138}{5197}is also able to create source distributions
{5198}{5233}and upload them to PyPI
{5234}{5297}Python's public package repository basically
{5298}{5363}so far, it has been generating a `setup.py`
{5364}{5393}and does nobody really noticed
{5394}{5500}but since version 3.5, which was released in November 2021
{5501}{5561}flit stop doing that by default
{5562}{5627}and thus we are seeing more and more packages without `setup.py`
{5628}{5656}in their source distributions
{5657}{5734}and so we are basically unable to build this projects right now
{5735}{5763}or this Python modules
{5764}{5830}a look at the Guix's repository in late January
{5831}{5896}and back then only 11 packages actually used Pip
{5897}{5939}or PyPA build as we've seen
{5940}{5996}but I think our ecosystem is quite old
{5997}{6060}with about half the packages not being the latest versions available upstream
{6061}{6089}according to `guix refresh`
{6090}{6193}so it's possible that more packages actually require support for this `pyproject.toml`
{6194}{6263}and we simply have not updated them yet for whatever reason
{6264}{6321}maybe because it's too difficult or nobody poped to do it yet
{6351}{6437}they are also more issues with our current Python build system
{6438}{6546}for instance `setup.py` test ???? has been deprecated for quite some time
{6547}{6589}since 2019 actually
{6590}{6675}and the default Python build system's default check phase relies on that
{6676}{6727}even though it doesn't work for a lot of packages right now
{6728}{6791}and thus, almost every package in our repository
{6792}{6872}needs to replace that check phase with a call to Pytest
{6873}{6930}which is the *de facto* standard right now for executing tests
{6931}{7021}???? tox does something similar but not really quite the same
{7022}{7104}another long standing issue is wrapping of Python path
{7105}{7133}or Guix Python path now
{7134}{7178}because it includes native inputs
{7179}{7222}and thus propagates way too many packages
{7223}{7307}and I'm sure we can find more issues with the current Python build system
{7308}{7481}thus, for almost a year now, I've been working on a modernized Python build system
{7482}{7513}that addresses some of this issues
{7514}{7588}unfortunately I haven't had much time lately to improve it for other
{7589}{7615}I hope to pick it up again
{7616}{7628}and get it into shape
{7629}{7657}and get it to master at some point
{7658}{7686}you can read about it here
{7687}{7771}in Guix issue #46848
{7772}{7858}or you can just pull a branch called `wip-python-pep517`
{7859}{7904}it's also build by the CI by the way
{7905}{7966}so you don't have to do the whole world rebuild by yourself
{7967}{8032}and it's using exactly the method I described above
{8034}{8148}the part written in Guile ends up reading the project's `pyproject.toml`
{8149}{8264}and then calls the build system's entrypoint `build_wheel` as we can see here
{8265}{8351}and if there's no `pyproject.toml` it's also not a problem actually
{8352}{8438}because setuptools, which is still provided by default,
{8439}{8473}and still included in the build system by default,
{8474}{8564}because setuptools provides a PEP 517 build backend
{8591}{8673}which simply uses the existing `setup.py` behind the curtains
{8674}{8757}so we don't have to, like provide extra logic for that
{8787}{8847}they were actually other options for this build phase
{8848}{8876}in how to do it basically
{8903}{8943}for example, we could also use the build module
{8944}{8963}that we have seen above
{8964}{9016}and some packages use currently
{9017}{9063}however that raises questions about bootstraping, like
{9064}{9134}how do you build the build package if it essentially requires itself
{9135}{9184}because the build module is part of the Python build system
{9185}{9221}and of of it's dependency
{9222}{9279}tomli for example is one of the dependency of build
{9280}{9337}and what do we do with tomli
{9338}{9367}like how do we install it without having build
{9368}{9434}so this is tricky, and I tried it in the beginning
{9435}{9457}and it works
{9458}{9521}but we need to do like some weird manual copying
{9522}{9569}and that's not really pretty to be honest
{9570}{9598}we could also use Pip
{9599}{9656}which is usually distributed with Python
{9657}{9721}also our python package bundles Pip and setuptools
{9722}{9772}this has caused a few issues in the past
{9773}{9801}because we cannot update either of them
{9802}{9853}for provide alternative newer versions
{9854}{9917}without having a full world rebuild basically
{9918}{9959}and also Pip being a massive project
{9960}{10004}it bundles quite a few of it's dependencies
{10005}{10033}and I feel "we" as a Guix project
{10034}{10091}should try to untangle that mess whenever we can
{10092}{10120}and in this case, we actually can
{10121}{10207}so, in conclusion, I think running this 4 lines of Python code
{10208}{10236}that you can see here
{10262}{10294}is a much simpler solution
{10295}{10351}than trying to bootstrap the build module,
{10352}{10468}and tomli and all the modules required to get a Python based builder basically
{10469}{10501}we can just do it in 4 lines of code
{10556}{10606}there are a few changes for packagers
{10607}{10654}which aime to reduce the need for custom phases
{10655}{10787}first, it is possible to override the build backend detected from the `pyproject.toml`
{10788}{10848}especially early adopters of PEP 517
{10849}{10905}use modules which have been renamed in the meantime
{10933}{10990}I think both Poetry and Flit did that at some point
{10991}{11048}and they split their build logic into a separate project
{11049}{11077}and renamed the backend
{11078}{11164}so, for some packages, it's necessary to override this detected build backend
{11165}{11222}and you can do that simply by specifying the `#:build-backend` option
{11223}{11280}and also if the detection is wrong at some point
{11281}{11338}we can just override it without causing too much damage
{11345}{11416}secondly, we have `#:configure-flags`
{11425}{11483}they have existed before, but where applied only during install
{11484}{11512}but not during the build phase
{11513}{11552}which, I believe, is a ????, we'll see
{11553}{11628}and this are kind of arguments to `setup.py`
{11629}{11676}but their syntax depends on the build backend unfortunately
{11677}{11738}so sometimes you can pass the arguments directly
{11739}{11788}sometimes you need to pass them as an option to another option
{11789}{11813}like we can see here
{11814}{11870}I think that's the setuptools specific way to do it
{11871}{11906}so you have an option
{11907}{11976}and then you have to pass the actual option as an argument to that option
{11977}{12063}and, in theory, they could also look completly different
{12064}{12120}like with no relationship to any command line flag
{12121}{12181}it's just legacy code/stuff right now
{12182}{12208}and it's not pretty
{12209}{12271}and it's one of the things that have been criticized about PEP 517
{12272}{12353}this configure flags are not really specified how they are supposed to look
{12354}{12440}but this solution exists, so we have to swim with the flow, kind of
{12444}{12500}and, finally, they are `#:test-flags`
{12501}{12614}my proposal was to auto detect test binary based on the package's native inputs
{12615}{12673}so if pytest is available, it will just use pytest
{12674}{12723}if nose is available, it will use nose
{12724}{12737}and so on
{12738}{12826}now, test flags override the default arguments passed to this test command
{12827}{12892}and, hopefully, this can reduce the number of custom phases required
{12893}{12925}to exclude tests mostly
{12926}{13015}in this case, we are simply don't want to run the `test_legacy_callbacks` function
{13016}{13064}and so we can just exclude it with the flags
{13092}{13157}and the package will build just fine, probably, hopefully
{13158}{13220}so, where do we go from here?
{13221}{13310}the first step would be to finalize the work on the Python build system
{13311}{13404}and get the branch ready into a state where it can be merged
{13405}{13484}unfortunately, right now, quite a few packages fail to build there
{13485}{13571}because their previously disabled test suite fails
{13572}{13612}not necessarily disabled, but it was never ran
{13613}{13713}because the test runner was buggy and the `setup.py test` command simply did nothing
{13714}{13745}and didn't detect edge cases
{13746}{13816}we could also go a different route and add a new build system
{13817}{13841}without touching the old one
{13842}{13913}and just slowly migrate every Python package we have over
{13914}{13945}instead of doing the big switch at once
{13949}{13988}that was also one of the suggestions that was certainly possible
{13989}{14027}I have explored that it would work
{14028}{14154}as I mentioned earlier, currently the native inputs end up in wrapped binaries
{14155}{14280}and I think this is one thing we should address too while doing the whole world rebuild anyway
{14281}{14354}but I haven't done any work on that yet, unfortunately
{14355}{14444}and finally, the pypi import needs support to read `pyproject.toml`
{14445}{14499}not so much for the runtime dependencies,
{14500}{14582}this is fine, we have another metadata file which contains the runtime dependencies
{14583}{14615}but for build time and test dependencies
{14616}{14668}because they are not included in the wheel file,
{14669}{14754}there's a file called metadata, and it doesn't have this data
{14755}{14870}so right now, if you import a `pyproject.toml` based project
{14871}{14934}you will lack the build dependencies basically
{14935}{15020}for example flit will not be in the native inpust right now
{15021}{15091}and that's it for me for now
{15092}{15166}I hope to answer some of your questions during the live Q and A
{15167}{15253}and in the PDF version of this slides, you'll find some clickable links bellow
{15254}{15340}to different talks and to the standards I have referenced
{15341}{15369}so, I hope I'll see you soon!