Please review: documentation for python build system

[Hartmut Goebel]

Hi,

we are about to merge the wip-python-build-system. I'd like to add some
commentary there explaining how the different install methods of Python
behave. This could save someone else the effort of anaylsing and testing
this again is some questions about the Python build system arise.

I ask you to review the text below. Since we want to merge soon, I'd ask
you to review soon. Thanks.


In Python there are different ways to install packages: distutils,
setuptools,
easy_install and pip.  All of these are sharing the file setup.py,
introduced
with distutils in Python 2.0.  setup.py can be considered as a kind of
Makefile accepting targets (or commands) like "build" and "install".  As of
autumn 2016 the recommended way to install Python packages is using pip.

For both distutils and setuptools running "python setup.py install" is
the way
to install Python packages.  With distutils the "install" command basically
copies all packages into <prefix>/lib/pythonX.Y/site-packages.

Some time later "setuptools" have been established to enhance distutils.  To
use setuptools, the developer imports setuptools in setup.py.  When
importing
setuptools, the original "install" command gets overwritten by setuptools'
"install" command.

easy_install and pip are both command-line tools capable to search and
download the package source from PyPI (the Python Package Index).  Both of
them import setuptools and execute the "setup.py" file under their control.
Thus the "setup.py" behaves as if the developer had imported setuptools
within
setup.py - even is still using only distutils.

Setuptools' "install" command (to be more precise: the "easy_install"
command
which is called by "install") will put the path of the currently installed
version of each package and it's dependencies (as declared in setup.py) into
an "easy-install.pth" file.  In guix each packages gets it's own
"site-packages" directory and thus an "easy-install.pth" of it's own.  To
avoid conflicts this file gets renamed to <packagename>.pth in phase
rename-pth-file.  To ensure the .pth-file will be process, easy_install also
creates a basic "site.py" in each "site-packages" directory. The file is the
same for all packages, thus there is no need to rename it.

The .pth-files contain the file-system paths (pointing to the store) of all
dependencies.  So the dependency is hidden in the .pth file but is not
visible
in the file-system.  Now if packages A and B both required packages P,
but in
different versions, guix will not detect this when installing both A and
B to
a profile. (For details and example see
https://lists.gnu.org/archive/html/guix-devel/2016-10/msg01233.html.)

Now pip behaves a bit different: it always executes "setup.py" with the
option
"--single-version-externally-managed" set.  This makes setuptools' "install"
command to *not* run "easy_install" but the original "install" command and
thus no .pth-file (and no site.py) will be created. The "site-packages"
directory only contains the package and the related .egg-info directory.

This is exactly what we need for guix and this is what we mimic in the
install phase below.

As a draw back, the magic of the .pth file of linking to the other required
packages is gone and these packages have now to be declared as
"propagated-inputs".

Note: Importing setuptools also adds two sub-commands:
"install_egg_info" and
"install_scripts".  These sub-commands are executed even if
"--single-version-externally-managed" is set, thus the .egg-info
directory and
the scripts defined in entry-points will always be created.

Note: Even if the "easy-install.pth" is not longer created, we kept this
phase.  There still may be packages creating an "easy-install.pth" manually
for some good reason.



-- 
Regards
Hartmut Goebel

| Hartmut Goebel          | address@hidden               |
| www.crazy-compilers.com | compilers which you thought are impossible |