Re: [gpsd-dev] GPSD and the Python 3 transition

[Fred Wright]

On Thu, 24 Mar 2016, Eric S. Raymond wrote:

> Fred Wright <address@hidden>:
> > > > In fact, although I'm not a fan of monoversioned shebang lines in 
> > > > general,
> > > > given that Python packages are normally installed in a version-specific
> > > > location, some means for writing the corresponding shebang lines into 
> > > > the
> > > > installed executables should probably exist, at least as an option.
> > >
> > > I am directly and strongly opposed to this.  Polyglot Python (that is, 
> > > capable
> > > of running under 2 or 3) is not difficult enough to write that the extra
> > > layer of build complexity required to hack in version-specific shebang 
> > > lines
> > > is justified.
> >
> > This isn't strictly a versioning issue at all.  The problem is that Python
> > packages (including the 'gps' package) get installed into a directory
> > within the tree of one specific Python installation, and imports of such a
> > package only work when run with that exact same Python installation.  In
> > particular, if a package is installed into the default Python and then the
> > default Python later changes, Python programs with a generic shebang line
> > using that package stop working.
>
> Ah, yes, you're right.  But...
>
> Much of the Python code doesn't require loading libraries at all.
> Notably this includes gpscap.py, jsongen.py, maskaudit.py, and
> valgrind_audit.py. It makes sense to want these to be polyglot, and
> *not* patch their shebang lines, even under your worst-case
> assumptions.

Those are build-time utilities, which necessarily need to avoid
dependencies on installed libraries.  For consistency, they should be run
with the scons Python (which is what happens now), regardless of which
Python version is the target for installation.  That's of course implied
for any Python code that's imported rather than run as a separate
executable, but explictly using the scons Python in the latter case (and
hence ignoring the shebang lines) is maximally consistent.

Things like regression tests, OTOH, should be run with the target Python,
for maximum consistency with the intended target environment.

Side note on imports in scons:

Scons adds the SConstruct directory to sys.path while processing the
SConstruct file, but then removes it before running the builders.  So
imports from the SConstruct top level can see modules in the build
directory, but imports from builder functions can't (unless they reference
modules that wer already imported; "from foo import bar" always imports
all of foo, and the only thing selective about it is the "publication" of
the name(s)).

> One reason I have a twitch reaction to hacking shebang lines is that
> the last person who advocated this (and sent a patch which I somewhat
> incautiously took) misunderstood the problem and created build
> machinery to shebang-patch two utilities that don't do package loads
> at all.  I still need to fix this; it's an unneeded wart on a build
> that is too complicated by half.

Yeah, I noticed that, though only for valgrind-audit.

> This leaves us with the following: gegps, gpscat, gpsfake, gpsprof,
> test_maidenhead.py, xgps, and xgpsspeed. They have all possible
> combinations of depending one one or both of gps and clienthelpers.
>
> But, OK, supposing we patch the shebangs in these.  What problem would this
> actually solve?  You say "the default Python later changes", but if that's
> the case we don't actually have any guarantee that the old library locations
> will continue to exist. For example, on my Ubuntu 15 system:
>
> address@hidden:~/software/gpsd$ ls /usr/lib/python
> python2.7/ python3/   python3.4/ python3.5/
>
> Notice what's missing?  The 3.x point releases before 4. And 2.6.  And 2.5.
> What patching shebangs actually does is make us hostage to distributors'
> whims about when a Python point release is no longer interesting.

If the version of Python where the libraries were installed is removed,
then the libraries were removed with it, and so the install is broken no
matter what you do.  But changing the default Python *without* removing
the one where the libraries were installed shouldn't break the install,
which is what happens with the generic shebang.

This is especially significant for developers with multiple versions of
Python installed for testing, e.g.:
---------------------------------------------------------------------------
MacPro:~ fw$ port select python
Available versions for python:
        none
        python24
        python25-apple
        python26
        python26-apple
        python27 (active)
        python27-apple
        python31
        python32
        python33
        python34
        python35
---------------------------------------------------------------------------
But even end users will probably wind up with at least two for the Python
2->3 transition (i.e., the next few years).

For users with exactly one python2 and possibly one python3, the 3.16
approach of using python2 in the shebang line works fine, and it simply
never uses Python 3.  Though that change broke things for users with no
python2 command.

> Better, I think, to bail out gracefully on a failed import with a message
> that says you should either reinstall GPSD from source or upgrade to a GPSD
> package version matching the system Python.

I don't see how that's significantly more helpful than the system message
that says that the Python version requested by your program doesn't exist.
And of course anything managed through a packaging system shouldn't allow
you to uninstall a Python without uninstalling all programs that list it
as a dependency.

What I'm proposing is an *option* to rewrite shebang lines in the
installed versions of the programs.  It's really up to the packaging
system to decide where things will be placed (and hence what to put
there), but knowledge of *what* to patch belongs in the GPSD SConstruct,
rather than having the packaging system poking tentacles into it and
having to be updated whenever this list of affected programs changes.

Meanwhile, the default python_libdir should be based on whatever Python is
the target version for the install, rather than the scons Python (which
for now is guaranteed not to be any flavor of Python 3).

If a packaging system decides to support multiple Python versioning by the
brute-force method of installing multiple copies of the libraries, then it
would also be useful to have an install target for just those, so that it
could iterate over Python versions without duplicating installs of the C
code (or worse, uninstalling the C code if just one Python version were
uninstalled).  It would be better, of course, to install to one place and
symlink appropriately, but I don't know of any system that does that.

Side note:  It would probably make sense to redefine the existing 'python'
option to mean "include Python code in installs" rather than "use Python
code at all".  Back when the build system was make-based, it may have been
useful to have an option to avoid any dependency on Python, but now that
it's scons-based, Python avoidance is pretty pointless.  In particular,
it's rather silly to have a Python conditional to handle the case where
Python is unavailable. :-) If this were reworked, then the regression
tests wouldn't need to be suppressed in the python=no case.

FYI - the updated xgps passes a brief smoke test on the Mac after
installing py-gobject3 from MacPorts.

Fred Wright