[gpsd-dev] Python Shebang Lines

[Fred Wright]

I'd like to reraise the issue of shebang lines in Python programs.  To
review, the problem is as follows:

Prior to Python 3, the only issue was the location of any third-party
libraries used by the program.  In the case of running programs directly
from the build directory, this was misleadingly a non-problem, since
Python always adds the program's directory to the module search path.  So
any Python program run from the main gpsd build directory could always
find the modules in gps/.  For *installed* Python programs, however, this
wouldn't necessarily work, since the Python libraries are installed in the
tree of a specific Python installation (this is normal practice not unique
to gpsd), and could only be found if the program was run with the same
version of Python as was used for the install.  With generic shebang
lines, if the system default Python differed from the install Python
(originally always the SCons Python), the programs wouldn't work.

With the advent of Python 3, things got more complicated.  The C
extensions need to be built differently depending on whether they're built
for Python 2 or Python 3.  And in some cases, the specific version of
Python 3 even matters.  So not only does the above problem with installed
Python programs still exist, but in some cases, even running from the
build directory doesn't work, if the system default Python differs from
the Python version used to build the extensions.  In fact, since the
default is still to build for the SCons Python, this is guaranteed to fail
when the system default Python is any Python 3, since SCons only works
with Python 2.

Way back, I implemented a target_python option to allow the Python version
for building and installing the extensions to be specified, but this
doesn't currently do anything for the "executables".  The regression tests
work because the script always invokes the target_python explicitly for
any Python programs that *it* runs, but this doesn't help user-invoked
programs.  So essentially, the current target_python mechanism is
incomplete.

The main thing to remember here is that, although a program written to be
compatible with a wide range of Python versions *which only uses standard
Python libraries* can work just fine with a generic shebang line,
programs that rely on third-party libraries will generally have issues if
they use generic shebang lines and are run on systems with more than one
Python version installed.

At one point, Eric said that he thought that it was rare for people to
have multiple Python versions installed, but I think that's likely to
become a lot less true with the advent of Python 3.  Python 3 is
sufficiently incompatible with Python 2 that it will probably be quite
some time before it's reasonable to have *only* Python 3 installed.  And
of course, until SCons is fixed for Python 3, then any system with a
default Python 3 which is also capable of building gpsd is guaranteed to
have at least two Python versions installed.

It's also worth noting that this issue isn't strictly about Python
versions, but actually about specific Python installs.  It's possible to
have two (or more) installs of a given Python version, in which case
merely specifying the version in the shebang line may not be sufficient to
guarantee the correct selection.  In such (admittedly less common) cases,
the full path to the proper Python install is needed.

I've currently implemented and tested a mechanism for optionally rewriting
shebang lines based on the same target_python option that already controls
the library builds.  The default value is still blank, meaning that the
libraries are built and installed for the SCons Python, and that the
shebang lines are the unchanged generic versions.

The original meaning of the target_python option is as follows:

1) Blank (default): Build and install for SCons Python

2) Non-blank: Use the value as a command (either absolute or
PATH-relative) to launch a Python interpreter, and build and install for
that Python version.

For shebang purposes, this gets expanded a bit, to handle both absolute
and PATH-relative values appropriately:

1) Blank (default): No change to shebang lines.

2) Starting with '/': Prefix this with "#!" to form the replacement
shebang line.

3) Non-blank, not starting with '/': Prefix this with "#!/usr/bin/env " to
form the replacement shebang line.

Note that specifying 'python' as the value is case #3 but with a NOP
shebang rewrite.

In addition, because it's sometimes necessary to use an absolute path,
it's useful to let the script determine this as a convenience to the user,
rather than requiring the user to figure this out (particularly since it
already has code to determine the path internally for its own purposes,
anyway).  Since this is potentially applicable to any of the above cases,
it makes the most sense to indicate it via a prefix character.  The
character I've currently chosen is '@', which is somewhat mnemonic since
it means indirect addressing in some contexts.

Thus, the additional cases determine the shebang lines as follows:

4) '@': Prefix SCons Python's sys.path with "#!".

5) '@/path/to/python': Prefix that Python's sys.path with "#!".

6) '@python_command':  Prefix that Python's sys.path with "#!".

Note that the apparent NOP '@' in case #5 may not atually be a NOP if the
absolute path returned by sys.path differs from the supplied absolute
path.

In terms of mechanism, rather than modifying any files in place or using
the typical "substituter", it just renames the existing Python programs to
"sources" with the ".py" extension, and then the "executables" without the
extension are "built" by copying with a "builder" that optionally replaces
the first line.  That means that even the "unbuilt" versions can still be
run if desired, though in most cases they won't work in a completely
virgin build directory anyway, due to needing the C extension(s) built.

An added bonus is that the builder nodes provide a place to attach
dependencies on the C extensions, where needed.

I wanted to post this here to make sure nobody screams before pushing the
change to the main repo.  At present, the default behavior is unchanged,
other than the internal renaming of the six programs to XXX.py and the
need to "build" to get them back without the .py.  That's probably not the
best default, but it makes sense to keep policy changes and mechanism
changes separate.

Fred Wright

P.S.:  Python 3.6 was officially released Friday, and I've been including
it in testing, finding one minor whitespace bug in the process.