Re: [gpsd-dev] GPSD and the Python 3 transition

[Fred Wright]

On Thu, 24 Mar 2016, Eric S. Raymond wrote:
> Fred Wright <address@hidden>:
> > > This does mean the floor Python version we can use is now 2.7; 2.6
> > > is not compatible enough with 3.x for my polyglot trickery to work.
> >
> > Which specific things don't work in 2.6?
>
> There are some 3.x syntactic innovations that were backported into
> 2.7.2 and following that aren't allowed in 2.6 or very early
> 2.7s. Until, that is, a mob of pitchfork-wielding Pythonistas stormed
> Dr. van Rossum's castle yelling "Give us back our forward
> compatibility!", or something like that.
>
> I think the "as" syntax for exception-catching was one of them, but
> I've forgotten the details. It was a confusing time.  The Hammer Films
> B-movie about it would be a real thrill-ride.

Definitely not that one, which *does* work in 2.6 (or a sufficiently late
2.6, anyway).  I know the official doc on writing python2/3-compatible
code says not to bother supporting anything older than 2.7, but that was
written by the same folks who used to claim that making code compatible
with both 2 and 3 was a bad idea altogether.  That is, until they realized
that their vision of a world where everyone instantly and simultaneously
ported all their Python code to Python 3 was pure fantasy. :-)

> > I presume you were at least pointing python_libdir at the appropriate
> > place for the Python 3 you were using, since the default (obtained from
> > the scons Python) is guaranteed to be inappropriate for Python 3.
>
> Hadn't got that far yet.  Solving the xgps problem was higher on my list.
> But yes, I'd figured out the theory.

Well, you have to do that for any proper test, since otherwise the 'gps'
package won't be importable at all, unless you're running from the build
directory, where Python "helpfully" adds the directory of the invoking
script to sys.path.  Hence, all proper testing of this type needs to be
done using *installed* programs.

> > In fact, although I'm not a fan of monoversioned shebang lines in general,
> > given that Python packages are normally installed in a version-specific
> > location, some means for writing the corresponding shebang lines into the
> > installed executables should probably exist, at least as an option.
>
> I am directly and strongly opposed to this.  Polyglot Python (that is, capable
> of running under 2 or 3) is not difficult enough to write that the extra
> layer of build complexity required to hack in version-specific shebang lines
> is justified.

This isn't strictly a versioning issue at all.  The problem is that Python
packages (including the 'gps' package) get installed into a directory
within the tree of one specific Python installation, and imports of such a
package only work when run with that exact same Python installation.  In
particular, if a package is installed into the default Python and then the
default Python later changes, Python programs with a generic shebang line
using that package stop working.

E.g., here, after running "scons install" from HEAD:
----------------------------------------------------------------------------------
MacPro:~ fw$ for p in $(which -a python2.7); do echo $p; $p $(which gpsfake) 
-T; done
/opt/local/bin/python2.7
sys darwin platform Darwin-13.4.0-x86_64-i386-64bit: WRITE_PAD = 0.03000
/usr/bin/python2.7
Traceback (most recent call last):
  File "/usr/local/bin/gpsfake", line 16, in <module>
    import gps
ImportError: No module named gps
/usr/local/bin/python2.7
Traceback (most recent call last):
  File "/usr/local/bin/gpsfake", line 16, in <module>
    import gps
ImportError: No module named gps
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
sys darwin platform Darwin-13.4.0-x86_64-i386-64bit: WRITE_PAD = 0.03000
----------------------------------------------------------------------------------
The first and last cases are actually both using the MacPorts python2.7,
which is the one used by scons and hence the one where the 'gps' package
got installed.  The other two cases fail simply because they're not *the
same* python2.7 as the one that ran scons at install time.

I agree that this totally sucks, but it's not something GPSD is in a
position to fix, and the only way to make the executables work reliably is
to make sure that they always use whichever Python has the 'gps' package
installed (currently the "scons Python" by default).

It's disappointing to do all the work to allow changing "#!/usr/bin env
python2" back to "#!/usr/bin/env python" only to find that the latter
never really worked as desired anyway, but that's unfortuately the way it
is (for anyone with multiple versions of Python installed).  The advent of
Python 3 will make this worse by making it more common for non-developers
to have multiple Pythons.

Running programs from the build directory covers up the import problem
(see above), and lets one see the actual version issues:
----------------------------------------------------------------------------
MacPro:gpsd fw$ python2.7 ./gpsfake -T
sys darwin platform Darwin-13.4.0-x86_64-i386-64bit: WRITE_PAD = 0.03000
MacPro:gpsd fw$ python2.6 ./gpsfake -T
sys darwin platform Darwin-13.4.0-x86_64-i386-64bit: WRITE_PAD = 0.03000
MacPro:gpsd fw$ python2.5 ./gpsfake -T
./gpsfake:106: Warning: 'as' will become a reserved keyword in Python 2.6
  File "./gpsfake", line 106
    except getopt.GetoptError as msg:
                               ^
SyntaxError: invalid syntax
MacPro:gpsd fw$ python2.4 ./gpsfake -T
  File "./gpsfake", line 106
    except getopt.GetoptError as msg:
                               ^
SyntaxError: invalid syntax
----------------------------------------------------------------------------
But this isn't a valid test of what an end user would see.

BTW, it seems like a stretch to use "polyglot" in this context.  It's not
as if Python 2 and Python 3 are completely different languages.  Though
the non-monotonic relation between version numbers and features may make
it seem that way. :-)

Fred Wright