Cfengine: Python module distribution

[Chip Seraphine]

Plz assign to Chip.

----------  Forwarded Message  ----------

Subject: approved!
Date: Thursday 27 May 2004 08:36
From: Tim Flechtner <timf@trdlnk.com>
To: Chip Seraphine <chip@trdlnk.com>

josh and jay gave the go-ahead on the pytho schtuff.  i've attached the 
most recent revision (minor corrections from the last), just for your 
reference.  i'd like to get moving on this asap, so please let me know 
when cfengine has imposed its will.

thanks!
-tim


-------------------------------------------------------



-- 

Chip Seraphine
Unix Administrator
TradeLink, LLC
312-264-2048
chip@trdlnk.com

This is an attempt to document our proposed installation of python and python 
modules (both Tradelink-developed and third party).


Python Interpreter - Solaris:
On both Solaris x86 and Solaris sparc, python will be installed in a 
subdirectory of /opt/lang/python directory named after its version (eg 
python-2.3.3).

Why a new directory?
Currently, on both these platforms, we have several different installations of 
python (eg /opt/python, /opt/Gcc3/python), with different versions of python in 
each installation, and different modules built for each installation/version 
pair.  This is confusing and hard to support.  Having one installation, with 
the superset of all required modules installed will not only be unambiguous, 
but will reduce the number of needed NFS mounts, which is important since we 
are moving to linux, where the NFS implementation is not as well done as on 
Solaris.

Why /opt/lang?
The new directory hierarchy we've agreed upon calls for four NFS mounts; 
/opt/lang for interpreted and compiled languages, /opt/app for non-language 
third party packages, /opt/tradelink for in-house developed packages, and 
/opt/admin for admin-specific utilities.  While there might be some things 
which fall into the gray area between languages and not languages (eg m4), 
python is unambiguously a language, and certainly not written in-house (except 
maybe for Skips extra-Tradelink contributions).


Python Interpreter - Linux:
Python will be installed via RPM on our linux boxes.  This means that it will 
go under /usr.  In order to allow scripts to work independent of the platform, 
there will be a symlink from /opt/lang/bin named python pointing to 
/usr/bin/python.  This symlink will also exist 'underneath' the nfs mount, so 
that python will still work when nfs is unavailable.



Tradelink Python Modules:
We will have one tradelink package, under which we will install all in-house 
developed packages and modules.  This will be accomplished by having a symlink 
from each /opt/lang/python/lib/python-<version>/site-packages (and from 
/usr/lib/python<version>/site-packages on linux) named tradelink, pointing to 
/opt/tradelink/lib/pythonpath-<version> in the python installation.  The 
pointed to directory will have a file '__init__.py', which is how python 
understands a directory represents a package.  Each in-house developed package 
will exist under these /opt/tradelink/lib/pythonpath-<version> directories, and 
be versioned as described in the next section.

Why have an uber-package named tradelink?
This allows us to avoid name collisions, just as namespaces do in c++.  It will 
be clear in python scripts that they are using Tradelink-developed modules, 
because the imports will look something like:
     import tradelink.<package>.v<major version> as <package>

Why have a bunch of /opt/tradelink/lib/pythonpath-<version>?
The byte-compiled python scripts (the .pyc files) are not guaranteed to be 
compatible across python releases.  Having one directory per python version 
protects against possible incompatibilities.

Why no tradelink.pth?
We want to avoid people implicitly picking up tradelink modules, we want them 
to have to import tradelink.<whatever>.


Tradelink Python Modules - Versioning:
Each in-house developed python module will be in a package named the same way a 
Tradelink c++ library is (eg we have ds c++ library, so the equivalent python 
package would be named ds).  Within that package will be one module per major 
release, named v<major-number>.

Why all this versioning crap?
Just as it is important to version shared objects, so that installing a new 
version doesn't impact currently installed binaries which depend on earlier 
builds, it is important to not squash earlier installs of python modules, 
becase other modules and scripts may well be using them.  By explicitly 
creating modules for each major version, we protect against this.  One 
difference between shared objects and python modules is that we don't need to 
protect against changes in a ABIs (what with there not being any for python), 
so whereas we we need to version shared objects on (at least) every major and 
minor version change, it is only necessary to version python modules on every 
major version change.

Why have modules named v<major-number>?
This makes it quite clear what version you are using, and very easy to change.  
Imports will look like this:
     import tradelink.preference.v1 as preference
later,if you want to upgrade to version 2, it would change to:
     import tradelink.preference.v2 as preference


Third Party Python Modules:
Each third part python module will be installed in the site-packages directory 
of the python directory with which it is associated.  There will be a 
3rdParty.pth file listing this directory.

Why not just install third party modules in each python version's site-packages 
directory?
Our distro of choice is Mandrake, which is an RPM distro.  RPM is a marvelous 
tool that makes many complicated chores a snap, but it does absolutely insists 
on exclusive control of the non-config files it manages.  The Mandrake python 
module RPMs (e.g. python-gtk) drop their payloads into site-packages, which 
greatly complicates any issue of synchronizing/distributing the contents of 
that directory (as well as creating potential name collision issues).  The 
traditional move when one's locally-rolled files  bump up against the RPM 
package manager is to get the hell out of RPMs way, which is exactly what the 
3rdparty subdirectory business will accomplish.

Why the name 3rdParty?
We wanted a name communicating clearly the fact that things installed therein 
were not developed in-house.

Why have a separate directories for each version?
Same reason as in the Tradelink answer to 'Why have a bunch of 
/opt/tradelink/lib/pythonpath-<version>?'