[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Cfengine: Python module distribution
Cfengine: Python module distribution
Thu, 27 May 2004 10:47:29 -0500
Plz assign to Chip.
---------- Forwarded Message ----------
Date: Thursday 27 May 2004 08:36
From: Tim Flechtner <address@hidden>
To: Chip Seraphine <address@hidden>
josh and jay gave the go-ahead on the pytho schtuff. i've attached the
most recent revision (minor corrections from the last), just for your
reference. i'd like to get moving on this asap, so please let me know
when cfengine has imposed its will.
This is an attempt to document our proposed installation of python and python
modules (both Tradelink-developed and third party).
Python Interpreter - Solaris:
On both Solaris x86 and Solaris sparc, python will be installed in a
subdirectory of /opt/lang/python directory named after its version (eg
Why a new directory?
Currently, on both these platforms, we have several different installations of
python (eg /opt/python, /opt/Gcc3/python), with different versions of python in
each installation, and different modules built for each installation/version
pair. This is confusing and hard to support. Having one installation, with
the superset of all required modules installed will not only be unambiguous,
but will reduce the number of needed NFS mounts, which is important since we
are moving to linux, where the NFS implementation is not as well done as on
The new directory hierarchy we've agreed upon calls for four NFS mounts;
/opt/lang for interpreted and compiled languages, /opt/app for non-language
third party packages, /opt/tradelink for in-house developed packages, and
/opt/admin for admin-specific utilities. While there might be some things
which fall into the gray area between languages and not languages (eg m4),
python is unambiguously a language, and certainly not written in-house (except
maybe for Skips extra-Tradelink contributions).
Python Interpreter - Linux:
Python will be installed via RPM on our linux boxes. This means that it will
go under /usr. In order to allow scripts to work independent of the platform,
there will be a symlink from /opt/lang/bin named python pointing to
/usr/bin/python. This symlink will also exist 'underneath' the nfs mount, so
that python will still work when nfs is unavailable.
Tradelink Python Modules:
We will have one tradelink package, under which we will install all in-house
developed packages and modules. This will be accomplished by having a symlink
from each /opt/lang/python/lib/python-<version>/site-packages (and from
/usr/lib/python<version>/site-packages on linux) named tradelink, pointing to
/opt/tradelink/lib/pythonpath-<version> in the python installation. The
pointed to directory will have a file '__init__.py', which is how python
understands a directory represents a package. Each in-house developed package
will exist under these /opt/tradelink/lib/pythonpath-<version> directories, and
be versioned as described in the next section.
Why have an uber-package named tradelink?
This allows us to avoid name collisions, just as namespaces do in c++. It will
be clear in python scripts that they are using Tradelink-developed modules,
because the imports will look something like:
import tradelink.<package>.v<major version> as <package>
Why have a bunch of /opt/tradelink/lib/pythonpath-<version>?
The byte-compiled python scripts (the .pyc files) are not guaranteed to be
compatible across python releases. Having one directory per python version
protects against possible incompatibilities.
Why no tradelink.pth?
We want to avoid people implicitly picking up tradelink modules, we want them
to have to import tradelink.<whatever>.
Tradelink Python Modules - Versioning:
Each in-house developed python module will be in a package named the same way a
Tradelink c++ library is (eg we have ds c++ library, so the equivalent python
package would be named ds). Within that package will be one module per major
release, named v<major-number>.
Why all this versioning crap?
Just as it is important to version shared objects, so that installing a new
version doesn't impact currently installed binaries which depend on earlier
builds, it is important to not squash earlier installs of python modules,
becase other modules and scripts may well be using them. By explicitly
creating modules for each major version, we protect against this. One
difference between shared objects and python modules is that we don't need to
protect against changes in a ABIs (what with there not being any for python),
so whereas we we need to version shared objects on (at least) every major and
minor version change, it is only necessary to version python modules on every
major version change.
Why have modules named v<major-number>?
This makes it quite clear what version you are using, and very easy to change.
Imports will look like this:
import tradelink.preference.v1 as preference
later,if you want to upgrade to version 2, it would change to:
import tradelink.preference.v2 as preference
Third Party Python Modules:
Each third part python module will be installed in the site-packages directory
of the python directory with which it is associated. There will be a
3rdParty.pth file listing this directory.
Why not just install third party modules in each python version's site-packages
Our distro of choice is Mandrake, which is an RPM distro. RPM is a marvelous
tool that makes many complicated chores a snap, but it does absolutely insists
on exclusive control of the non-config files it manages. The Mandrake python
module RPMs (e.g. python-gtk) drop their payloads into site-packages, which
greatly complicates any issue of synchronizing/distributing the contents of
that directory (as well as creating potential name collision issues). The
traditional move when one's locally-rolled files bump up against the RPM
package manager is to get the hell out of RPMs way, which is exactly what the
3rdparty subdirectory business will accomplish.
Why the name 3rdParty?
We wanted a name communicating clearly the fact that things installed therein
were not developed in-house.
Why have a separate directories for each version?
Same reason as in the Tradelink answer to 'Why have a bunch of
- Cfengine: Python module distribution,
Chip Seraphine <=