[Nhproxy-discuss] Designing a shared library to support multiple interface versions

[J. Ali Harlow]

Dear libtool users,

I've been struggling with how best to design a shared library to
support multiple interface versions. I've used shared libraries before
but this is the first time I've needed to design one from scratch and
I've spent quite some time thinking about how best to do so.

I've come to the conclusion that the best thing is to define a unique
initialization routine for each numbered interface. This surprises me
somewhat since it's not something that I recall meeting in any shared
library that I have used so I'm worried that I'm making a mistake.

I would be very grateful if you would either read through my notes
below and tell me about any inaccuracies or point me to some more
information on this topic. As an aside, it would be nice if the libtool
manual contained pointers to such resources if they are available.

This is in relation to NetHack Proxy (http://www.nongnu.org/nhproxy/)
although I don't think there's anything specific to that particular
package in my notes.

Many thanks,

Ali.

Background
----------

Libtool uses the concept of interface numbers where the application and
the DSO must agree on the interface number to use.

Strictly speaking, the interface number is a subset of the full ABI
definition since calling conventions etc., also need to be taken into
account. For normal purposes however we ignore this and assume that
for each platform the interface number suffices to uniquely identify
the ABI. Where this is not the case (eg., under MS-Windows with
mingw32/cygwin) the convention is to use different DSO names to further
distinguish ABIs (libxxx-#.dll vs. cygxxx-#.dll).

Libtool
-------

Libtool defines three qualtities:
* the current interface (the highest interface supported by a DSO)
* the interface age (current-age is the lowest interface supported)
* the interface implementation revision

Libtool attempts to encode the interface range implemented by a DSO
into the binary so that the dynamic linker can choose a DSO that
implements the interface number needed by the application (using the
DSO with the higher revision number where possible). In practice,
however, this is not possible on all platforms (including MS-Windows).
About all you can assume is that libtool will ensure that the dynamic
linker will choose a DSO which has the same lowest interface supported
as that supported by the version of the package that the application
was linked with.

How interfaces are used
-----------------------

There are three stages where interfaces are important to an
application:
        * compile
        * link
        * start-up

At compile time, the API to which the application conforms together
with the supplied header files produce a set of object files which talk
an ABI denoted by an interface number.

At link time, these object files are combined with an import library
which implements an ABI denoted by an interface number. If the header
files came from the same version of the package as the import library
then this will be the same interface number.

At start-up, a suitable DSO is found and linked to the import library.
The DSO may implement a range of interfaces.

Requirements
------------

In order to make best use of libtool's version system, there has to be
some way in which the application can inform the DSO which interface
it wishes to use and some way of ensuring that the application doesn't
attempt to run if the DSO doesn't support that interface.

We would also like to use a system that coped with the possibility of
compiling an application against header files from one version of the
package but linked against the import library from a different version.
This unfortunate situation happens all too often in practice.

Cases to consider
-----------------

1. The application was built using a version of the package whose
current interface number was I1. The dynamic linker chose a DSO
whose current interface number was also I1.

In this case, no special care need to be taken. The application and
DSO can communicate using interface I1.

2. The application was built using a version of the package whose
current interface number was I1. The dynamic linker chose a DSO
whose current interface number was I2 (I2 > I1) but which also
implements I1.

In this case, the DSO just needs to know that it should communicate
using interface I1 and all will be well.

3. The application was built using a version of the package whose
current interface number was I1. The latest DSOs installed has a
current interface number of I2 (I2 < I1).

In this case, proper communication cannot be achieved. Under all
platforms, the dynamic linker will fail with a suitable error
message.

4. The application was built using a version of the package whose
current interface number was I1. The dynamic linker chose a DSO
whose current interface number was I2 (I2 < I1).

Example:
        Version 1.0 of package implements interface 0 only:
                current is 0, age is 0, current-age is 0.
        Version 1.1 of package implements interfaces 0 and 1:
                current is 1, age is 1, current-age is 0.
        Application is built against version 1.1:
                I1 is 1.
        Application is run on a machine which has version 1.0
     installed:
                I2 is 0.

In this case, proper communication cannot be achieved. The dynamic
linker will fail if, and only if, interface I1 adds a symbol which
is used by the application (since it won't be present in the DSO).

5. The application was compiled using a version of the package whose
current interface number was I1 and linked with a version of the
package whose current interface number was I2 (I2 > I1) but which
also implements I1.

In this case, since both versions of the package implement interface
I1, the values of current-age must be the same. Thus, if the link
succeeds and if we don't do funky things in the import library,
the resultant application executable should act exactly as if it had
been linked with the import library from the earlier version of the
package. If interface I2 deleted or renamed a symbol and the import
library for the later version has the old symbol removed, then the
link will fail.

6. The application was compiled using a version of the package whose
current interface number was I1 and linked with a version of the
package whose current interface number was I2 (I2 > I1) which
doesn't implement I1.

This case may fail to link, if (as above) interface I2 deleted
or renamed a symbol used by the application. However, if the link
succeeds then the value of current-age used by the later version of
the package will be different (higher) than the value used by the
earlier version and thus the dynamic linker will be working with the
wrong information leading to the following cases:

a. The dynamic linker chose a DSO whose current interface number was
   I3 (I3 >= I2).
b. The dynamic linker chose a DSO whose current interface number was
   I3 (I1 < I3 < I2). The DSO does not implement interface I1.

   In these cases, proper communication cannot be achieved. If the
   DSO can detect that the application is attempting to communicate
   with an unsupported interface number, then a run-time error can
   be issued.

c. The dynamic linker chose a DSO whose current interface number was
   I3 (I1 <= I3 < I2). The DSO also implements interface I1.
d. The dynamic linker chose a DSO whose current interface number was
   I3 (I3 < I1).

   These cases are only possible if somebody has renamed the DSO
   since we know that current-age for the import library must be
   greater than I1. As it happens, the first will work while the
   second cannot. It may fail at start-up if interface I1 added any
   symbols to I3 or at run-time if the DSO can detect that the
   application is attempting to communicate with an unsupported
   interface number.

7. The application was compiled using a version of the package whose
current interface number was I1 and linked with a version of the
package whose current interface number was I2 (I2 < I1). Interface
I2 is supported by both versions of the package.

This case may fail to link, if interface I1 added or renamed a
symbol used by the application. However, if the link succeeds then
the values of current-age must be the same (since both versions of
package implement interface I2). Thus, as in case 5, barring funky
import libraries things should work fine.

8. The application was compiled using a version of the package whose
current interface number was I1 and linked with a version of the
package whose current interface number was I2 (I2 < I1). Interface
I2 is no longer supported by the later version of package.

This case may fail to link, if interface I1 added or renamed a
symbol used by the application. However, if the link succeeds
then the value of current-age used by the earlier version of the
package will be different (lower) than the value used by the later
version and thus the dynamic linker will be working with the wrong
information. For the dynamic linker to succeed it must chose a DSO
which supports interface I2. Such a DSO cannot also support
interface I1 and therefore proper communication cannot be achieved.
If the DSO can detect that the application is attempting to
communicate with an unsupported interface number, then a run-time
error can be issued.

Techniques
----------

1. Divide and conquer. This is the approach used by Microsoft for
their main win32 API. It works well for large APIs such as this
where otherwise interface numbers would climb too quickly.

Each symbol has it's own interface which is defined once and never
changed. If a new interface is required then a new symbol is added
to implement it. Entry points can use parameters to extend their
interface (eg., the use of an initial field in structures to contain
the structure size).

Advantages: Very general.

Disadvantages: Awkward to both set up and use.

2. Run-time setting of interface number. Eg., package_set_abi(1).

Advantages: This provides protection against cases where the dynamic
linker would otherwise link to a DSO which isn't able to communicate
using the same interface as the application. It also allows DSOs
to modify their behaviour to communicate with interfaces that they
do support.

Disadvantages: No protection at link-time against mis-matched
headers and libraries. Errors caused by incompatible interfaces may
cause either an error from the dynamic linker or from the DSO at
run-time leading to the potential for confusion.

3. Build-time setting of interface number. Eg., package_set_abi1().

Advantages: This has all the advantages of the above with the added
benefit of trapping mismatched headers and libraries at link-time.
Also, attempting to start the application with an incompatible DSO
will produce consistent error messages (from the dynamic linker).

Disadvantages: More symbols means some small cost in application
start-up time and disk space.

4. Restrict yourself to only one interface per DSO (eg., libtool's
-release).

Advantages: The dynamic linker will only ever choose a DSO which is
compatible and will know the name of the required DSO if it is not
installed on the end-user's system making it easier for them to
find and install it.

Disadvantages: The end-user is likely to need to install more
versions of the package than other techniques would allow.

5. Implement sub-interfaces. Here each DSO only implements one
interface, which may contain a number of sub-interfaces. This is
something like what COM uses: you use the interface to request
access to a vtable or equivalent for the sub-interface you're
interested in.

Advantages: Very powerful.

Disadvantages: Awkward to both set up and use.

Conclusions
-----------

Build-time setting of interface number seems like a no-brainer. This
could most easily be done with the following code in package's
configure.ac:

AC_DEFUN([package_lt_current],0)
PACKAGE_INIT=[package_init]package_lt_current
AC_SUBST([PACKAGE_INIT])

and this in package's package.h.in:

#define PACKAGE_INIT()          @PACKAGE_INIT@()

Each application calls PACKAGE_INIT() before using any other part of
package.

References
----------

Libtool manual, section 6
http://www.gnu.org/software/libtool/manual.html#Versioning