Re: [Gcl-devel] New user question - more explicit information concerning shared library use

[Camm Maguire]

Greetings, and thanks so much for your feedback!

Jordan Rosenthal <address@hidden> writes:

> Hi all,
> 
> We have a fairly large codebase for image processing that was
> developed on top of an older version of GCL (it is currently running
> on top of 2.4.4).  Here's a little bit of info about what we're running
> 
> -------
> - Linux OS (although we had a version on Solaris)
> - GCL 2.4.4
> - extensive use of the GCL to TCL/TK 8.4 link
> - On top of the TCL/TK we have built the BLT library extension (this
> does not affect the GCL build directly) and a custom TK extension
> - Our system loads in a bunch of compiled files on top of GCL and does
> a si::save to create a new image
> -------
> 
> One of the biggest annoyances I have with our system is that it is
> really painful to use external C and LISP libraries.  For C libraries,

Its much better now, but hopefully will still improve further with
2.7.0, hopefully out by summer's end.  There is a recent thread on
gcl-devel -- your ideas most welcome.

> we have been using the EXTRAS make variable to link in certain
> symbols, but this is not fun to do.  External LISP libraries never
> seem to work out of the box, because most of them are dependent on
> some of the newer ANSI features (e.g., objects don't exists in GCL
> 2.4.4).
> 
> Lo and behold, I've recently out found that newer versions of GCL are
> being worked on by the group reading this email!  It is moving towards
> ANSI compliance and apparently has the ability to link to shared
> libraries with the compiler::link command.  (I have also seen posts
> regarding the gcl-elf-loader that can do this at runtime.)
> 

The magnificent elf-loader work has not yet been fully integrated, as
there are several possibilities esp. regarding unexec we still need to
examine.  Hopefully this will be resolved in 2.7.0.

> I downloaded GCL 2.6.6 and have compiled without any (major)
> issues. However, I am having trouble with a few things in the new
> version:
> 
> 1) There are a bunch of configure options to get dynamic loading of
> compiled files (e.g., static bfd, local bfd, dynamic bfd, dlopen,
> etc). Which is the preferred method?  Currently, I am using the
> dynamic bfd (i.e., the shared bfd libraries on my system).  Also, how
> does the gcl-elf-loader fit into the picture?
> 

dlopen is the least preferred.  One of the bfd's are slightly better
than cust-reloc, as it has been more fully exercised.  I prefer locbfd
which will carry the bfd object modules around in libgcl, as if you
want to run compiler::link at some later date on a different machine
with different bfd version, you will definitely have problems.
dynsysbfd will fail to startup in such a case, as the bfd soname
policy is to make every point release binary incompatible.  ld.so will
just punt in such a case.

This said, our local bfd is somewhat old, but quite serviceable.  But
the bottom line is that if you are not using either locbfd or
cust-reloc, then you need to ensure that the same external bfd lib is
present whenever you might want to run compiler::link.


> 2) How do I use the compiler::link command?  The syntax is
> straight-forward, but whenever I do it I get errors.  I have tried
> both using and not using the system-p key, but in either case I get
> some sort of error.  In one case (I can't remember which) I get a
> "cannot relocate" error.  In the other case I get a "multiple
> instances of init_code defined" error.
> 

You must compile with :system-p t for compiler::link for the init
names to be unique as required by ld.  Cannot relocate likely
indicates that you forgot to link in your new shared library.

> I think my problems most likely stem from a lack of understanding
> about how GCL works internally, especially in regard to dynamic
> loading of object files.  Also, I've only been using LISP (the GCL
> 2.4.4 variety) for a little while now so sometimes that can be a
> little bit of a block.
> 
> If anyone could give me a brief tutorial (or point me to some reading
> material) on how this stuff works, I'd really appreciate it.  I've
> reversed engineered some of the code, but that is slow going and some
> sort of overview would be much appreciated.
> 
> Perhaps if it is simpler, we could work with a toy problem.  Let's say
> I have an external shared library...maybe the BLAS numerical library
> or the libxml2 library on Linux.  How do I create an image that let's
> me link to all the symbol's in these (shared) libraries from a clines
> construct later on?
> 

Hopefully this example will suffice:

=============================================================================
/tmp/b.l
=============================================================================
(clines "

extern double ddot_(int *,double *,int *,double *,int *);

static double
ddot(object x,object y,int n) {

int one=1;

return ddot_(&n,x->lfa.lfa_self,&one,y->lfa.lfa_self,&one);

}

")

(defentry %ddot (object object int) (double "ddot"))

(defun ddot (x y) 
  (unless (and (typep x '(vector long-float)) (typep y '(vector long-float)))
    (error "Bad argument types ~a ~a~%" (type-of x) (type-of y)))
  (%ddot x y (min (length x) (length y))))
=============================================================================
$ gcl
GCL (GNU Common Lisp)  2.6.7 CLtL1    May  1 2005 21:25:42
Source License: LGPL(gcl,gmp), GPL(unexec,bfd)
Binary License:  GPL due to GPL'ed components: (READLINE BFD UNEXEC)
Modifications of this banner must retain notice of a compatible license
Dedicated to the memory of W. Schelter

Use (help) to get some basic information on how to use GCL.

>(compile-file "/tmp/b.l" :system-p t :c-file t)

Compiling /tmp/b.l.
End of Pass 1.  
End of Pass 2.  
OPTIMIZE levels: Safety=0 (No runtime error checking), Space=0, Speed=3
Finished compiling /tmp/b.l.
#p"/tmp/b.o"

>(compiler::link (list "/tmp/b.o") "bb" "" "-lblas" t)
GCL (GNU Common Lisp)  April 1994  131072 pages
Building symbol table for /fix/t1/camm/debian/gcl/raw_bb ..
loading /usr/lib/gcl-2.6.7/unixport/../lsp/gcl_export.lsp
Initializing gcl_defmacro.o
Initializing gcl_evalmacros.o
Initializing gcl_top.o
Initializing gcl_module.o
loading /usr/lib/gcl-2.6.7/unixport/../lsp/gcl_autoload.lsp

>
("b")

>
#<"COMPILER" package>

>
#<"SLOOP" package>

>
#<"SERROR" package>

>
#<"ANSI-LOOP" package>

>
#<"DEFPACKAGE" package>

>
#<"TK" package>

>
#<"SYSTEM" package>

SYSTEM>
*COMMAND-ARGS*

SYSTEM>Initializing gcl_predlib.o
Initializing gcl_setf.o
Initializing gcl_arraylib.o
Initializing gcl_assert.o
Initializing gcl_defstruct.o
Initializing gcl_describe.o
Initializing gcl_iolib.o
Initializing gcl_listlib.o
Initializing gcl_mislib.o
Initializing gcl_numlib.o
Initializing gcl_packlib.o
Initializing gcl_seq.o
Initializing gcl_seqlib.o
Initializing gcl_trace.o
Initializing gcl_sloop.o
Initializing gcl_serror.o
Initializing gcl_destructuring_bind.o
Initializing gcl_loop.o
Initializing gcl_defpackage.o
Initializing gcl_make_defpackage.o
Initializing gcl_cmpinline.o
Initializing gcl_cmputil.o
Initializing gcl_debug.o
Initializing gcl_info.o
Initializing gcl_cmptype.o
Initializing gcl_cmpbind.o
Initializing gcl_cmpblock.o
Initializing gcl_cmpcall.o
Initializing gcl_cmpcatch.o
Initializing gcl_cmpenv.o
Initializing gcl_cmpeval.o
Initializing gcl_cmpflet.o
Initializing gcl_cmpfun.o
Initializing gcl_cmpif.o
Initializing gcl_cmplabel.o
Initializing gcl_cmplam.o
Initializing gcl_cmplet.o
Initializing gcl_cmploc.o
Initializing gcl_cmpmap.o
Initializing gcl_cmpmulti.o
Initializing gcl_cmpspecial.o
Initializing gcl_cmptag.o
Initializing gcl_cmptop.o
Initializing gcl_cmpvar.o
Initializing gcl_cmpvs.o
Initializing gcl_cmpwt.o

Loading /usr/lib/gcl-2.6.7/unixport/../lsp/sys-proclaim.lisp
Finished loading /usr/lib/gcl-2.6.7/unixport/../lsp/sys-proclaim.lisp
Loading /usr/lib/gcl-2.6.7/unixport/../cmpnew/sys-proclaim.lisp
Finished loading /usr/lib/gcl-2.6.7/unixport/../cmpnew/sys-proclaim.lisp
Loading /usr/lib/gcl-2.6.7/unixport/../gcl-tk/tk-package.lsp
Finished loading /usr/lib/gcl-2.6.7/unixport/../gcl-tk/tk-package.lsp
Loading /usr/lib/gcl-2.6.7/unixport/../cmpnew/gcl_cmpmain.lsp
Warning: COMPILE-FILE is being redefined.
Warning: COMPILE is being redefined.
Warning: DISASSEMBLE is being redefined.
Finished loading /usr/lib/gcl-2.6.7/unixport/../cmpnew/gcl_cmpmain.lsp
Loading /usr/lib/gcl-2.6.7/unixport/../cmpnew/gcl_lfun_list.lsp
Finished loading /usr/lib/gcl-2.6.7/unixport/../cmpnew/gcl_lfun_list.lsp
Loading /usr/lib/gcl-2.6.7/unixport/../cmpnew/gcl_cmpopt.lsp
Finished loading /usr/lib/gcl-2.6.7/unixport/../cmpnew/gcl_cmpopt.lsp
Loading /usr/lib/gcl-2.6.7/unixport/../lsp/gcl_auto_new.lsp
Finished loading /usr/lib/gcl-2.6.7/unixport/../lsp/gcl_auto_new.lsp

Initializing /tmp/b.o
T

>
"bb"

>(by)
$ ./bb
GCL (GNU Common Lisp)  2.6.7 CLtL1    May  1 2005 21:25:42
Source License: LGPL(gcl,gmp), GPL(unexec,bfd)
Binary License:  GPL due to GPL'ed components: (READLINE BFD UNEXEC)
Modifications of this banner must retain notice of a compatible license
Dedicated to the memory of W. Schelter

Use (help) to get some basic information on how to use GCL.

>(setq a (make-array 10 :element-type 'long-float))

#(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0)

>(setq b (make-array 12 :element-type 'long-float))

#(0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0)

>(setf (aref a 3) 4.0)

4.0

>(setf (aref b 3) 2.5)

2.5

>(ddot a b)

10.0

>(by)
$ 
=============================================================================

BTW, if time permits, I'd like to get some sort of standardized blas
option into 2.7.0, as we use it here two quite extensively.

Take care,



> Any help will be much appreciated.
> 
> Thanks,
> 
> Jordan Rosenthal
> 
> PS I also noticed a gcl-users mailing list, but I didn't post there
> because it seems to have very little activity.  If the gcl-devel list
> is not the right place to post this question, just let me know and
> I'll move this discussion to the gcl-users list.
> 
> 
> _______________________________________________
> Gcl-devel mailing list
> address@hidden
> http://lists.gnu.org/mailman/listinfo/gcl-devel
> 
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah