RE: using make to run binaries

[Dave Korn]

On 17 September 2007 15:22, andreas wrote:

>>> We have got a problem using 'make' to start binaries: we use 'make' not
>>> only for compilation but also for testing our compiled binaries on input
>>> data like this: # make output.data
>>> to create a line like this:
>>> # /path/to/binary -i input.data [more args] > output.data
>>> 
>>> The problem is that output.data differs from the output obtained by the
>>> very same command line when run manually.

> The program is called always with the absolute path like in the example
> above.

  OK, just checking that you did so in your manual commandline tests also.

>> 2.  How did you check the environment vars were the same?
> I compared the environment vars at the start of the program. I modified
> the program to output the current vars to cerr. Apart from three vars
> set by make (concerning the make level) there was no difference compared to
> manual launch. 

  Right, that seems conclusive.

>> Are you certain you're using the same shell as make uses? Dunno, how can I
>> check this? 

  Make will invoke your program using 'sh -c', unless you have set SHELL in
your environment, in which case it uses that.  So it depends on whether you
have SHELL defined, and whether 'sh' works out to be the same shell you use
for command-line testing.  (e.g. sh is bash on most linux systems, if you're
using ksh or zsh for testing it may well behave differently).

>> 3.  In what way do the output files differ?  Is it something the program is
>> supposed to do?
> We do predictions on a set of chemical compounds stored in the input
> file. The XML output reports those predictions. The output is then
> further processed by perl scripts. The difference is that the XML output
> created by manual launch contains more chemical information. We use a
> chemical library (openbabel) which is loaded as a shared library by our
> program so it could be that two different libraries are used. However,
> this can be ruled out as we have only one single version of the library
> on our host. Also, the problem could be reproduced on another host.
> 
>> 4.  (I'm staking my 50 qatloos on this one:)  Are you in a different
>> working 
>> directory when you try the command-line test as compared to when you run it
>> under make?  Are there different input.data files in those different
>> directories?  Are you looking by accident at the wrong output.data file?
> Sorry, but this is also not an option ;-) We had this now literally
> dozens of times, checking it over and over again and it differs
> *always*.

  Ok, forgive me for asking, I just wanted to be really sure that all the
possibilities for simple errors were checked and excluded.  Unfortunately,
they are, which only leaves us with the complicated errors.

  The thing is, make doesn't do anything particularly bizarre when it launches
an executable.  It takes one shortcut: if it's a simple command-line, with no
redirection or pipes or other complex shell features, it just fork-and-execs
the new application directly.  However, you've got an output redirection
there, which triggers make to do things "the slow way"; that is, it builds an
actual command line (by prefixing "sh -c" to your commands from the make rule)
and invokes it with a call to system().

  So, if you're really sure you're invoking the same executable in both cases,
and that the environment is the same in both cases, and the working directory,
and the command-line options, and the input files, you have a very tricky
problem to solve.  The situation smells like undefined behaviour to me; if
you're using an uninitialised variable or in some other way depending on
random memory contents, that could trigger the different behaviour in your
program.  Your description of the problem is a bit vague, but by "more
chemical information", I'm guessing the library is either returning more
records, or more data fields in each record.  You might yet be able to find
out what's going on by putting strace-alike wrappers around the calls to the
library functions, see if you can verify that the program is driving the
library in the same way with the same calls to the same functions passing the
same arguments in each case.


    cheers,
      DaveK
-- 
Can't think of a witty .sigline today....