Problems with library preloading

[Manuel Arriaga]

Hi everyone,

[I apologize if this isn't strictly GNU libc-related, but please consider 
offering some helpful advice. That way, the GPLed code I wrote will be more 
useful to everybody.]


I am the author of libtrash (http://www.m-arriaga.net/software/libtrash), a 
shared library which is meant to be preloaded. It overrides a set of GNU libc 
functions.

I have problems with some apps; when they run, the original function in GNU 
libc is invoked, rather than the one I define in my code. Can you explain to 
me why this might happen?  Please consider the following example:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <stdio.h>
#include <stdarg.h>
#include <fcntl.h>
#include <sys/types.h>


int open(const char *path, int flags, ...)
{
   int (*real_open) (const char*, int, ...);

   printf("entering open(): %s\n", path);

   real_open = dlsym(RTLD_NEXT, "open");
   
   if (dlerror())
     return -1;
   
   if (flags & O_CREAT)
     {
        va_list arg_list;

        mode_t mode;

        va_start(arg_list, flags);

        mode = va_arg(arg_list, mode_t);

        va_end(arg_list);
        
        return real_open(path, flags, mode);
     }
   else
     {
        return real_open(path, flags);
     }
}

If you compile this code with the command

$ gcc -Wall -W (source file) -nostartfiles -shared -fPIC 
-Wl,-soname,libopen.so.0 -o libopen.so.0.0 -ldl

and then preload it when running most apps, all calls to open() are "logged" 
to stdout. Yet, if you run, e.g.

$ LD_PRELOAD=(absolute path to libopen.so.0.0) cp a.txt b.txt

, and although strace shows that cp calls 

open("a.txt", O_RDONLY|0x8000)              = 4
open("b.txt", O_WRONLY|O_TRUNC|0x8000)      = 5

those two calls aren't logged to stdout. "My" open() isn't being invoked.

So my question is: why aren't all calls to open() being redirected to the 
preloaded object? Why does this redirection only work in some cases?

Best regards,

Manuel Arriaga