I am expecting a solution that works on both OS X and GNUstep.
This is an implementation detail which can differ on OS X and GNUstep. As far as I understand it, it's only needed inside CoreBase/Core Foundation. Why would you need to even determine if something is a small pointer in another library?
Luboš can correct me if I'm wrong, but you don't need to know whether the pointer is a "small object" outside very small amount of runtime code and toll-free bridging-related code, which will be implemented in core GNUstep libraries; and for any binary compatibility you will not and should not need to know this implementation detail about the kind of object that is stored in a pointer.