Re: [Gluster-devel] libPhenom

[Jeff Darcy]

On 09/23/2013 04:51 PM, Jay Vyas wrote:
> It makes me wonder where the event queuing and message passing is happening
> in gluster as it stands....

We don't really do message passing except between processes.  Most of that
lives in the following places in your local source tree:

        rpc/transport (lowest level)
        rpc/rpc-lib (approximately session layer)
        xlators/protocol/client
        xlators/protocol/server

We don't generally rely on event queuing either, though some translators (e.g.
io-cache) do it internally.  The system is really call-based rather than event-
or message-based, so what we do is more similar to multi-level method
dispatch/delegation than anything else.  Each user request is represented by a
call stack (call_stack_t in .../libglusterfs/src/stack.h) which might be
created in one of several places:

        libgfapi
        protocol/fuse translator (client side)
        protocol/server translator
        nfs/server translator

These are then passed through translators using the STACK_WIND macro and the
"fops" dispatch table (equivalent to a C++ vtbl) in each translator, generating
stack frames (call_frame_t).  Ultimately they'll hit one of two translators
which do something other than delegate or dispatch:

        protocol/client to send a request to the server
        storage/posix to send a request to the local I/O subsystem on a server

After that the request "unwinds" using the STACK_UNWIND macro which invokes
callbacks (continuations) at each level it went through before, and eventually
gets destroyed when it comes back to the translator (or libgfapi) which created 
it.

There are many other details and special cases, but that's the gist of it.