The error that Riccardo got is complaining that shmget() can't allocate memory,
which sounds more like a resource leak. This actually has nothing to do with
X11 - the error message is generated in response to a system call failing, not
in response to an X11 event. The most likely cause of this is that we are
allocating a lot of shared memory segments and leaking them. Shared memory is
generally a more scarce resource than local memory. On FreeBSD, for example,
the default only allows 128 shared memory objects per process and 8192 pages of
shared memory. If we're leaking shared memory, it's very easy to go over this
limit.
In fact, the use of shared memory with the Cairo back end is currently
completely wrong - it may work, but it's a design that will give absolutely the
worst possible performance possible from Cairo, because we're implicitly
telling Cairo not to use hardware acceleration for drawing and then telling the
X server to perform a redundant copy if it wants to do hardware acceleration.
This is something that Opal will fix by not trying to make Cairo look like
libart, so I'm not sure if it's worth bothering chasing a short-term fix.