Re: GENERIC: error receiving data -- Resource temporarily unavailable

[Jan-Henrik Haukeland]

Jesse Guardiani <address@hidden> writes:

> Howdy list,
>
> I just started using Monit last Friday. VERY cool little
> program. I'm running Monit 4.2 (without HTTP) on:
>
[..]
> I am using monit to monitor ClamAV's clamd daemon. It has
> a bad habit of:
>
> a.) consuming high resources for no good reason
> b.) crashing (which I fix quickly via daemontools)
> c.) hanging for no good reason. (This is really what I need monit for.)
>
> Anyway, today I saw this in my log:
>
> --------------------------------------------------------
> Apr  5 08:25:37 chortos monit[14259]: GENERIC: error receiving data -- 
> Resource temporarily unavailable
> Apr  5 08:25:37 chortos monit[14259]: Event: 'clamd' failed protocol test 
> [generic] at UNIX[/tmp/clamd].
[..]
> Apr  5 08:56:31 chortos monit[14259]: Event: loadavg(1min) of 10.2 matches 
> resource limit [loadavg(1min)>10.0]
> --------------------------------------------------------
>
> What does "GENERIC: error receiving data -- Resource temporarily unavailable" 
> mean?

Generic is the protocol test name used when you specify a send/except
sequence. "Resource temporarily unavailable" means that monit cannot
open a socket and/or read/write on a socket. There are two things you
can do: 

1. Increase the socket timeout (using the with TIMEOUT x SECONDS
statement in a connection test, see the manual)

2. Try to intercept clamd earlier by testing CPU usage before it goes
bananas on your system.

I don't know about FreeBSD, but on Linux and Solaris if you have a
load around 10 you have a system under considerably stress and
services may start to fail, such as open a socket connection.

I think I would use the following in the monit control file. I.e. have
a shorter poll cycle to be able to follow clamd closer and check the
CPU usage of clamd as well.

set daemon 30 
[...]
if cpu is greater than 80% for 2 cycles then restart


> And, more importantly, why did that message at 08:26 not trigger a
> restart event? 

Good question. This may be caused by the high load, but it should be
investigated. The new event model added to monit 4.3 beta, may also
handle this better.

-- 
Jan-Henrik Haukeland