cfd doesn't accept any more connections (1.6.3)

[Oliver Cook]

We are running cfd 1.6.3 on a FreeBSD server (3.2
STABLE).

After cfd has been running for about a week to 10 days
or so (sometimes more), cfd will stop accepting packets
from cfengine clients. cfd cannot be killed with a SIGTERM
but a SIGKILL has to be used to kill the process. It
will then restart normally and work for the next 7-10 days.

Here is a backtrace having attached to the 'stalled' process:
(I obscured the IP and host in frame 10 but apart from that it's
a verbatim backtrace)

#0  0x280f88ec in _thread_sys_select () from /usr/lib/libc_r.so.3
#1  0x280f7fa9 in _thread_kern_sched_state_unlock () from /usr/lib/libc_r.so.3
#2  0x280f7882 in _thread_kern_sched () from /usr/lib/libc_r.so.3
#3  0x280f7a3b in _thread_kern_sched_state () from /usr/lib/libc_r.so.3
#4  0x280ef89e in wait4 () from /usr/lib/libc_r.so.3
#5  0x280da27e in wait () from /usr/lib/libc_r.so.3
#6  0x806c353 in shell_cmd ()
#7  0x806bf1d in table_match ()
#8  0x806bd56 in hosts_access ()
#9  0x806bb0d in hosts_ctl ()
#10 0x804c941 in VerifyConnection (conn=0x80c5000,
    buf=0xbfbfaae2 "123.123.123.123 host.domain.net root 0") at cfd.c:1671
#11 0x804b693 in BusyWithConnection (conn=0x80c5000) at cfd.c:1065
#12 0x804b452 in HandleConnection (conn=0x80c5000) at cfd.c:964
#13 0x804b249 in SpawnConnection (sd_reply=2) at cfd.c:810
#14 0x804b051 in StartServer (argc=2, argv=0xbfbfdc8c) at cfd.c:705
#15 0x804a017 in main (argc=2, argv=0xbfbfdc8c) at cfd.c:121
#16 0x8049f6d in _start ()

Cfd is definately not sending any packets back to the client:

20:09:18.657744 host2.domain.net.1899 > 123.123.123.123.cfengine: S 
658280235:658280235(0) win 32120 <mss 1460,sackOK,timestamp +18880608 
0,nop,wscale 0> (DF)
20:09:35.502706 host2.domain.net.1903 > 123.123.123.123.cfengine: S 
715145577:715145577(0) win 32120 <mss 1460,sackOK,timestamp +18882292 
0,nop,wscale 0> (DF)
20:09:38.498039 host2.domain.net.1903 > 123.123.123.123.cfengine: S 
715145577:715145577(0) win 32120 <mss 1460,sackOK,timestamp +18882592 
0,nop,wscale 0> (DF)

I know the operating system on that server is quite old, but
I am retisent to say that's the problem! In any case, it's
nigh on impossible to take that machine out of service for
an upgrade.

Any help anyone can shed on the matter would be greatly
appreciated.

Cheers,

Ollie
-- 
Oliver Cook    Systems Administrator, ClaraNET
ollie@uk.clara.net               020 7903 3065