[lwip-users] pbuf misuse or snd_queuelen corruption.

[Andrew Foster]

Hello,

 

As a foreword, I have an LWIP 1.3.2 /FreeRTOS combo running on a Stellaris series MCU from Luminary. I’m using a pretty common 2 thread model where there is a blocking low level input thread that waits for a signal from the xMACInterruptSemaphore.  From this thread, the etharp, ethrarp and tcp_ip inputs are handed off. Additionally, there is the standard tcpip_thread, or “main lwIP thread” running.

 

That being said, my problem arises when trying to use the low level tcp api from context of the registered tcpip_thread. I’m trying to provide 4 simultaneous tcp connections but having trouble with pbuf usage. I’ve tried both flavors of tcp_write with copy set TRUE and FALSE, each presents its own problem.

 

With copy set to TRUE, I find myself with circular references in:

 

pbuf_cat(struct pbuf *h, struct pbuf *t)

 

This happens while trying to find the last pbuf in the chain. It appears that the next pointer gets set to the current pbuf, resulting in a circular reference. Generally, this happens after trying to transmit data rapidly. Could this possibly be due to the multiple thread model described above?

 

With copy set to FALSE and MEMP_NUM_PBUF == 1 I get the best results:

 

Since I have a static location for outgoing data, this provides a faster operation and less lwIP memory pools. However, if I queue data rapidly, I get a memerr due to lack of PBUF_ROM. This inevitably causes a tcp_write error and renders all open tcp connections useless. Each tcp_write after, regardless of delay, causes the same error. Closing the connection does not solve the consecutive error problem.

 

Since this was my best attempt, I tried to increase the size of MEMP_NUM_PBUF to correct the memerr. This produces a different error resulting in the snd_queuelen for the connected TCP PCB getting decremented beyond the actual amount of tcp segments pending.

 

This happens in tcp_receive when

        LWIP_DEBUGF(TCP_QLEN_DEBUG, ("tcp_receive: queuelen %"U16_F" ... ", (u16_t)pcb->snd_queuelen));

        LWIP_ASSERT("pcb->snd_queuelen >= pbuf_clen(next->p)", (pcb->snd_queuelen >= pbuf_clen(next->p)));

        pcb->snd_queuelen -= pbuf_clen(next->p);

 

I’m somewhat stuck at this point. Could you provide any suggestions of things to try or potential design flaws based on my previous description? I’ve also included some relative lwipopts below for reference. Any ideas would be greatly appreciated.

 

LWIP_OPTS:

//-----------------------------------------------------------------------------

//Internal Memory Pool Sizes

//-----------------------------------------------------------------------------

#define MEMP_NUM_PBUF                  1      

#define MEMP_NUM_TCP_SEG            16

 

//-----------------------------------------------------------------------------

// TCP options

//-----------------------------------------------------------------------------

#define LWIP_TCP                     1

#define TCP_TTL                        255

#define TCP_WND                      8448     // (default 2048). TCP receive window.

#define TCP_MAXRTX                 12       // Max number of retransmissions of data segments.

#define TCP_SYNMAXRTX           4        // Max number of retransmissions of SYN segments.

#define TCP_QUEUE_OOSEQ     1        // Controls if TCP should queue segments that arrive out of order. Define to 0 if your device is low on memory.

#define TCP_MSS                       1200     // (default 128) TCP Maximum segment size.

#define TCP_SND_BUF                3072     // (default is 256) TCP sender buffer space (bytes).

#define TCP_SND_QUEUELEN     (MEMP_NUM_TCP_SEG) //Stellaris used MEMP_NUM_TCP_SEG (or 8).

 

//#define TCP_CALCULATE_EFF_SEND_MSS     1

//#define TCP_SNDLOWAT                                 (TCP_SND_BUF/2)

//#define TCP_LISTEN_BACKLOG                       0

//#define TCP_DEFAULT_LISTEN_BACKLOG       0xff

 

//-----------------------------------------------------------------------------

// Pbuf options

//-----------------------------------------------------------------------------

#define PBUF_LINK_HLEN              16       // (default is 14) The number of bytes that should be allocated for a link level header.

#define PBUF_POOL_BUFSIZE       192      // The size of each pbuf in the pbuf pool.

#define ETH_PAD_SIZE                   2        // (default 0)

 

//-----------------------------------------------------------------------------

// Platform specific locking

//-----------------------------------------------------------------------------

#define SYS_LIGHTWEIGHT_PROT           1        // (default 0)

#define NO_SYS                                      0

 

Thanks,

Andrew Foster