How to manage network buffers for incoming UDP packets?

This site uses cookies. By continuing to browse this site, you are agreeing to our Cookie Policy.

  • How to manage network buffers for incoming UDP packets?

    Hi,

    I observed a deadlock in by application because the IP-stack ran out of network buffers.

    My solution is to make send non-blocking (see below)
    but my question is if there is a better way to configure allocation strategy of network buffers.

    Situation:
    Assume a thread needs to receive and, from time to time, send UDP multicasts (for some kind of service announcement and discovery).
    Most of the time, this thread is blocked in a select() call waiting for incoming network packets.
    This select call is configured with a timeout, which, when fired causes the thread to send a UDP multicast by itself.

    I observed a situation, where, between the thread being woken-up from select due to timeout and the thread reaching the socket's send call,
    a lot of network packets arrive on the same socket (from other devices).
    Those incoming network packets seem to consume all available network buffers.
    This causes the send-call to block, for it can not allocate a network buffer to send out its message.
    This situation causes not only to deadlock this thread but renders the entire IP-stack unusable, for there is not a single network buffer available anymore.

    Could my observation be correct? Can this situation occur?
    Is there a way to deal with it?
    As a work-around, I configured the send to be non-blocking and skip sending out this single message.
    Can I limit number of network packets consumed by incoming UDP (multicasts) packets to ensure packets are available for send-calls?

    Thanks a lot
    Stefan

  • RE: How to manage network buffers for incoming UDP packets?

    Dear Stefan,

    stefan.waldschmidt wrote:

    Could my observation be correct? Can this situation occur?
    The short answer here is yes, if the PacketBuffer configuration is insufficient and/or does not match the configuration of the SocketBuffer.

    By default an UDP SocketBuffer is configured to hold up to 2048 bytes. This can be changed for a socket via setsockopt(SO_RCVBUF).

    The PacketBuffer configuration of the system should be prepared to handle the situation where the complete size of the SocketBuffer
    consumes PacketBuffers, for of ALL sockets that can be open at the same time.

    For UDP this is hard to calculate as each UDP message consumes one PacketBuffer. This means with 16 PacketBuffers in the System it
    would be possible to consume all PacketBuffers by 16 incoming 1 byte UDP messages.
    For TCP things work different. As TCP is a stream, new incoming data might get merged into existing packet buffers.

    With embOS/IP V3.04 we have added checks that prevent most of the deadlock situations that might occur due to small PacketBuffer
    configurations. However for UDP this is still tricky and hard to avoid.

    stefan.waldschmidt wrote:

    Is there a way to deal with it?
    As a work-around, I configured the send to be non-blocking and skip sending out this single message.
    This would have been our first suggestion as well. The stack itself is in "blocking state" as no more data can be received (one packet is kept free on Rx, to ensure sending a TCP retransmit or ACK is still possible). In your case I assume the UDP send() call is blocking as the only free PacketBuffer is not big enough for your data to send.

    stefan.waldschmidt wrote:

    Can I limit number of network packets consumed by incoming UDP (multicasts) packets to ensure packets are available for send-calls?
    I will check if we can add this for the next version. As explained above, a good PacketBuffer configuration for UDP is hard to estimate.
    Thank you for the suggestion.

    As soon as I have something ready for testing I will let you know by mail.

    Regards,
    Oliver
    Please read the forum rules before posting.

    Keep in mind, this is *not* a support forum.
    Our engineers will try to answer your questions between their projects if possible but this can be delayed by longer periods of time.
    Should you be entitled to support you can contact us via our support system: segger.com/ticket/
    Or you can contact us via e-mail.

  • Hi Oliver,

    thank you so much for your fast response.
    Your explanation really helps me a lot to better understand what is going on.

    And thanks for pointing out the SO_RCVBUF socket option I was not aware of.

    So we can either live with the "non-blocking workaround" (which seems to work fine)
    or do some math and fine-tune the buffer sizes, based on your explanations.

    Thanks again
    Stefan

  • Hi Stefan,

    But please be aware that SO_RCVBUF only means a limitation of the socket buffer in bytes.
    This will not prevent you from wasting packet buffers for like 1 byte UDP packets.

    We will check what we can do about an optional, additional NumPackets limitation for UDP/RAW Rx socket buffers.
    Stay tuned.

    Regards,
    Oliver
    Please read the forum rules before posting.

    Keep in mind, this is *not* a support forum.
    Our engineers will try to answer your questions between their projects if possible but this can be delayed by longer periods of time.
    Should you be entitled to support you can contact us via our support system: segger.com/ticket/
    Or you can contact us via e-mail.

  • RE: RE: How to manage network buffers for incoming UDP packets?

    SEGGER - Oliver wrote:

    Dear Stefan,

    stefan.waldschmidt wrote:

    Could my observation be correct? Can this situation occur?
    The short answer here is yes, if the PacketBuffer configuration is insufficient and/or does not match the configuration of the SocketBuffer.

    By default an UDP SocketBuffer is configured to hold up to 2048 bytes. This can be changed for a socket via setsockopt(SO_RCVBUF).

    The PacketBuffer configuration of the system should be prepared to handle the situation where the complete size of the SocketBuffer
    consumes PacketBuffers, for of ALL sockets that can be open at the same time.

    For UDP this is hard to calculate as each UDP message consumes one PacketBuffer. This means with 16 PacketBuffers in the System it
    would be possible to consume all PacketBuffers by 16 incoming 1 byte UDP messages.
    For TCP things work different. As TCP is a stream, new incoming data might get merged into existing packet buffers.

    With embOS/IP V3.04 we have added checks that prevent most of the deadlock situations that might occur due to small PacketBuffer
    configurations. However for UDP this is still tricky and hard to avoid.

    stefan.waldschmidt wrote:

    Is there a way to deal with it?
    As a work-around, I configured the send to be non-blocking and skip sending out this single message.
    This would have been our first suggestion as well. The stack itself is in "blocking state" as no more data can be received (one packet is kept free on Rx, to ensure sending a TCP retransmit or ACK is still possible). In your case I assume the UDP send() call is blocking as the only free PacketBuffer is not big enough for your data to send.

    stefan.waldschmidt wrote:

    Can I limit number of network packets consumed by incoming UDP (multicasts) packets to ensure packets are available for send-calls?
    I will check if we can add this for the next version. As explained above, a good PacketBuffer configuration for UDP is hard to estimate.
    Thank you for the suggestion.

    As soon as I have something ready for testing I will let you know by mail.

    Regards,
    Oliver

    SEGGER - Oliver wrote:

    I think this is the best answer for your query

    SEGGER - Oliver wrote: