Network Buffers

Overview

Network buffers are a core concept of how the networking stack (as well as the Bluetooth stack) pass data around. The API for them is defined in include/zephyr/net_buf.h.

The data of a buffer is kept in a block of storage whose size is fixed when the buffer is allocated. The data does not have to start at the beginning of the storage: there can be free space before it, called the headroom, and after it, called the tailroom.

        block-beta
  %% The widest label per column sets the text size, so the long labels span several columns
  columns 16
  space:2 size<["size"]>(x):12 space:2
  space:2 headroom["headroom"]:3 len["len"]:5 tailroom["tailroom"]:4 space:2
  p0["__buf"]:4 p1["data"]:2 space:2 p2["data + len"]:4 p3["__buf + size"]:4

  classDef ptr fill:none,stroke:none
  classDef free stroke-dasharray:5,fill-opacity:0.3
  class p0,p1,p2,p3 ptr
  class headroom,tailroom free

Layout of a network buffer

Four fields of the buffer describe this layout:

data: Points to the start of the data.
len: Length of the data behind the data pointer.
size: Size of the storage, headroom and tailroom included.
__buf: Start of the storage. Not to be accessed directly: use the data pointer instead.

The headroom is returned by net_buf_headroom(), the tailroom by net_buf_tailroom(), and net_buf_tail() returns data + len, where the next byte added to the buffer goes. The room left for more data is the tailroom, not size minus len: size counts the headroom as well.

Creating buffers

A net_buf is allocated from a pool, which is defined first, with the macro of the pool’s type (see Pool Types). A net_buf_simple needs no pool (see Simple Buffers). A pool of fixed-size buffers is defined with:

NET_BUF_POOL_FIXED_DEFINE(pool_name, buf_count, buf_size, user_data_size, NULL);

The pool is a static variable, so if it’s needed to be exported to another module a separate pointer is needed.

Once the pool has been defined, buffers can be allocated from it with:

buf = net_buf_alloc(&pool_name, timeout);

There is no explicit initialization function for the pool or its buffers, rather this is done implicitly as net_buf_alloc() gets called.

If there is a need to reserve space in the buffer for protocol headers to be prepended later, it’s possible to reserve this headroom with:

net_buf_reserve(buf, headroom);

In addition to actual protocol data and generic parsing context, network buffers may also contain protocol-specific context, known as user data. The user data size is a property of the pool: every buffer of the pool has user_data_size bytes of it, returned by net_buf_user_data().

The last argument of the pool definition is an optional destroy callback. When the last reference to a buffer is released, the callback is called instead of returning the buffer to the pool, and it must eventually call net_buf_destroy() to return it.

The buffers have native support for being passed through k_fifo kernel objects. Use k_fifo_put() and k_fifo_get() to pass buffer from one thread to another.

Buffers can also be kept in a sys_slist_t through their node field. net_buf_slist_put() and net_buf_slist_get() append and remove a buffer under a spinlock, so that a list can be shared between threads and interrupt handlers as long as every access to it goes through these two functions. Use them instead of sys_slist_append() and sys_slist_get() for such a list. A buffer has a single node field, so it can be in only one FIFO or list at a time, however many references to it exist.

Pool Types

The pool type determines where the data of its buffers comes from:

Pool	Data storage	Data allocation
`NET_BUF_POOL_FIXED_DEFINE`	A static array with one chunk of `data_size` bytes per buffer.	Always the full chunk. Never waits.
`NET_BUF_POOL_VAR_DEFINE`	A heap of `data_size` bytes shared by the buffers of the pool.	The size requested for each buffer. Waits for heap memory until the allocation timeout expires.
`NET_BUF_POOL_VAR_ALIGN_DEFINE`	Same as `NET_BUF_POOL_VAR_DEFINE`.	Same as `NET_BUF_POOL_VAR_DEFINE`, with the start of the data aligned to `align` and the allocation rounded up to a multiple of it. A request for less than `align` bytes fails, unless it is for no data at all.
`NET_BUF_POOL_HEAP_DEFINE`	The system heap, through `k_malloc()`. Needs `CONFIG_HEAP_MEM_POOL_SIZE`.	The size requested for each buffer. Never waits: fails at once when the heap is exhausted.

In every pool type the buffers themselves come from a fixed array of buf_count entries, and an allocation waits for a free one until its timeout expires. The data is allocated after that, within what is left of the same timeout. An allocation from a heap pool can therefore return NULL even with K_FOREVER. Each allocation from a variable or heap pool also uses a few bytes of the heap for bookkeeping, which the size of a variable pool’s heap has to allow for.

net_buf_alloc() allocates a buffer with the fixed data size of its pool and is meant for fixed pools: from a variable or heap pool it returns a buffer without data storage. Use net_buf_alloc_len() with those. net_buf_alloc_with_data() works with every pool type: the buffer then uses storage given by the caller, and the pool allocates no data for it. The buffer starts with all of that storage as its data, so it has no tailroom, and releasing the buffer does not release the storage: the caller keeps it valid for as long as the buffer exists.

Allocation Timeouts

The timeout of an allocation is how long it may wait for a free buffer and, in a variable pool, for memory for the data. An interrupt handler cannot wait and has to use K_NO_WAIT.

A thread that waits with K_FOREVER on a pool whose buffers only it frees, directly or through work that it processes itself, never wakes up. Use a finite timeout in such a context and handle the NULL return.

Common Operations

The network buffer API provides some useful helpers for encoding and decoding data in the buffers. To fully understand these helpers it’s good to understand the basic names of operations used with them:

        block-beta
  columns 24
  space:2 push<["push"]>(left):4 space:10 add<["add"]>(right):4 space:4
  headroom["headroom"]:6 len["len"]:10 tailroom["tailroom"]:8
  space:6 pull<["pull"]>(right):4 space:2 remove<["remove"]>(left):4 space:8

  classDef free stroke-dasharray:5,fill-opacity:0.3
  class headroom,tailroom free

Where the four operations change the data of a buffer

Operation	Works at	Changes	Needs	Typical use
Add	End of the data	`len`	Enough tailroom	Encoding a message from its start
Remove	End of the data	`len`	Enough data	Decoding a trailer, such as a checksum
Push	Start of the data	`data` and `len`	Enough headroom	Prepending the header of a lower layer
Pull	Start of the data	`data` and `len`	Enough data	Decoding a header

Each operation has a function per type of data, for instance for Add:

net_buf_add_mem(): Copies a memory area.
net_buf_add_u8(): Adds an 8-bit value.
net_buf_add_le16(), net_buf_add_be16(): Add a 16-bit value in little-endian or big-endian byte order. Both byte orders also exist for 24, 32, 40, 48 and 64 bits.

Push, Pull and Remove have the same set, named net_buf_push_*(), net_buf_pull_*() and net_buf_remove_*(). The fixed-width variants of Pull and Remove return the value they pulled or removed. net_buf_add(), net_buf_push() and net_buf_pull() move the boundary by a length without copying anything, and return a pointer into the buffer:

struct foo_hdr *hdr = net_buf_push(buf, sizeof(*hdr));

hdr->opcode = FOO_OP_WRITE;
hdr->len = sys_cpu_to_le16(len);

Simple Buffers

net_buf is built around net_buf_simple, which has only the four fields of the buffer layout. What net_buf adds is what a buffer needs to be passed between contexts: the reference count, the pool that the buffer returns to, the link for FIFOs and lists, the fragment chain and the user data.

A net_buf_simple is enough for a buffer that one function or object uses on its own, such as a short message encoded on the stack. NET_BUF_SIMPLE_DEFINE defines one together with its storage, and NET_BUF_SIMPLE_DEFINE_STATIC a static one. Each of the add, remove, push and pull functions has a net_buf_simple_ variant:

NET_BUF_SIMPLE_DEFINE(msg, 8);

net_buf_simple_add_u8(&msg, opcode);
net_buf_simple_add_le16(&msg, handle);

A parser that hands a buffer to code that pulls data from it can save the position of the data first with net_buf_simple_save() and return to it afterwards with net_buf_simple_restore():

NET_BUF_SIMPLE_DEFINE(msg, 32);
struct net_buf_simple_state state;

net_buf_simple_add_mem(&msg, data, len);

net_buf_simple_save(&msg, &state);
err = parse_header(&msg);
net_buf_simple_restore(&msg, &state);

The saved state is the position of the data, not its content: data written into the buffer in between is not undone.

Fragments

Data that does not fit in one buffer, or that is put together from pieces, is kept in a chain of buffers linked through their frags field. The first buffer of the chain is its head.

        block-beta
  columns 17
  block:b0:4
    columns 4
    t0["head"]:4
    h0[" "] l0["len"]:2 r0[" "]
    f0["frags"]:4
  end
  a0<["&nbsp;"]>(right)
  block:b1:4
    columns 4
    t1["fragment"]:4
    h1[" "] l1["len"]:3
    f1["frags"]:4
  end
  a1<["&nbsp;"]>(right)
  block:b2:4
    columns 4
    t2["fragment"]:4
    l2["len"]:3 r2[" "]
    f2["frags"]:4
  end
  a2<["&nbsp;"]>(right)
  n["NULL"]:2

  classDef ptr fill:none,stroke:none
  classDef free stroke-dasharray:5,fill-opacity:0.3
  class t0,t1,t2,n ptr
  class h0,r0,h1,r2 free

A chain of three buffers

Each buffer of the chain has its own headroom and tailroom, and its len counts only its own data. net_buf_frags_len() returns the length of the data in the whole chain.

net_buf_frag_add() adds a fragment to the end of a chain, net_buf_frag_insert() inserts one after a given buffer, and net_buf_frag_del() removes one and releases the chain’s reference to it. Who owns the buffers of a chain is described in Ownership.

Some functions work on the data of a whole chain: net_buf_linearize() copies data out of it into a contiguous buffer, net_buf_append_bytes() adds data to its end and allocates more fragments as needed, and net_buf_data_match() compares data with its content. Each returns a length that can be less than requested. net_buf_skip() removes data from the start of a chain, releases the fragments it empties and returns what is left of the chain.

Ownership

A net_buf is reference counted, unlike a net_buf_simple. A newly allocated buffer has one reference, and the buffer returns to its pool when the last reference is released. Each reference has exactly one owner. The owner either moves the reference to a new owner, such as a FIFO, a list or a function that takes ownership of the buffer, or releases it. After the move the previous owner must no longer use the buffer, unless it holds another reference of its own, acquired with net_buf_ref().

Two helpers operate on the pointer that holds a reference, so that the previous owner is left with NULL instead of a pointer to a buffer it no longer owns. net_buf_take() moves the reference out of the pointer, and net_buf_drop() releases it. net_buf_drop() is meant for a pointer that outlives the release, such as a structure member, or one that may already be NULL. A local pointer that holds a reference as it goes out of scope only needs net_buf_unref().

A fragment chain is owned through its head. Each buffer in the chain owns a reference to the next fragment and releases it when the buffer itself is freed, so releasing the last reference to the head releases the chain up to the first fragment that is still referenced elsewhere. net_buf_frag_insert() takes ownership of the fragment it is given, and so does net_buf_frag_add() when the head is not NULL. With a NULL head, net_buf_frag_add() returns the fragment with a new reference and the caller keeps its own.

A function or callback that is given a buffer does one of the following with the reference, and its documentation should say which:

Borrows: The function may use the buffer until it returns, but may neither keep nor move it. To keep a borrowed buffer, it acquires a reference of its own with net_buf_ref() first.
Takes ownership: The reference moves to the function on every return, errors included.
Takes ownership on success: The reference moves only if the function succeeds. On error the caller still owns the buffer.

Putting a buffer in a FIFO or a list moves the reference to the queue. The receiving side may process and free the buffer even before k_fifo_put() returns, for instance when a higher-priority thread is waiting on the FIFO. Move the reference with net_buf_take() in the same expression:

k_fifo_put(&tx_queue, net_buf_take(&buf));

Do this even where buf goes out of scope right after the call: a plain pointer argument looks the same as one that is only borrowed, while net_buf_take() shows at the call site that the reference moves.

Passing net_buf_take(&buf) as an argument only works for calls that always take ownership, such as k_fifo_put(), net_buf_slist_put() and net_buf_frag_insert(). When a function that takes ownership only on success is given a plain pointer, the caller’s pointer is unchanged either way, and only the return value tells whether the buffer is still its own.

New functions that take ownership should therefore take a pointer to the caller’s buffer pointer, and move the reference out with net_buf_take() when they take it. The caller’s pointer is then NULL exactly when ownership has moved, and net_buf_drop() is correct after the call whatever the outcome:

int foo_send(struct foo *foo, struct net_buf **buf)
{
    if (!foo->ready) {
        return -EAGAIN;
    }

    k_fifo_put(&foo->tx_queue, net_buf_take(buf));

    return 0;
}

err = foo_send(foo, &buf);
if (err != 0) {
    LOG_WRN("Not sent (err %d)", err);
}

/* Releases the buffer only if foo_send() did not take it */
net_buf_drop(&buf);

Changing an existing function to this form breaks its callers, so it is best done when the function is reworked anyway.

Debugging

CONFIG_NET_BUF_LOG: Enables the logs of the library. At the warning log level or higher, one of them is a warning when an allocation with K_FOREVER has to wait for a buffer, which points at the deadlock described in Allocation Timeouts.
CONFIG_NET_BUF_POOL_USAGE: Tracks how many buffers of each pool are free and the most that were in use at the same time, returned by net_buf_get_available() and net_buf_get_max_used(). This helps to size a pool.
CONFIG_NET_BUF_HARDENING: Adds checks to the add, remove, push and pull functions that do not depend on assertions, which are disabled by default. An operation that is refused does not always report it to the caller. net_buf_is_valid() checks on demand that a buffer is consistent and still referenced.

Common Mistakes

Ownership: A buffer that is not released on an error path is lost to its pool, and the pool runs dry much later, far from the cause. A buffer that is released twice can return to its pool while another owner still uses it. Follow the rules in Ownership: every reference has one owner, a move to a queue or to a function that always takes ownership is written with net_buf_take(), and a stored reference is released with net_buf_drop().
Two threads on one buffer: The pool is protected against concurrent allocation and release, but the fields of a buffer are not: two threads must not change the data, len or frags of one buffer at the same time. Pass the buffer from one thread to the other through a queue, or let each hold its own reference and leave the buffer unchanged. Pulling data changes it too: a thread that parses a shared buffer pulls from a net_buf_simple of its own, set up with net_buf_simple_clone(), which points to the same data.
Headroom and tailroom: net_buf_push() needs headroom, which net_buf_reserve() sets aside before any data is added, and net_buf_add() needs tailroom: the room left for more data is net_buf_tailroom(), not size. Running out of either is checked only by assertions, which are disabled by default, or by CONFIG_NET_BUF_HARDENING. A push or a pull also moves the data pointer, so a copy of it taken before is stale afterwards, and nothing checks that.
Pool sizing: A pool shared by independent data flows, such as transmit and receive, lets one of them starve the other. Give each flow its own pool and size it from the number of buffers that are in use at the same time, measured with CONFIG_NET_BUF_POOL_USAGE (see Debugging).
User data: Every layer that holds a buffer can read and write its user data, and the user data changes hands together with the buffer. Do not assume that it is preserved when the buffer is passed on, unless the receiving API documents that. State that only one module needs can instead be kept in an array next to the pool, indexed with net_buf_id().

API Reference

Network Buffer Library