Zero copy in libxtc

How to move bytes through messages, sockets, TLS, files and devices without the library copying them -- and what it still copies, and why.

---

Zero copy in libxtc

1. Zero copy in libxtc
   1. Messages between processes
   2. Sockets and framing
   3. TLS
   4. Files and block devices
   5. Accelerators
   6. What libxtc still copies, and why
   7. Alternatives not chosen

---

A runtime that sits between your code and the kernel has many chances to copy your data: into a message envelope, out of it again, into a framing buffer, into a TLS record, into an I/O bounce buffer. libxtc’s rule is that used right, it does not force a copy on any data path. Where a convenient API copies (because it takes a const buffer that stays yours), there is a sibling that moves or borrows the buffer instead. This page lists, path by path, how to move bytes without a copy, and what the library still copies and why.

“Copy” here means a user-space memcpy of the payload. The kernel’s copy between a user buffer and a socket is the price of a socket; the only ways around it are kernel TLS + sendfile (below) and, for the socket itself, MSG_ZEROCOPY-style APIs that libxtc does not wrap.

Messages between processes

xtc_send(to, data, size) takes a const buffer that stays the caller’s, so it must copy. Small messages (up to 256 bytes) travel inline in a pooled envelope and are copied out again at xtc_recv; larger ones are copied once, into a block that xtc_recv hands over as is.

xtc_send_owned(to, buf, size) moves a buffer instead: zero copies at any size, same loop or across threads. The buffer must come from libxtc’s allocator (xtc_malloc, or a block libxtc gave you to free with xtc_free, such as one from xtc_recv), because the receiver frees it with xtc_free. On XTC_OK it belongs to the receiver; on any error it is still yours, unchanged – free it or retry, the same backpressure contract as xtc_send.

#include <stdio.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/uio.h>
#include <unistd.h>

#include "xtc.h"
#include "xtc_loop.h"
#include "xtc_net.h"
#include "xtc_proc.h"

#define SZ (64 * 1024)

static void *g_sent;      /* the producer's pointer, for the check */
static int   g_ok;

static void
consumer(void *arg)
{
	void *msg = NULL;
	size_t n = 0;
	(void)arg;
	if (xtc_recv(&msg, &n, -1) != XTC_OK)
		return;
	/* Same block the producer filled: it was moved, not copied. */
	g_ok = (msg == g_sent && n == SZ && ((char *)msg)[SZ - 1] == 'z');
	xtc_free(msg);                    /* the consumer owns it now */
}

static void
producer(void *arg)
{
	xtc_pid_t to = *(xtc_pid_t *)arg;
	char *buf = xtc_malloc(SZ);       /* must come from libxtc's allocator */
	if (buf == NULL)
		return;
	memset(buf, 'z', SZ);
	g_sent = buf;
	if (xtc_send_owned(to, buf, SZ) != XTC_OK)
		xtc_free(buf);                /* on ANY error it is still ours */
	/* on XTC_OK: do not touch buf again */
}

int
main(void)
{
	xtc_loop_t *loop = NULL;
	xtc_pid_t cons, prod;
	struct iovec iov[3];
	void *frame = NULL;
	size_t flen = 0;
	int sv[2], frame_ok;

	/* Part 1: zero-copy message. */
	if (xtc_loop_init(&loop) != XTC_OK)
		return 1;
	if (xtc_proc_spawn(loop, consumer, NULL, NULL, &cons) != XTC_OK ||
	    xtc_proc_spawn(loop, producer, &cons, NULL, &prod) != XTC_OK)
		return 1;
	(void)xtc_loop_run(loop);
	(void)xtc_loop_fini(loop);

	/* Part 2: one frame from three pieces, no assembly copy. */
	if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) != 0)
		return 1;
	iov[0].iov_base = (void *)"HDR:";
	iov[0].iov_len = 4;
	iov[1].iov_base = (void *)"body bytes";
	iov[1].iov_len = 10;
	iov[2].iov_base = (void *)":END";
	iov[2].iov_len = 4;
	frame_ok = xtc_net_send_framev(sv[0], iov, 3) == XTC_OK &&
	    xtc_net_recv_frame(sv[1], &frame, &flen, 0, 1000000000LL) == XTC_OK &&
	    flen == 18 && memcmp(frame, "HDR:body bytes:END", 18) == 0;
	xtc_free(frame);
	close(sv[0]);
	close(sv[1]);

	printf("moved without a copy: %s; framev round trip: %s\n",
	    g_ok ? "yes" : "NO", frame_ok ? "yes" : "NO");
	return g_ok && frame_ok ? 0 : 1;
}

Tested source: docs/_includes/snippets/17_zero_copy_msg.c

Forwarding is free: a block from xtc_recv can go straight back out with xtc_send_owned. A received network frame (xtc_net_recv_frame) is an xtc_free-able block too, so a pump that reads frames and hands them to a process moves them without a copy – libxtc’s own cross-process child pump does exactly that.

Measured (one loop, producer writing every byte, 16-vCPU x86-64 host): moving instead of copying is 4.2x the message rate at 4 KiB, 10x at 64 KiB and 7.9x at 1 MiB; at 64 bytes the two are the same, so keep xtc_send for small messages.

Channels (xtc_chan_*) carry void *: they never copy. The lifetime of what the pointer points at is yours.

gen_server / gen_statem calls (xtc_svr_call, xtc_fsm_call) copy each direction once: the request (or reply) and its routing header must become one message, and the payload you pass is const. The frame is then moved, and the caller is handed the very block the server built. (Before the zero-copy work a call round trip copied the payload seven times.)

Process groups (xtc_pg_send) send one copy per member: a fan-out of a const buffer. Zero-copy fan-out would need shared, reference-counted, immutable messages with their own release call – deliberately not added; if you fan out large payloads, send each member an xtc_send_owned block, or share one immutable buffer by pointer over a channel and count references yourself.

Sockets and framing

xtc_net_send_frame(fd, buf, len) writes the 4-byte length prefix and the payload in one sendmsg from your buffer. To send a frame built from several pieces (a header struct, a body that lives elsewhere, a trailer), use xtc_net_send_framev(fd, iov, n): up to 63 pieces, one frame on the wire, no assembly buffer.

xtc_net_recv_frame reads the prefix, allocates the frame once, and receives the payload straight into it; the block is yours (xtc_free) and can be moved on with xtc_send_owned. UDP (xtc_net_udp_*) and the credentials calls take your buffers as they are.

TLS

By default, TLS is done in user space: your plaintext is copied into the TLS library’s record buffer, encrypted, and written to the socket.

xtc_tls_enable_ktls(tls), called before the handshake on an fd-mode connection, asks for kernel TLS (Linux tls.ko, FreeBSD KTLS; OpenSSL backend). After the handshake xtc_tls_ktls_active(tls) says whether the kernel took the transmit (XTC_TLS_KTLS_TX) and receive (XTC_TLS_KTLS_RX) sides; if it declined (module not loaded, a cipher it does not do), the connection is an ordinary user-space TLS connection and nothing else changes.

With kernel TX active, xtc_tls_sendfile(tls, fd, off, len, &n) sends a file region encrypted by the kernel; the file’s pages never enter user space. Without it the call returns XTC_E_NOSYS, which is your cue to pread + xtc_tls_write:

#include <arpa/inet.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <poll.h>
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <unistd.h>

#include "xtc.h"
#include "xtc_net.h"
#include "xtc_tls.h"

#define FILE_SZ (1024 * 1024)

static int g_cli_fd;
static size_t g_got;

static void
wait_io(xtc_tls_t *t, int fd)
{
	struct pollfd p = { fd, xtc_tls_wants_write(t) ? POLLOUT : POLLIN, 0 };
	(void)poll(&p, 1, 5000);
}

/* A plain user-space TLS client that reads everything. */
static void *
client(void *arg)
{
	xtc_tls_opts_t o = { 0 };
	xtc_tls_ctx_t *ctx = NULL;
	xtc_tls_t *t = NULL;
	char buf[65536];
	int rc;
	(void)arg;
	o.verify_peer_mode = XTC_TLS_VERIFY_NONE;   /* self-signed demo cert */
	if (xtc_tls_ctx_create(XTC_TLS_CLIENT, &o, &ctx) != XTC_OK ||
	    xtc_tls_create(ctx, g_cli_fd, &t) != XTC_OK)
		return NULL;
	while ((rc = xtc_tls_handshake(t)) == XTC_E_AGAIN)
		wait_io(t, g_cli_fd);
	while (rc == XTC_OK && g_got < FILE_SZ) {
		size_t n = 0;
		rc = xtc_tls_read(t, buf, sizeof buf, &n);
		if (rc == XTC_OK) {
			g_got += n;
		} else if (rc == XTC_E_AGAIN) {
			wait_io(t, g_cli_fd);
			rc = XTC_OK;
		}
	}
	xtc_tls_destroy(t);
	xtc_tls_ctx_destroy(ctx);
	return NULL;
}

/* Send len bytes of file fd over tls: zero copy when the kernel holds
 * the transmit side, read + write otherwise. */
static int
serve_file(xtc_tls_t *tls, int sock, int fd, size_t len)
{
	char buf[16384];
	size_t off = 0;
	while (off < len) {
		size_t n = 0;
		int rc = xtc_tls_sendfile(tls, fd, (int64_t)off, len - off, &n);
		if (rc == XTC_E_NOSYS) {             /* no kernel TX: fall back */
			ssize_t r = pread(fd, buf, sizeof buf, (off_t)off);
			if (r <= 0)
				return XTC_E_IO;
			rc = xtc_tls_write(tls, buf, (size_t)r, &n);
		}
		if (rc == XTC_OK)
			off += n;
		else if (rc == XTC_E_AGAIN)
			wait_io(tls, sock);
		else
			return rc;
	}
	return XTC_OK;
}

int
main(void)
{
	char cert[] = "/tmp/xtc-snip18-cert-XXXXXX", key[64], path[64];
	xtc_tls_opts_t o = { 0 };
	xtc_tls_ctx_t *ctx = NULL;
	xtc_tls_t *tls = NULL;
	struct sockaddr_in sa;
	socklen_t sl = sizeof sa;
	pthread_t th;
	char cmd[512], *data;
	int lfd, sv0, fd, rc, kt;

	if (mkstemp(cert) < 0)
		return 1;
	(void)snprintf(key, sizeof key, "%s.key", cert);
	(void)snprintf(path, sizeof path, "%s.dat", cert);
	(void)snprintf(cmd, sizeof cmd, "openssl req -x509 -newkey rsa:2048 "
	    "-nodes -days 1 -subj /CN=localhost -keyout %s -out %s "
	    ">/dev/null 2>&1", key, cert);
	o.cert_file = cert;
	o.key_file = key;
	if (system(cmd) != 0 ||
	    xtc_tls_ctx_create(XTC_TLS_SERVER, &o, &ctx) != XTC_OK) {
		printf("no openssl CLI or no TLS in this build: skipped\n");
		(void)unlink(cert);
		(void)unlink(key);
		return 0;
	}

	/* The file to serve. */
	data = malloc(FILE_SZ);
	fd = open(path, O_RDWR | O_CREAT | O_TRUNC, 0600);
	if (data == NULL || fd < 0)
		return 1;
	memset(data, 'k', FILE_SZ);
	if (write(fd, data, FILE_SZ) != FILE_SZ)
		return 1;

	/* A loopback TCP connection (kernel TLS needs TCP). */
	memset(&sa, 0, sizeof sa);
	sa.sin_family = AF_INET;
	sa.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
	lfd = socket(AF_INET, SOCK_STREAM, 0);
	g_cli_fd = socket(AF_INET, SOCK_STREAM, 0);
	if (bind(lfd, (struct sockaddr *)&sa, sizeof sa) != 0 ||
	    listen(lfd, 1) != 0 ||
	    getsockname(lfd, (struct sockaddr *)&sa, &sl) != 0 ||
	    connect(g_cli_fd, (struct sockaddr *)&sa, sizeof sa) != 0 ||
	    (sv0 = accept(lfd, NULL, NULL)) < 0)
		return 1;
	close(lfd);
	(void)xtc_net_setnonblock(sv0);
	(void)xtc_net_setnonblock(g_cli_fd);

	/* Server: ask for kernel TLS, then handshake. */
	if (xtc_tls_create(ctx, sv0, &tls) != XTC_OK)
		return 1;
	rc = xtc_tls_enable_ktls(tls);           /* XTC_E_NOSYS: not here */
	(void)rc;
	if (pthread_create(&th, NULL, client, NULL) != 0)
		return 1;
	while ((rc = xtc_tls_handshake(tls)) == XTC_E_AGAIN)
		wait_io(tls, sv0);
	kt = xtc_tls_ktls_active(tls) & XTC_TLS_KTLS_TX;
	if (rc == XTC_OK)
		rc = serve_file(tls, sv0, fd, FILE_SZ);
	(void)pthread_join(th, NULL);

	printf("served %zu bytes %s\n", g_got, kt ?
	    "with xtc_tls_sendfile (kernel TLS, zero copy)" :
	    "with pread + xtc_tls_write (user-space TLS)");
	xtc_tls_destroy(tls);
	xtc_tls_ctx_destroy(ctx);
	xtc_net_close(sv0);                      /* not close(2): see above */
	close(g_cli_fd);
	close(fd);
	free(data);
	(void)unlink(path);
	(void)unlink(cert);
	(void)unlink(key);
	return rc == XTC_OK && g_got == FILE_SZ ? 0 : 1;
}

Tested source: docs/_includes/snippets/18_tls_sendfile.c

Measured on a 32-vCPU x86-64 host (Linux 6.12, OpenSSL 3.5.7, AES-GCM, loopback, sender and receiver on two pinned cores), serving a file:

mode	MiB/s	sender CPU s/GiB	copy share of sender CPU
user-space TLS, pread + xtc_tls_write	1887	0.543	37.8%
kernel TLS, pread + xtc_tls_write	1345	0.760	20.2%
kernel TLS, xtc_tls_sendfile	2105	0.485	3.4%

Note the middle row: kernel TLS alone was slower here. The kernel’s AES-GCM cost more than OpenSSL’s, and the copy from your buffer into the kernel remains. Enable kernel TLS for connections that mostly serve files with xtc_tls_sendfile, not as a blanket default, and measure on your hardware.

Two rules for a kernel-TLS connection. Close its socket with xtc_net_close, not close(2): on Linux the kernel flushes a queued record at close and, if the peer has gone, raises SIGPIPE in the closing thread; xtc_net_close holds that off. And a write to a peer that has gone returns XTC_E_IO even though the kernel itself reports the bytes as accepted.

A connection on a custom transport (xtc_tls_create_transport) has no socket to hand the kernel, so it always does TLS in user space.

Files and block devices

xtc_aio_pread / pwrite / preadv / pwritev pass your buffer straight to the kernel (an io_uring SQE, a syscall on the blocking pool, or the simulator’s inline call): no bounce buffer. For O_DIRECT, the rules for your buffer’s address, file offset and length come from xtc_fs_dio_align(fd, &mem, &off, &len), and xtc_fs_dio_alloc gives you a buffer that meets them. xtc_bdev_pread / pwrite likewise.

libxtc does not (yet) expose io_uring registered buffers or SEND_ZC; the payload is never copied by libxtc either way.

Accelerators

xtc_batch stores only the void * you submit; xtc_accel_wait_fence waits on a fence or sync_file fd; xtc_accel_run_blocking runs your function off the loop. libxtc never touches device memory, so how bytes reach a GPU (pinned host memory, mapped or unified memory, DMA-BUF) is entirely the driver’s and your choice. The batching examples show both styles: one reads results back with a copy (clEnqueueReadBuffer), another maps the buffer.

What libxtc still copies, and why

what	copies	why
xtc_send <= 256 B	2 (in, out)	inline pooled envelope; at this size the copy costs about the same as moving (measured)
xtc_send > 256 B	1	the buffer is const and stays yours; use xtc_send_owned
xtc_svr_call / xtc_fsm_call	1 per direction	header + const payload become one message
xtc_pg_send	1 per member	fan-out of a const buffer
xtc_log_write	1	formatting is the copy
user-space TLS	record copy + crypto	use kernel TLS + xtc_tls_sendfile where it pays

The examples follow the same rules. examples/07_kaka moves produced records into the partition with xtc_send_owned. examples/08_tnt echoes with xtc_tnt_recv_take (keep the reactor’s receive buffer) and xtc_tnt_io_send_owned (send it, the runtime frees it after), so the echoed bytes are never copied into the Isolate. The copies left in the examples are their own choices: rexis copies a value into its reply buffer so the reply stays valid after the read lock is dropped; the sqlxtc WAL copies records into its ring because the ring is the group commit (many committers’ records leave in one write).

Alternatives not chosen

• Reference-counted messages for every send. It would make fan-out free but add an atomic per message and a release call that is not xtc_free to every receiver – a new receive contract for one use.
• Receive into a caller buffer (xtc_recv_into). The envelope for a large message already is the receiver’s buffer; a receive-into would add the copy back.
• Kernel TLS on by default. Measured slower for plain writes on a host whose CPU has fast user-space AES; it pays for sendfile.