Blocking work and I/O

Files, sockets, timers, and blocking C libraries -- without ever stalling the loop.

---
  1. 1. Suspending I/O the loop already understands
  2. 2. Async file I/O
  3. 3. The blocking pool: escaping to a thread, on purpose
  4. 4. Attached compute: GPUs and NPUs
  5. Scaling across cores: the executor
  6. The other direction: bridging INTO the runtime
  7. Determinism: testing the whole thing
  8. What you have learned

The event loop’s one rule is: never block the thread that runs it. A fiber that calls a slow synchronous function – read() on a cold file, a CPU-bound compression pass, a legacy C library that does its own blocking I/O – stalls every other fiber on that loop until it returns. This chapter is about the three ways libxtc lets you do slow work without breaking that rule.

1. Suspending I/O the loop already understands

For sockets and timers, libxtc suspends the fiber and lets the OS poller wake it. You do not manage readiness yourself:

  • Timers: xtc_proc_sleep(ns) parks the fiber; the loop wakes it when the timer fires (see chapter 2).
  • Sockets: the xtc_net_* API (xtc_net(3)) does non-blocking connect / accept / read / write and suspends the fiber until the socket is ready, driven by io_uring / epoll / kqueue / IOCP under the hood.
  • Arbitrary fds: xtc_proc_wait_fd(fd, events, timeout, &revents) parks the fiber until fd is readable/writable – the building block for wrapping any fd-based protocol.

2. Async file I/O

File reads and writes do not have a clean readiness model on every OS, so libxtc presents one portable async file API, xtc_aio_* (xtc_aio(3)): xtc_aio_pread, xtc_aio_pwrite, xtc_aio_fsync, xtc_aio_fdatasync. Where the platform has a native completion mechanism (io_uring on Linux, IOCP on Windows, kqueue AIO on the BSDs, event-port AIO on illumos) the operation completes with zero extra threads. Everywhere else it transparently falls back to the blocking pool (below) – so you write storage code once, as if async file I/O is always available, and it is always correct.

3. The blocking pool: escaping to a thread, on purpose

When you must call something that will block and has no async form – a third-party library, a CPU-bound kernel – hand it to the blocking pool with xtc_blocking_run(fn, arg, &result) (xtc_blocking(3)). libxtc runs fn(arg) on a dedicated worker thread, suspends the calling fiber (so the loop keeps serving everyone else), and resumes the fiber with the result when the worker finishes. The blocking is real – but it happens on a pool thread, not the loop thread.

/* inside a fiber: compute a slow hash without stalling the loop */
int result;
xtc_blocking_run(hash_a_big_buffer, buf, &result);
/* the fiber resumes here once the pool thread is done */

Why an explicit pool call instead of auto-detecting blocking? A library cannot know which of your function calls will block. Making the escape hatch explicit means the one place a thread hop happens is visible in the code and in a profile – you can see, and budget, every departure from the single-threaded model. It is the same philosophy as unsafe in Rust: not forbidden, but marked.

4. Attached compute: GPUs and NPUs

An accelerator – a GPU, or an NPU (the neural/vision engine on a modern laptop or server) – is, from the loop’s point of view, just another async device: you submit opaque work, it runs on a coprocessor, and it signals completion through a fence. On Linux a fence is a pollable file descriptor (a sync_file / dma-fence), which is exactly the kind of readiness event the loop already waits on for sockets. So libxtc parks a fiber on a GPU/NPU completion the same way it parks one on a socket read (xtc_accel(3)):

/* inside a fiber: the consumer's runtime submitted work and gave us a
 * completion fence fd; park until the device is done -- no OS thread
 * held while we wait. */
int fence_fd = submit_inference(model, input);   /* your runtime */
if (xtc_accel_wait_fence(fence_fd, -1) == XTC_OK)
	use_result();

xtc_accel_probe enumerates the GPUs and NPUs present; GPU and NPU are the same abstraction here (both DRM devices whose completions are the same kind of fence fd), so one API covers both and the device kind is just a tag. For a runtime that only offers a blocking submit-and-wait, xtc_accel_run_blocking routes it through the pool above instead.

The abstraction stops at the fence. libxtc links no GPU/NPU runtime – not Level Zero, CUDA, Vulkan, or OpenVINO – and owns no tensors, no device memory, no model format. Your code (or the vendor runtime it links) submits the work and produces the fence; libxtc’s job is only to park a fiber on that fence and wake it, deterministically-testable under the simulator. That boundary is what keeps this a concurrency primitive rather than a compute-runtime shim. Built only where the platform has the DRM/accel subsystem (auto-detected; --without-accel to force off).

Scaling across cores: the executor

One loop uses one core. To use all of them, run an executor – N loops, one per core, with work-stealing between them (xtc_exec(3)). A process spawned on a busy loop can be stolen and run on an idle one; messages and completions cross loops safely. Your process code does not change – it is still spawn / send / recv – it just runs on more cores.

A thread pool per subsystem. The mainstream C server design gives each subsystem its own thread pool and synchronizes with locks and queues between them. It works, but the thread count grows with the number of subsystems, context switches dominate under load, and the lock graph becomes the system’s hardest correctness problem. The executor model uses one thread per core regardless of how many subsystems you have; concurrency comes from fibers, not threads, and the only threads are the N loop threads plus the blocking pool. Fewer threads, no cross-subsystem lock graph.

The other direction: bridging INTO the runtime

The blocking pool sends work out of a loop. Sometimes you need the inverse: something outside the runtime – a C library’s completion callback, a signal handler’s follow-up, an embedder’s own I/O thread – needs to run an effect on a loop and get its result back. That is the dispatcher (xtc_dispatch(3)), libxtc’s answer to Cats Effect’s Dispatcher.

xtc_dispatch(loop, fn, arg, &fut, &handle) spawns a fiber on loop that runs fn(arg), and hands back a future for the result plus an optional cancel handle. It is safe to call from any OS thread, including one libxtc knows nothing about – the effect body itself:


/* The effect to run on the runtime.  Returns a result the future
 * carries back to the caller. */
static int
compute(void *arg)
{
	return (int)(intptr_t)arg * 2;
}

Tested source: docs/_includes/snippets/09_dispatch.c

and the submit-and-await, from a foreign thread:


int
main(void)
{
	pthread_t th;
	xtc_future_t *fut = NULL;
	intptr_t out = 0;

	if (xtc_exec_init(&g_exec, 2) != XTC_OK)
		return 1;
	xtc_exec_set_service_mode(g_exec, 1);
	if (pthread_create(&th, NULL, run_exec, NULL) != 0)
		return 1;
	usleep(5000);                       /* let the workers spin up */

	/* From this foreign thread, submit compute(21) to run on a loop
	 * and await the doubled result. */
	if (xtc_dispatch(xtc_exec_loop(g_exec, 0), compute,
	    (void *)(intptr_t)21, &fut, NULL) != XTC_OK)
		return 1;
	if (xtc_future_wait(fut, &out, -1) != XTC_OK)
		return 1;

	printf("dispatch result: %ld\n", (long)out);   /* 42 */

	(void)xtc_exec_stop(g_exec);
	(void)pthread_join(th, NULL);
	(void)xtc_exec_fini(g_exec);
	return out == 42 ? 0 : 1;
}

Tested source: docs/_includes/snippets/09_dispatch.c

The future always resolves exactly once: with fn’s value on a normal return, or XTC_E_ABORTED if you xtc_dispatch_cancel the handle or the fiber crashes – never lost, never doubled, never a hang. Cancellation is cooperative and composes with xtc_uncancelable(3) and xtc_scope(3), so a resource acquired under a scope is still released on the cancellation path.

Why a blessed one-call front door? Nothing here is new machinery: a cross-thread xtc_proc_spawn already posts to the target loop’s MPSC inbox and pings its poller, and xtc_promise_set is already safe from any thread. Consumers kept re-assembling exactly that pair by hand at every callback boundary. xtc_dispatch is that composition, packaged with the cancellation core so the common bridge is one call that falls into the pit of success.

Determinism: testing the whole thing

Because all concurrency flows through the loop, libxtc can replace the real scheduler and I/O with a deterministic simulation driven by a seed – the FoundationDB / TigerBeetle approach. The same process code runs, but scheduling, timers, message ordering, and injected faults (partitions, latency, disk errors, crashes) are reproducible from the seed. This is how libxtc finds concurrency bugs before they ship; see the test/sim/ suite and the deterministic-testing notes.

What you have learned

  • Never block the loop thread; suspend instead.
  • Sockets/timers/fds suspend via the OS poller; files via xtc_aio_*; everything else via the explicit xtc_blocking_run pool.
  • The executor scales the same process code across cores with work-stealing.
  • Deterministic simulation replays the whole concurrent system from a seed.

That completes the guide. For the mental-model shift from threads-and-locks thinking, read Thinking in libxtc; to see it all assembled into real programs, read the Examples.

← Links, monitors, and supervisors · Next: Resource limits →