xtc_exec(3)

---

xtc_exec(3)

L2 multi-loop work-stealing executor

XTC_EXEC(3) Library Functions Manual XTC_EXEC(3)

xtc_exec_init, xtc_exec_fini, xtc_exec_run, xtc_exec_set_service_mode, xtc_exec_get_service_mode, xtc_exec_set_eager_rebalance, xtc_exec_get_eager_rebalance, xtc_exec_set_steal_backoff, xtc_exec_get_steal_backoff, xtc_exec_set_preempt, xtc_exec_stop, xtc_exec_n_loops, xtc_exec_loop, xtc_exec_loop_stats, xtc_exec_loop_id, xtc_shard_id, xtc_shard_count, xtc_exec_spawn, xtc_exec_spawn_on, xtc_exec_async, xtc_exec_async_on, xtc_exec_class_create, xtc_exec_class_shares, xtc_exec_class_latency, xtc_exec_class_runs, xtc_exec_class_vruntime, xtc_exec_set_stall_budget, xtc_loop_set_stall_budget, xtc_loop_set_stall_cb, xtc_loop_stall_count — L2 multi-loop work-stealing executor

#include <xtc_exec.h>

int
xtc_exec_init(xtc_exec_t **out, int n_loops);

int
xtc_exec_fini(xtc_exec_t *exec);

int
xtc_exec_run(xtc_exec_t *exec);

void
xtc_exec_set_service_mode(xtc_exec_t *exec, int on);

int
xtc_exec_get_service_mode(xtc_exec_t *exec);

void
xtc_exec_set_eager_rebalance(xtc_exec_t *exec, int on);

int
xtc_exec_get_eager_rebalance(xtc_exec_t *exec);

void
xtc_exec_set_steal_backoff(xtc_exec_t *exec, int on);

int
xtc_exec_get_steal_backoff(xtc_exec_t *exec);

int
xtc_exec_set_preempt(xtc_exec_t *exec, int64_t interval_ns);

int
xtc_exec_stop(xtc_exec_t *exec);

int
xtc_exec_n_loops(xtc_exec_t *exec);

xtc_loop_t *
xtc_exec_loop(xtc_exec_t *exec, int idx);

int
xtc_exec_loop_stats(xtc_exec_t *exec, int idx, xtc_loop_stats_t *out);

int
xtc_exec_loop_id(void);

int
xtc_shard_id(void);

int
xtc_shard_count(void);

int
xtc_exec_spawn(xtc_exec_t *exec, xtc_task_fn fn, void *user, xtc_task_t **out);

int
xtc_exec_spawn_on(xtc_exec_t *exec, int idx, xtc_task_fn fn, void *user, xtc_task_t **out);

int
xtc_exec_async(xtc_exec_t *exec, xtc_coro_fn fn, void *arg, xtc_task_t **out);

int
xtc_exec_async_on(xtc_exec_t *exec, int idx, xtc_coro_fn fn, void *arg, xtc_task_t **out);

int
xtc_exec_class_create(xtc_loop_t *loop, int shares, int64_t latency_ns, xtc_exec_class_t *out);

int
xtc_exec_class_shares(xtc_exec_class_t cls);

int64_t
xtc_exec_class_latency(xtc_exec_class_t cls);

uint64_t
xtc_exec_class_runs(xtc_exec_class_t cls);

uint64_t
xtc_exec_class_vruntime(xtc_exec_class_t cls);

void
xtc_exec_set_stall_budget(xtc_exec_t *exec, int64_t budget_ns);

void
xtc_loop_set_stall_budget(xtc_loop_t *loop, int64_t budget_ns);

void
xtc_loop_set_stall_cb(xtc_loop_t *loop, xtc_stall_cb cb, void *user);

uint64_t
xtc_loop_stall_count(const xtc_loop_t *loop);

xtc_exec manages event loops, one per worker thread. Each loop owns a Chase-Lev work-stealing deque as its run queue plus an MPSC inbox for cross-thread wakers and remote spawns. The owner takes from its deque in FIFO order, the same end thieves steal from, so a task that yields (XTC_TASK_RESCHED) runs again only after every task queued before it. The inbox is lock-free and intrusive (no allocation per wake), so a cross-thread wake cannot fail.

() constructs an executor with n_loops loops; non-positive values default to a small reasonable count. () joins every worker, frees every loop.

() spawns n_loops worker threads (one per loop) and blocks on the calling thread until the executor reaches a quiescent state 2014 every spawned task DONE, every timer fired or cancelled, every inbox drained, every deque empty 2014 or until () is called from any thread.

() opts the executor out of that quiescent-state auto-stop. When on is non-zero, xtc_exec_run() blocks until xtc_exec_stop() is called explicitly and never stops merely because every loop went idle. Use it for a long-running service (a supervised xtc_app(3)), where a transient all-idle window during startup -- before every supervised child has been scheduled across loops -- must not terminate the application. A bare work-pool leaves it off so the executor drains and returns when its tasks finish.

() enables cooperative-assisted preemption: each worker arms a per-worker CPU-time interval timer at interval_ns (of that worker thread's own CPU time), and a timer tick makes that worker's xtc_yield_if_due(3) callers yield, so a long compute fiber that periodically consults a yield check is time-sliced without a manual budget. interval_ns of 0 disables it (the default). Call before xtc_exec_run(); workers read the setting at start. Returns XTC_E_NOSYS where the platform lacks per-thread CPU-time timers (the setting is stored but arming is a no-op). This does not preempt a fiber that never reaches a yield check.

A loop steals from a peer whenever its own run queue is empty, before it blocks in xtc_io(3) poll, whether or not it also owns parked fibers or pending timers; only migratable (xtc_proc(3), migratable) work, which sits on the stealable deque, is ever taken. While its run queue stays empty it bounds each blocking poll to 1 ms (32 ms with steal backoff on) and looks again, so a loop holding a long sleep or a timed receive still picks up a busy peer's backlog within about a millisecond. The victim scan prefers same-NUMA-node peers and starts at a rotating position. Releases up to 1.55.0 stole only when a loop was fully idle (no parked fibers and no pending timers), so a single timer on a loop switched stealing off for it.

() (OFF by default) makes stealing prompt rather than periodic: enqueuing a migratable task nudges one idle peer's poller so it steals at once instead of on its next tick. This trades a cross-thread wakeup per enqueue (when a peer is idle) and some extra migration for lower rebalance latency; pinned work (the default) is unaffected either way. A supervised server whose backends park on sockets opts in; a cache-locality-bound workload leaves it off.

() (OFF by default) controls how hard an IDLE loop scans for work. By default an idle worker re-polls (and re-attempts a steal) on a fixed 1 ms tick, so on a large executor that is mostly idle -- many carriers, tiny or infrequent work -- the aggregate cost of repeated empty steal scans is visible even though the machine is nearly idle. With steal backoff ON, a worker that has taken no work for several consecutive turns grows its idle poll timeout exponentially (from 1 ms toward a 32 ms cap); any real work resets it immediately. This cuts the idle-scan CPU of an over-provisioned executor at the cost of a few extra milliseconds of worst-case latency before an idle loop notices stealable work (a cross-thread enqueue still nudges it at once when eager rebalance is also on); the same 32 ms bound then applies to a loop whose run queue is empty but which owns parked fibers or timers. It is complementary to eager rebalance: eager controls whether an enqueue wakes an idle peer; backoff controls how often an idle loop bothers to look.

(), (), and () read back the current setting (0 for a NULL exec).

Every per- xtc_exec_t scheduler policy (service mode, eager rebalance, and any future addition -- steal) policy, NUMA affinity mode, migration eagerness, etc. follows one shape:

  • Named ()name / ()name; a boolean knob takes/returns a plain int (0/1).
  • Takes effect immediately; no re-init needed, and safe to call from any thread at any time (each knob is a single atomic field on the executor).
  • Documented with its COST as well as its effect -- a policy that trades CPU or syscalls for responsiveness (like eager rebalance) says so, so a caller can judge whether to opt in.

This is deliberately distinct from two other things that look similar:

xtc_proc(3)'s migratable
is a per-PROC ELIGIBILITY bit decided once at spawn (can this proc's coro be moved at all), not an executor-wide policy (how hard the executor tries to move eligible work). Keeping eligibility and policy separate means a consumer who wants some procs movable does not pay an executor-wide policy's runtime cost, and vice versa.
xtc_preempt(3)'s ()
is a PROCESS-WIDE toggle (no xtc_exec_t argument) governing the signal-context preemption redirect, not scoped to one executor.
xtc_cfg(3)
is a separate, PROCESS-GLOBAL named-tunable registry (log levels, backpressure thresholds, ops-facing settings meant to) survive a config-file reload with no per-instance scoping yet, so it does not back a per-exec knob like these -- the two mechanisms serve different scopes on purpose.

() puts the new task on a loop chosen by a round-robin counter. xtc_exec_spawn_on() forces placement on the given loop. The coroutine variants () and () mirror the contract.

Spawn calls are safe from any thread, including before (): work queued in advance is processed when the workers start. Spawns from a non-owner thread are routed through the target loop's inbox and surface on the loop's next step.

The run queue is a plain FIFO plus the work-stealing deque by default, which splits CPU roughly evenly among equally busy fibers. The optional scheduler adds weighted-fair scheduling: create a scheduling with (), giving it shares (1..1000) and an optional latency_ns bound, then place a process's tasks in it via xtc_proc_opts_t.sched_class at spawn or xtc_proc_set_class(3) from inside the process. A loop that carries more than one class picks the minimum-virtual-runtime class on each dispatch, so each class receives a CPU fraction proportional to its shares -- a class with 3x the shares of another gets about 3x the CPU. A non-zero latency_ns shrinks the loop's cooperative yield interval so the class is serviced promptly.

The feature is OFF by default with ZERO overhead: a loop with no class created runs the exact plain-FIFO + work-stealing path, byte-for-byte, and the virtual-runtime accounting activates only once a class exists. Untagged tasks race an implicit default lane so background work is never starved by always-ready class work. (), (), (), and () read a class handle's parameters and telemetry (run count, accumulated virtual runtime) back.

This is INSPIRED BY Glommio (Glauber Costa / ScyllaDB), whose executor gives each task queue shares and a latency class and picks the next queue by a CFS-style virtual-runtime heap (executor/mod.rs account_vruntime, shares.rs). The accrual formula is Glommio's: vruntime += (cost_ns * reciprocal) >> 12, where reciprocal = (1<<22)/shares.

When a single task run monopolizes a loop past a budget, the optional stall watchdog reports which task did it. Arm a per-loop budget with () (nanoseconds; 0 disables) or arm every loop of an executor at once with (). Install a report sink with (); with no callback the runtime logs a warning and emits a backtrace of the loop to stderr. () returns the number of over-budget reports (telemetry).

The check is a cheap wall-clock comparison at the run-end boundary in the loop step -- no watcher thread, no signal -- so it is a single branch on a disabled flag when off (zero overhead). This is INSPIRED BY Glommio's stall detector (executor/stall.rs).

A xtc_waker_t captured on one loop may be fired from any thread; cross-thread fires are routed through the target loop's inbox plus an () ping that breaks the target out of xtc_io(3) poll. The ping is sent only when the target is blocked (or about to block) in its poller: a running loop drains its inbox at the top of every step and checks it again before it blocks, so waking a busy loop costs no system call. The wake remains idempotent: many concurrent wakes collapse to at most one re-queue per yielded run.

() returns the executor index of the loop the calling task is running on, or -1 from a thread that is not a worker. Tests use this to verify spawn placement and steal behaviour. It is also the migration-detection idiom: because it re-reads the CURRENT resume's carrier every call (like xtc_self(3)), a proc that captures it before a yield and compares after learns whether it was work-stolen to a different loop in between -- see xtc_proc(3) MIGRATION for the pattern a consumer with carrier-affine cached state (e.g. a per-loop wait-set registration) should use.

() and () are the Seastar-style per-shard API: the 0-based index of the loop the caller runs on, and the number of shards. Unlike xtc_exec_loop_id() they treat a standalone loop (not under an executor) as shard 0 of 1, so a shared-nothing consumer can always index per-core state with xtc_shard_id() regardless of whether it runs under the executor or a single loop. Both return the off-loop sentinels (-1 and 0) only when the caller is not on any loop.

() fills out with loop idx's work counters: tasks_run (task steps executed on that loop) and steals (tasks that loop stole from a peer). The values are lock-free snapshots, for load-balance and tail-latency diagnosis. Note that explicitly placed work (()) and processes are pinned to their loop and are never stolen; only the unpinned general task pool (xtc_exec_spawn()) migrates.

XTC_OK on success; XTC_E_INVAL on a bad argument; XTC_E_NOMEM on allocation failure; XTC_E_INTERNAL on platform failure.

All public functions are safe to call from any thread. A given task handle (out_task) returned from a spawn must be used only in ways consistent with the M3/M4 task contracts; in particular, a xtc_waker_t captured from one task may travel anywhere.

static intptr_t hello(void *arg) {
        printf("hello from loop %d\n", xtc_exec_loop_id());
        return 0;
}

int main(void) {
        xtc_exec_t *e;
        xtc_exec_init(&e, 4);
        for (int i = 0; i < 4; i++)
                xtc_exec_async(e, hello, NULL, NULL);
        xtc_exec_run(e);
        xtc_exec_fini(e);
        return 0;
}

xtc_loop(3), xtc_async(3), xtc_io(3), xtc(7)

First appeared in xtc 0.0.1 (M5). M5.5 will add NUMA-aware steal ordering and configure-time auto-detect of CPU count via .

May 25, 2026 Debian

View the mdoc source