xtc_exec(3)

---

xtc_exec(3)

L2 multi-loop work-stealing executor

XTC_EXEC(3) Library Functions Manual XTC_EXEC(3)

xtc_exec_init, xtc_exec_fini, xtc_exec_run, xtc_exec_set_service_mode, xtc_exec_get_service_mode, xtc_exec_set_eager_rebalance, xtc_exec_get_eager_rebalance, xtc_exec_set_steal_backoff, xtc_exec_get_steal_backoff, xtc_exec_set_preempt, xtc_exec_stop, xtc_exec_n_loops, xtc_exec_loop, xtc_exec_loop_stats, xtc_exec_loop_id, xtc_shard_id, xtc_shard_count, xtc_exec_spawn, xtc_exec_spawn_on, xtc_exec_async, xtc_exec_async_on, xtc_exec_class_create, xtc_exec_class_shares, xtc_exec_class_latency, xtc_exec_class_runs, xtc_exec_class_vruntime, xtc_exec_set_stall_budget, xtc_loop_set_stall_budget, xtc_loop_set_stall_cb, xtc_loop_stall_count — L2 multi-loop work-stealing executor

#include <xtc_exec.h>

int
xtc_exec_init(xtc_exec_t **out, int n_loops);

int
xtc_exec_fini(xtc_exec_t *exec);

int
xtc_exec_run(xtc_exec_t *exec);

void
xtc_exec_set_service_mode(xtc_exec_t *exec, int on);

int
xtc_exec_get_service_mode(xtc_exec_t *exec);

void
xtc_exec_set_eager_rebalance(xtc_exec_t *exec, int on);

int
xtc_exec_get_eager_rebalance(xtc_exec_t *exec);

void
xtc_exec_set_steal_backoff(xtc_exec_t *exec, int on);

int
xtc_exec_get_steal_backoff(xtc_exec_t *exec);

int
xtc_exec_set_preempt(xtc_exec_t *exec, int64_t interval_ns);

int
xtc_exec_stop(xtc_exec_t *exec);

int
xtc_exec_n_loops(xtc_exec_t *exec);

xtc_loop_t *
xtc_exec_loop(xtc_exec_t *exec, int idx);

int
xtc_exec_loop_stats(xtc_exec_t *exec, int idx, xtc_loop_stats_t *out);

int
xtc_exec_loop_id(void);

int
xtc_shard_id(void);

int
xtc_shard_count(void);

int
xtc_exec_spawn(xtc_exec_t *exec, xtc_task_fn fn, void *user, xtc_task_t **out);

int
xtc_exec_spawn_on(xtc_exec_t *exec, int idx, xtc_task_fn fn, void *user, xtc_task_t **out);

int
xtc_exec_async(xtc_exec_t *exec, xtc_coro_fn fn, void *arg, xtc_task_t **out);

int
xtc_exec_async_on(xtc_exec_t *exec, int idx, xtc_coro_fn fn, void *arg, xtc_task_t **out);

int
xtc_exec_class_create(xtc_loop_t *loop, int shares, int64_t latency_ns, xtc_exec_class_t *out);

int
xtc_exec_class_shares(xtc_exec_class_t cls);

int64_t
xtc_exec_class_latency(xtc_exec_class_t cls);

uint64_t
xtc_exec_class_runs(xtc_exec_class_t cls);

uint64_t
xtc_exec_class_vruntime(xtc_exec_class_t cls);

void
xtc_exec_set_stall_budget(xtc_exec_t *exec, int64_t budget_ns);

void
xtc_loop_set_stall_budget(xtc_loop_t *loop, int64_t budget_ns);

void
xtc_loop_set_stall_cb(xtc_loop_t *loop, xtc_stall_cb cb, void *user);

uint64_t
xtc_loop_stall_count(const xtc_loop_t *loop);

xtc_exec manages event loops, one per worker thread. Each loop owns a Chase-Lev work-stealing deque as its run queue plus an MPSC inbox for cross-thread wakers and remote spawns.

() constructs an executor with n_loops loops; non-positive values default to a small reasonable count. () joins every worker, frees every loop.

() spawns n_loops worker threads (one per loop) and blocks on the calling thread until the executor reaches a quiescent state 2014 every spawned task DONE, every timer fired or cancelled, every inbox drained, every deque empty 2014 or until () is called from any thread.

() opts the executor out of that quiescent-state auto-stop. When on is non-zero, xtc_exec_run() blocks until xtc_exec_stop() is called explicitly and never stops merely because every loop went idle. Use it for a long-running service (a supervised xtc_app(3)), where a transient all-idle window during startup -- before every supervised child has been scheduled across loops -- must not terminate the application. A bare work-pool leaves it off so the executor drains and returns when its tasks finish.

() enables cooperative-assisted preemption: each worker arms a per-worker CPU-time interval timer at interval_ns (of that worker thread's own CPU time), and a timer tick makes that worker's xtc_yield_if_due(3) callers yield, so a long compute fiber that periodically consults a yield check is time-sliced without a manual budget. interval_ns of 0 disables it (the default). Call before xtc_exec_run(); workers read the setting at start. Returns XTC_E_NOSYS where the platform lacks per-thread CPU-time timers (the setting is stored but arming is a no-op). This does not preempt a fiber that never reaches a yield check.

Workers steal from peers when their own deque empties before going to xtc_io(3) poll. M5 uses a random victim policy; NUMA-near steal ordering arrives in M5.5.

() (OFF by default) controls WHEN a loop steals a migratable (xtc_proc(3), migratable) peer proc. By default a loop steals only when FULLY idle -- its run queue empty AND no parked fibers or timers -- and discovers a sibling's stealable work only on its next poll edge. Under a load where every loop owns parked fibers (e.g. many server backends parked on client sockets while a peer loop has a runnable query), no loop is ever fully idle, so migratable work sits on the stealable deque and is never taken. With eager rebalance ON, a loop whose RUN QUEUE is empty (even if it owns parked fibers or timers) steals before it blocks in the poller, and enqueuing a migratable task nudges one idle peer to steal it promptly. This trades some cross-loop migration (and its cache/NUMA cost) for reclaiming idle capacity under partial load; only migratable tasks are ever moved, so pinned work (the default) is unaffected either way. A supervised server whose backends park on sockets opts in; a cache-locality-bound workload leaves it off.

() (OFF by default) controls how hard an IDLE loop scans for work. By default an idle worker re-polls (and re-attempts a steal) on a fixed 1 ms tick, so on a large executor that is mostly idle -- many carriers, tiny or infrequent work -- the aggregate cost of repeated empty steal scans is visible even though the machine is nearly idle. With steal backoff ON, a worker that has taken no work for several consecutive turns grows its idle poll timeout exponentially (from 1 ms toward a 32 ms cap); any real work resets it immediately. This cuts the idle-scan CPU of an over-provisioned executor at the cost of a few extra milliseconds of worst-case latency before a deeply-idle loop notices stealable work (a cross-thread enqueue still nudges it at once when eager rebalance is also on). It is complementary to eager rebalance: eager controls WHEN an idle-but-not-drained loop steals; backoff controls how often a fully-idle loop bothers to look.

(), (), and () read back the current setting (0 for a NULL exec).

Every per- xtc_exec_t scheduler policy (service mode, eager rebalance, and any future addition -- steal) policy, NUMA affinity mode, migration eagerness, etc. follows one shape:

  • Named ()name / ()name; a boolean knob takes/returns a plain int (0/1).
  • Takes effect immediately; no re-init needed, and safe to call from any thread at any time (each knob is a single atomic field on the executor).
  • Documented with its COST as well as its effect -- a policy that trades CPU or syscalls for responsiveness (like eager rebalance) says so, so a caller can judge whether to opt in.

This is deliberately distinct from two other things that look similar:

xtc_proc(3)'s migratable
is a per-PROC ELIGIBILITY bit decided once at spawn (can this proc's coro be moved at all), not an executor-wide policy (how hard the executor tries to move eligible work). Keeping eligibility and policy separate means a consumer who wants some procs movable does not pay an executor-wide policy's runtime cost, and vice versa.
xtc_preempt(3)'s ()
is a PROCESS-WIDE toggle (no xtc_exec_t argument) governing the signal-context preemption redirect, not scoped to one executor.
xtc_cfg(3)
is a separate, PROCESS-GLOBAL named-tunable registry (log levels, backpressure thresholds, ops-facing settings meant to) survive a config-file reload with no per-instance scoping yet, so it does not back a per-exec knob like these -- the two mechanisms serve different scopes on purpose.

() puts the new task on a loop chosen by a round-robin counter. xtc_exec_spawn_on() forces placement on the given loop. The coroutine variants () and () mirror the contract.

Spawn calls are safe from any thread, including before (): work queued in advance is processed when the workers start. Spawns from a non-owner thread are routed through the target loop's inbox and surface on the loop's next step.

The run queue is a plain FIFO plus the work-stealing deque by default, which splits CPU roughly evenly among equally busy fibers. The optional scheduler adds weighted-fair scheduling: create a scheduling with (), giving it shares (1..1000) and an optional latency_ns bound, then place a process's tasks in it via xtc_proc_opts_t.sched_class at spawn or xtc_proc_set_class(3) from inside the process. A loop that carries more than one class picks the minimum-virtual-runtime class on each dispatch, so each class receives a CPU fraction proportional to its shares — a class with 3x the shares of another gets about 3x the CPU. A non-zero latency_ns shrinks the loop's cooperative yield interval so the class is serviced promptly.

The feature is OFF by default with ZERO overhead: a loop with no class created runs the exact plain-FIFO + work-stealing path, byte-for-byte, and the virtual-runtime accounting activates only once a class exists. Untagged tasks race an implicit default lane so background work is never starved by always-ready class work. (), (), (), and () read a class handle's parameters and telemetry (run count, accumulated virtual runtime) back.

This is INSPIRED BY Glommio (Glauber Costa / ScyllaDB), whose executor gives each task queue shares and a latency class and picks the next queue by a CFS-style virtual-runtime heap (executor/mod.rs account_vruntime, shares.rs). The accrual formula is Glommio's: vruntime += (cost_ns * reciprocal) >> 12, where reciprocal = (1<<22)/shares.

When a single task run monopolizes a loop past a budget, the optional stall watchdog reports which task did it. Arm a per-loop budget with () (nanoseconds; 0 disables) or arm every loop of an executor at once with (). Install a report sink with (); with no callback the runtime logs a warning and emits a backtrace of the loop to stderr. () returns the number of over-budget reports (telemetry).

The check is a cheap wall-clock comparison at the run-end boundary in the loop step — no watcher thread, no signal — so it is a single branch on a disabled flag when off (zero overhead). This is INSPIRED BY Glommio's stall detector (executor/stall.rs).

A xtc_waker_t captured on one loop may be fired from any thread; cross-thread fires are routed through the target loop's inbox plus an () ping that breaks the target out of xtc_io(3) poll. The wake remains idempotent: many concurrent wakes collapse to at most one re-queue per yielded run.

() returns the executor index of the loop the calling task is running on, or -1 from a thread that is not a worker. Tests use this to verify spawn placement and steal behaviour. It is also the migration-detection idiom: because it re-reads the CURRENT resume's carrier every call (like xtc_self(3)), a proc that captures it before a yield and compares after learns whether it was work-stolen to a different loop in between -- see xtc_proc(3) MIGRATION for the pattern a consumer with carrier-affine cached state (e.g. a per-loop wait-set registration) should use.

() and () are the Seastar-style per-shard API: the 0-based index of the loop the caller runs on, and the number of shards. Unlike xtc_exec_loop_id() they treat a standalone loop (not under an executor) as shard 0 of 1, so a shared-nothing consumer can always index per-core state with xtc_shard_id() regardless of whether it runs under the executor or a single loop. Both return the off-loop sentinels (-1 and 0) only when the caller is not on any loop.

() fills out with loop idx's work counters: tasks_run (task steps executed on that loop) and steals (tasks that loop stole from a peer). The values are lock-free snapshots, for load-balance and tail-latency diagnosis. Note that explicitly placed work (()) and processes are pinned to their loop and are never stolen; only the unpinned general task pool (xtc_exec_spawn()) migrates.

XTC_OK on success; XTC_E_INVAL on a bad argument; XTC_E_NOMEM on allocation failure; XTC_E_INTERNAL on platform failure.

All public functions are safe to call from any thread. A given task handle (out_task) returned from a spawn must be used only in ways consistent with the M3/M4 task contracts; in particular, a xtc_waker_t captured from one task may travel anywhere.

static intptr_t hello(void *arg) {
        printf("hello from loop %d\n", xtc_exec_loop_id());
        return 0;
}

int main(void) {
        xtc_exec_t *e;
        xtc_exec_init(&e, 4);
        for (int i = 0; i < 4; i++)
                xtc_exec_async(e, hello, NULL, NULL);
        xtc_exec_run(e);
        xtc_exec_fini(e);
        return 0;
}

xtc_loop(3), xtc_async(3), xtc_io(3), xtc(7)

First appeared in xtc 0.0.1 (M5). M5.5 will add NUMA-aware steal ordering and configure-time auto-detect of CPU count via .

May 25, 2026 Debian

View the mdoc source