Scheduling and CPU shares

Weight CPU between workloads with proportional-share scheduling, and catch a fiber that hogs a core -- both opt-in, both inspired by Glommio.

---

The gap: FIFO splits CPU evenly
Proportional share: weighting CPU between classes
1. Latency classes
2. Zero overhead when unused
The stall watchdog: who hogged the core?
When to reach for this
Demonstrated in
See also

By default libxtc’s run queue is a plain FIFO plus a work-stealing deque: equally busy fibers get roughly equal CPU, and there is no way to say “the flush workload should get 3x the query workload” or “this class must be scheduled within 500us”. This chapter covers the two opt-in scheduler features that add exactly that – proportional-share scheduling and the over-budget stall watchdog – both off by default with zero overhead until you use them.

Both are directly inspired by Glommio (Glauber Costa / ScyllaDB), the thread-per-core io_uring runtime whose executor gives each task queue shares and a latency class and picks the next queue by a CFS-style virtual-runtime heap, and whose stall detector reports which task monopolized a core. libxtc’s north star is the same multi-tenant / mixed-workload database use, so these were the one capability worth adopting outright.

The gap: FIFO splits CPU evenly

Put two groups of equally busy, well-behaved worker fibers on one loop – call them class A and class B, each doing equal compute chunks with a cooperative xtc_yield between them. Under the plain FIFO run queue the CPU split is about 1:1, no matter what weighting you want. There is no knob. That is the gap.

Create a scheduling class with xtc_exec_class_create, give it shares in 1..1000, and place a process’s tasks in it with xtc_proc_opts_t.sched_class at spawn (or xtc_proc_set_class from inside the process). A loop that carries more than one class picks the minimum-virtual-runtime class on each dispatch, so each class gets a CPU fraction proportional to its shares:

#include <stdio.h>
#include <string.h>
#include <stdatomic.h>

#include "xtc.h"
#include "xtc_loop.h"
#include "xtc_exec.h"
#include "xtc_proc.h"

#define BUDGET 900

static atomic_int g_total;   /* shared work budget across both classes */

/* Workers of BOTH classes contend for ONE shared budget, each round taken
 * by whichever class the scheduler picks next.  That is what makes this a
 * proportional-share demonstration: the classes are competing for a
 * scarce resource, so the higher-shares class gets more of it.
 *
 * (An earlier version gave every worker a fixed private round count.
 * That is deterministic but measures nothing: equal work in means equal
 * runs out, and it reported 1.00:1 every time.  Shares weight vruntime
 * ACCRUAL -- they decide who is picked next when work is contended, not
 * how much total work a task is allowed to do.) */
static void
worker(void *arg)
{
	(void)arg;
	while (atomic_fetch_add_explicit(&g_total, 1, memory_order_relaxed)
	    < BUDGET)
		xtc_yield();
}

struct ctx {
	xtc_loop_t      *loop;
	xtc_exec_class_t a;
	xtc_exec_class_t b;
};

static void
spawner(void *arg)
{
	struct ctx *c = arg;
	xtc_proc_opts_t oa, ob;
	int i;

	memset(&oa, 0, sizeof oa);
	memset(&ob, 0, sizeof ob);
	oa.sched_class = c->a;   /* place these tasks in class A */
	ob.sched_class = c->b;   /* and these in class B */
	for (i = 0; i < 3; i++) {
		(void)xtc_proc_spawn(c->loop, worker, NULL, &oa, NULL);
		(void)xtc_proc_spawn(c->loop, worker, NULL, &ob, NULL);
	}
}

int
main(void)
{
	xtc_loop_t *loop;
	xtc_exec_class_t a = NULL, b = NULL;
	struct ctx c;
	double ratio;

	if (xtc_loop_init(&loop) != XTC_OK)
		return 1;

	/* Class A: 3 shares.  Class B: 1 share.  Optional latency bound 0
	 * (none).  With no class created the loop would be a plain FIFO. */
	if (xtc_exec_class_create(loop, 3, 0, &a) != XTC_OK ||
	    xtc_exec_class_create(loop, 1, 0, &b) != XTC_OK)
		return 1;

	/* L3: report any run that hogs the loop for more than 50 ms.  Off
	 * by default; this arms it (a NULL callback logs + backtraces). */
	xtc_loop_set_stall_budget(loop, 50 * 1000 * 1000LL);

	c.loop = loop;
	c.a = a;
	c.b = b;
	if (xtc_proc_spawn(loop, spawner, &c, NULL, NULL) != XTC_OK)
		return 1;

	(void)xtc_loop_run(loop);

	/* Class A has 3 shares to class B's 1, so A is picked more often.
	 * Report the observed counts rather than asserting a ratio -- see the
	 * note below on why an exact ratio is not a real-hardware property. */
	ratio = xtc_exec_class_runs(b)
	    ? (double)xtc_exec_class_runs(a) / (double)xtc_exec_class_runs(b)
	    : 0.0;
	{
		unsigned long long stalls = xtc_loop_stall_count(loop);
		printf("class A runs=%llu, class B runs=%llu, ratio=%.2f:1 "
		    "(shares 3:1), stalls=%llu\n",
		    (unsigned long long)xtc_exec_class_runs(a),
		    (unsigned long long)xtc_exec_class_runs(b),
		    ratio, stalls);

		(void)xtc_loop_fini(loop);
		/* The well-behaved workers never trip the stall watchdog.
		 *
		 * NOTE what this does and does not assert.  It does NOT assert
		 * a run ratio, because on real hardware it cannot: vruntime
		 * accrues by MEASURED run time floored to a minimum quantum,
		 * and a yield-only worker's real elapsed time is dominated by
		 * scheduling jitter -- so the accrual, and with it the pick
		 * order, is noisy.  Asserting `ratio >= 2.0` here made this a
		 * flaky release gate: observed 0.10:1 to 16.65:1 across runs on
		 * an unchanged tree.
		 *
		 * There is also a genuine counter-effect worth understanding:
		 * the favoured class drains the shared budget FASTER and so
		 * finishes and stops running sooner, after which only the
		 * slower class is left to accumulate runs.  A raw run count at
		 * the end of a fixed budget is therefore not a clean CPU-share
		 * signal on real hardware.
		 *
		 * Strict proportionality IS proven -- under the deterministic
		 * simulator, where the virtual clock makes every run cost
		 * exactly the floor, so the run ratio equals the share ratio
		 * exactly and reproducibly: test/sim/test_sim_sched_shares.c.
		 * That is the right place for the numeric claim; this snippet's
		 * job is to show the API and print what it observed. */
		return stalls == 0 ? 0 : 1;
	}
}

Tested source: docs/_includes/snippets/10_sched_shares.c

Class A has 3 shares and class B has 1, so A is picked more often. The virtual-runtime accrual is Glommio’s exact formula: vruntime += (cost_ns * reciprocal) >> 12, where reciprocal = (1<<22)/shares, so a higher-shares class accrues virtual time more slowly and is chosen more often.

Be precise about what the snippet above prints, because the honest claim is narrower than “3:1 shares gives 3:1 run counts” on real hardware:

Accrual is weighted by measured run time, floored to a minimum quantum. For a yield-only worker the real elapsed time is dominated by scheduling jitter, so the observed run ratio is noisy – it is not a stable 3:1.
A shared work budget adds a counter-effect: the favoured class drains it faster, finishes sooner, and then stops running, after which only the slower class accumulates runs. A raw end-of-budget run count is therefore not a clean CPU-share signal.

Strict proportionality – run ratio equal to share ratio, exactly and reproducibly – is proven under the deterministic simulator, where the virtual clock makes every run cost exactly the floor: test/sim/test_sim_sched_shares.c. That is where the numeric guarantee lives. On real hardware, treat shares as a weighting that holds in aggregate over a sustained contended workload, not as an exact ratio you can assert over a few hundred dispatches.

Untagged work (any task with no class) races an implicit default lane in the same pick, so background work is never starved by always-ready class work – it simply gets a default weight.

Latency classes

Pass a non-zero latency_ns to xtc_exec_class_create and the loop’s cooperative yield interval shrinks to that bound (like Glommio’s reevaluate_preempt_timer), so a latency-sensitive class is serviced promptly instead of waiting behind a throughput backlog. This is how you say “compaction gets 20% of the CPU, but queries must be scheduled within 1ms” on a single core.

Zero overhead when unused

A loop with no class created runs the exact plain-FIFO + work-stealing path, byte-for-byte; the virtual-runtime accounting activates only once a class exists. You pay nothing until you opt in.

The stall watchdog: who hogged the core?

libxtc is cooperatively scheduled – a fiber that never yields monopolizes its loop. The over-budget stall watchdog turns that from a silent tail-latency mystery into an alertable signal: arm a per-loop budget with xtc_loop_set_stall_budget (or every loop of an executor at once with xtc_exec_set_stall_budget), and when a single run exceeds it the runtime reports which task did it.

xtc_loop_set_stall_budget(loop, 50 * 1000 * 1000LL);  /* 50 ms */
xtc_loop_set_stall_cb(loop, my_report_fn, ctx);       /* or NULL to log */

With no callback the runtime logs a warning and emits a backtrace of the loop to stderr. xtc_loop_stall_count returns how many overruns fired. The check is a cheap wall-clock comparison at the run-end boundary of the loop step – no watcher thread, no signal – so it is a single branch on a disabled flag when off. This is inspired by Glommio’s stall detector.

When to reach for this

Mixed workloads on one core (compaction vs queries, flush vs read path): give each a class and weight the shares.
A latency SLA on one workload while another does bulk throughput: a latency class.
Diagnosing “something occasionally pins a core for 200ms”: arm the stall budget and let it name the culprit.

If your workload is a single kind of task, or you are happy with even sharing, leave both off – the default FIFO + work stealing is simpler and faster.

Demonstrated in

examples/03_supervised_app.c – the minimal case: two supervised children share one loop as two classes (a latency-bounded foreground counter with 3x shares vs a best-effort background stats printer), plus the stall watchdog.
examples/06_sqlxtc – the real case: the SQL server enables the over-budget stall watchdog so a runaway query or structure-modification that hogs a worker loop is logged with a backtrace instead of silently stalling every other connection on that loop.