Debugging and observing

Finding bugs in the message-passing model with GDB/LLDB and the runtime's own introspection.

---

This guide is task-oriented: a libxtc program is misbehaving, and you want to find out why. It pairs the native-debugger tooling (tools/gdb/xtc-gdb.py, tools/lldb/xtc_lldb.py) with the programmatic introspection libxtc already exposes (xtc_stats, xtc_reg, xtc_pdict, xtc_alloc_audit, the yield watchdog). Read docs/guide/transitioning.md first if you have not – most “libxtc bugs” are one of the anti-patterns there, and knowing them tells you what to look for.

Loading the tools

GDB (and VS Code / CLion, which drive GDB)

(gdb) source /path/to/libxtc/tools/gdb/xtc-gdb.py

or once, in ~/.gdbinit:

source /path/to/libxtc/tools/gdb/xtc-gdb.py

In VS Code add to the C/C++ launch configuration:

"setupCommands": [
  { "text": "source ${workspaceFolder}/tools/gdb/xtc-gdb.py" }
]

In CLion: Settings -> Build, Execution, Deployment -> Debugger -> GDB, add the source line to the startup commands, or put it in ~/.gdbinit.

LLDB (and VS Code / CLion on macOS)

(lldb) command script import /path/to/libxtc/tools/lldb/xtc_lldb.py

Build with -g (the default build does). The tools work on a live process (run/attach/breakpoint) and on a core dump.

The five-minute triage

When you do not yet know what is wrong, attach (or load the core) and:

(gdb) xtc-loops      # how many loops, how busy, how many procs
(gdb) xtc-procs      # every proc: mailbox depth, state, links
(gdb) xtc-stranded   # if something looks parked forever (see below)

xtc-procs is the single most useful command. Its columns are the vital signs:

mbox – live mailbox depth. A large or growing number is the most common pathology (see “stuck or growing mailbox” below).
peak – the high-water mark; a peak far above the current depth means a burst the proc fell behind on.
state – SCHEDULED / RUNNING / PARKED(reason) / DONE. The park reason (fd, timer, mailbox) tells you what the proc is waiting for.
lnk/mon – links and monitors, for tracing supervision.

The commands, with example output

Every command works on a live process (run / attach / breakpoint) and on a core dump; run them while stopped so the registry symbols resolve. This is what each prints (addresses will differ; the shape is the point). The LLDB commands are identical.

`xtc-loops` – the scheduler at a glance

(gdb) xtc-loops
loop 0x555555b8e2a0  [running]  procs=3  alive=3  tasks_run=1408  steals=12
loop 0x555555b9f100  [idle]     procs=1  alive=1  tasks_run=530   steals=9

One line per loop (one per core under the executor). tasks_run is how much work each loop has done; a large steals skew means one loop is feeding the others (uneven placement).

`xtc-procs` – the observer process table

(gdb) xtc-procs
proc               pid        mbox  peak  save state              lnk/mon
0x555555b8f420     <0.1.1>       0     4     0 parked(fd)          0/1
0x555555b8fae0     <0.2.1>    2043  2043     0 parked(mailbox)     1/0
0x555555b90200     <0.3.1>       0     1     0 running             0/0
(3 procs)

The columns are the vital signs. Here <0.2.1> has a mailbox of 2043 with a matching peak – a consumer that fell behind its producers. Pass a loop address to scope the listing: xtc-procs 0x555555b9f100.

`xtc-proc ADDR` – one process in detail

(gdb) xtc-proc 0x555555b8fae0
proc 0x555555b8fae0  pid=<0.2.1>  loop=0x555555b8e2a0
  state        : parked(mailbox)
  alive        : 1   kill_pending=0
  mailbox      : depth=2043 peak=2043 cap=4096 saved=0 recv_total=118 drop_total=0
  wm           : lvl=0 fired=0
  links        : 1   monitors=0  monitored_by=0
  recovery     : armed=1 fired=0 crit_depth=0
  entry fn     : 0x555555559abc <consumer_body>

recv_total against depth tells the story: 118 received, 2043 still waiting – the producer is far ahead. entry fn names the body, so you know which proc this is without a name registry.

`xtc-mailbox ADDR` – who is queued, and from whom

(gdb) xtc-mailbox 0x555555b8fae0
  [  0] env 0x7ffff0001200  from=<0.4.1>  size=64
  [  1] env 0x7ffff0001260  from=<0.4.1>  size=64
  [  2] env 0x7ffff00012c0  from=<0.5.1>  size=64
  ...
  2043 message(s); save-queue=0

The from= column identifies the producers – here two of them (<0.4.1> and <0.5.1>) are flooding one consumer.

`xtc-self` – map a thread back to its proc

(gdb) thread 3
[Switching to thread 3 (LWP 12807)]
(gdb) xtc-self
proc 0x555555b90200  pid=<0.3.1>  loop=0x555555b8e2a0
  state        : running
  ...

Use it after thread apply all bt shows a thread you want to tie back to a libxtc process.

`xtc-trace` – the causal message trace

Covered in full in the causal-trace recipe below; it prints the send/receive ring in happens-before (HLC) order so you can follow which message caused which.

Recipe: the whole program hangs

Symptom: nothing progresses; CPU may be idle (everyone parked) or one core pinned (a proc not yielding).

(gdb) thread apply all bt        # what is each OS thread doing
(gdb) xtc-procs                  # what is each proc doing

Then reason:

One proc RUNNING, everyone else SCHEDULED and not advancing, one core hot. A CPU-bound proc is not yielding (Shift 1). The bt of the running thread shows where it is spinning.
A thread stuck in a blocking syscall (read, fsync, pthread_mutex_lock, nanosleep) in thread apply all bt, while procs sit PARKED. Something blocked the loop thread (Shift 2). The backtrace names the offending call; it should have been xtc_blocking_run / xtc_proc_sleep / an xtc_amutex.
Everyone PARKED(mailbox), no timers pending, no progress. A message that should have been sent never was, or two procs are each waiting for the other (a request/reply cycle). xtc-mailbox on the suspects shows who is waiting for what.

Map a thread to the proc it is running: select the thread and

(gdb) thread 3
(gdb) xtc-self

When a completion seems to have been lost

If a fiber is parked on I/O that you believe finished, check whether the completion is sitting unreaped in the ring:

(gdb) xtc-rings

The unreaped column is CqTail - CqHead – completions the kernel has posted that libxtc has not consumed. Cross-check it against /proc/<pid>/fdinfo/<ring_fd>, which reports the same thing from the kernel’s side.

Sample it three times a few seconds apart. A single non-zero reading means nothing: under load, completions arrive continuously and a snapshot catches them mid-flight (a healthy busy loop was measured at 218 unreaped, then 1 three seconds later). A count that stays pinned while nothing progresses is the real signal.

Once you have a persistently non-empty CQ, one inference is sound: a poller cannot be blocked in io_uring_wait_cqe on a ring whose CQ is non-empty – liburing checks the CQ before entering the kernel. So that loop’s worker is somewhere other than its own poll: running a task, blocked on a peer’s ring, or gone. Use the io column to join against the io= argument in a blocked thread’s xtc_io_poll frame.

When a fiber looks like it is never resumed

The case above assumes the waiting proc is waiting for something you can name. The harder case is a proc parked on an operation that has already completed – a lost wake. xtc-stranded is built for exactly that:

(gdb) xtc-stranded
proc states:
    PARKED       5
park kinds (PARKED procs only):
    park=-        2
    park=timer    3

proc               pid        park     wake_pending   verdict
0x514810           <0.0.1>    timer    clear
0x517890           <0.3.1>    -        clear          <-- SUSPECT: no source, no latched wake
(5 parked, 2 suspect)

Read it as follows.

park='-' with wake_pending clear. No armed wake source – not an fd, not a timer. This is the shape a fiber has while waiting for an xtc_aio_* completion to be reaped and dispatched. With no latched wake either, and if the operation has demonstrably finished, nothing remains to re-deliver the wake. Report it.
park='-' with WAKE_PENDING. A wake arrived while the proc was still deciding to park, and was latched for the scheduler to consume. Transient and correct. If it persists across several samples, the consume path is broken – a different bug, so say which you saw.
park='fd' / 'timer' / 'mailbox'. Waiting on a real source. Check the source before suspecting the runtime: is the fd actually readable, has the deadline actually passed?

Sample it three times about a second apart, alongside thread apply all bt. A genuine strand is byte-identical every time and burns no CPU; an infinite loop is also identical but pins a core. That pair of observations is what tells them apart, and it is the single most useful thing to include in a bug report.

Long-lived idle receivers look exactly like suspects. A fiber blocked in xtc_recv(..., -1) forever – a supervisor, an acceptor, any service fiber – parks with no armed source and no latched wake, which is the same shape as a lost wake. xtc-stranded excludes procs with local_id == 0 (the per-loop service fiber) and says so in its tally, because a consumer found 31 of 33 reported “suspects” were exactly those. But if your own service fibers have a non-zero local_id, they will still be listed: cross-check the suspects against what you know parks forever by design before reporting anything.

One caution: a suspect is not yet a defect. Confirm the wake source really completed first. For an xtc_aio park under the io_uring backend, ls /proc/<pid>/task | wc -l and the presence of iou-wrk-* threads (check their comm) show the kernel accepted and serviced the submission; a still-pending syscall is not a strand.

Recipe: a stuck or growing mailbox (the classic)

Symptom: memory climbs; one proc lags; eventually the OOM killer.

(gdb) xtc-procs        # find the proc with the large/growing mbox

A proc whose mbox is large and whose peak keeps rising is a consumer that cannot keep up with its producers (Shift 4). Confirm the senders:

(gdb) xtc-mailbox 0x55...   # the from= column identifies producers

Fixes are architectural: bound the mailbox (mailbox_cap), convert fire-and-forget xtc_send to a request/reply xtc_svr_call (which rate-limits the producer to the consumer’s pace), or shed load at a watermark. In production, the same depth is visible without a debugger via the inspection API / xtc_stats counters.

Recipe: tracing what caused what (causal message trace)

Symptom: a request misbehaves and the cause is spread across several procs – you can see the symptom in one mailbox but not which message, from which proc, set it off. This is the hardest class of bug in a message-passing system: “who sent this, and what caused that.”

libxtc’s seq_trace answers it. It is an opt-in, bounded ring of send/receive events, each stamped with a hybrid logical clock (HLC) so the causal order survives even when the wall-clock order the records arrived in lies. It is OFF by default and costs a single relaxed atomic load on the message hot path when disabled, so you turn it on for a specific investigation and off again – it is not an always-on tax.

From inside the program, enable it, run the suspect workload, and dump the records in causal order (xtc_trace(3)):

int was_on = xtc_trace_enable(1);   /* opt in */
xtc_trace_reset();                  /* start from a clean ring */
run_suspect_workload();

static int on_rec(const xtc_trace_rec_t *r, void *u) {
    (void)u;
    printf("HLC=%llu kind=%d self=%llu peer=%llu cause=%llu\n",
           (unsigned long long)r->hlc, r->kind,
           (unsigned long long)r->self,
           (unsigned long long)r->peer,
           (unsigned long long)r->cause);
    return 0;                       /* 0 = keep going */
}
xtc_trace_dump(on_rec, NULL);       /* visits in HLC-ascending order */
if (!was_on)
    xtc_trace_enable(0);            /* stop paying for it */

From a stopped program or a core dump, the same ring is in the debugger:

(gdb) xtc-trace
HLC now: 1700000042
hlc          kind  self         peer         cause/detail
HLC100       SEND  self=0x..a   peer=0x..b               detail=64
HLC205       RECV  self=0x..b   peer=0x..a   cause=100    detail=64
HLC206       SEND  self=0x..b   peer=0x..c               detail=16
HLC310       RECV  self=0x..c   peer=0x..b   cause=206    detail=16
(4 events)

Read it by following the cause edges. A RECV’s cause is the HLC of the SEND that produced the message it received, so a RECV with cause=100 is the delivery of the SEND stamped HLC100. Chain them: proc A sends (HLC100), proc B receives it (cause=100) and in turn sends to C (HLC206), C receives that (cause=206). That chain is the path of one request as it fanned out across A, B, and C – the trail you could not see in any single mailbox.

The HLC order is the truth. A single global clock ticks once per traced event (its high 48 bits are microseconds, its low 16 a logical counter), so a send’s stamp is always strictly less than the stamp of the receive it causes. When two cores disagree about which message landed first – because wall-clock arrival across threads is not a reliable order – sort by HLC and the happens-before is restored. Both xtc_trace_dump and xtc-trace already present the records in HLC-ascending order for exactly this reason.

The ring is bounded and overwrites its oldest records when full, so if the interesting events scroll off, narrow the window: xtc_trace_reset immediately before the action, and dump immediately after.

Recipe: a leak

libxtc can attribute live allocations to the proc that made them. In the program, early:

xtc_alloc_audit_enable(1);

then at any point (or from a test):

size_t live_bytes, live_count;
xtc_alloc_audit_proc_leaks(suspect_pid, &live_bytes, &live_count);

A proc that has exited but whose live_count is nonzero leaked. This is the recon_alloc / bin_leak analog. Globally, xtc_alloc_audit_stats(&bytes, &count) is the total live footprint.

Recipe: a fiber that ran too long (latency spikes, p99)

Turn on the cooperative-yield watchdog:

xtc_yield_set_budget(loop, 5LL * 1000 * 1000);   /* 5ms quantum */

Then xtc_yield_due_count(loop) counts how often a fiber blew its budget, and xtc_yield_check() is the in-fiber “am I over budget?” query a long computation can poll to decide to yield or to honor a cancellation. A high due-count points at a proc that needs to yield more or offload (Shift 1).

Recipe: a contained fault (R1)

If a proc took a SIGSEGV/SIGBUS and was contained, its recovery frame records it:

(gdb) xtc-proc 0x55...
  recovery     : armed=1 fired=1 crit_depth=0

fired=1 means the fault handler ran and the proc was unwound to its recovery point (or aborted if crit_depth > 0, preserving critical-section semantics). The faulting backtrace is on the thread at fault time; in a core dump from the abort path it is the top frame.

Recipe: who holds / waits on a lock

For lock-order bugs, libxtc has an opt-in WITNESS-style tracker on xtc_lwlock:

xtc_lwlock_track_enable(1);
/* ... run the workload ... */
int v = xtc_lwlock_track_violations();   /* lock-order inversions seen */

It records the acquire-order graph keyed by tranche and reports inversions, the classic source of deadlocks when you must hold more than one lock. Prefer, of course, not holding two (Shift 3).

Naming procs in the output

libxtc does not store a name on the proc, so xtc-procs shows pids. If you register procs by name (xtc_reg_register(reg, "wal", pid)), cross-reference with xtc_reg_whereis. A future revision may add a short name field to the proc for zero-setup labels.

Programmatic observability (no debugger)

For dashboards and always-on monitoring rather than a stuck-process post-mortem:

xtc_stats – counters and latency histograms; dump them in Prometheus text form with xtc_metrics_dump_prometheus(fd).
xtc_inspect – xtc_inspect_procs(cb, user) and xtc_proc_info(pid, &info): the in-process form of the debugger’s xtc-procs, snapshotting every live proc (or one by pid) without a debugger attached.
xtc_exec_loop_stats(exec, i, &st) – per-loop tasks_run / steals (scheduler utilization).
xtc_reg – a name registry to enumerate well-known procs.
xtc_pdict – a per-proc dictionary for ad-hoc state you want to read back.

xtc_inspect_procs is the data behind a “SHOW PROCESSES” admin command: it walks the same per-loop slot tables xtc-procs does, takes a best-effort snapshot under their locks, then runs your callback after the locks are released (so the callback may call back into the proc and loop APIs). Mailbox depth (info.mbox_len) is the vital sign, exactly as in the debugger view:

static int show(const xtc_proc_info_t *p, void *u) {
    (void)u;
    printf("pid=%llu mbox=%zu peak=%zu state=%d\n",
           (unsigned long long)p->pid,
           p->mbox_len, p->mbox_peak, p->run_state);
    return 0;            /* 0 = keep going, nonzero = stop */
}
xtc_inspect_procs(show, NULL);

Link/monitor topology is deliberately absent from the live snapshot (a proc mutates those lists lock-free, so a cross-thread walk would race); for that you still need the debugger’s xtc-proc, which runs against a stopped program. See xtc_inspect(3).

The causal message trace (libxtc’s seq_trace) is shipped and opt-in; see the recipe above and xtc_trace(3).

In-process dumps without a debugger (xtc_dump / XTC_PANIC)

When you cannot attach a debugger or collect a core – a container with core_pattern disabled, a CI runner, a customer site – the same proc, loop, and mailbox view is available from inside the program:

#include "xtc_dump.h"

xtc_dump(STDERR_FILENO);   /* loops, procs, mailbox depths, backtrace */

The output uses the same field names as xtc-procs/xtc-loops, so a dump and a gdb session read alike:

=== xtc runtime dump ===
thread backtrace (4 frames):
...
loops:
  loop 0  procs=2 alive=2 tasks_run=3 steals=0
procs:
  <0.0.1> parked    park=timer   mbox=2/4096 peak=2 recv=2 drop=0
  <0.1.1> running   park=-       mbox=0/4096 peak=0 recv=0 drop=0
=== end dump ===

The C backtrace is symbolized on glibc, macOS, and the BSDs (execinfo) and on musl via the compiler’s builtin _Unwind_Backtrace (frames symbolized best-effort via dladdr), which needs no third-party library. The optional libunwind backend (--with-libunwind=yes) is a fallback for the rare target whose toolchain ships neither execinfo nor a working _Unwind_Backtrace. Where none is available the frames are addresses-only, and on a platform with no backend at all the dump prints thread backtrace: unavailable (no backtrace backend; attach a debugger to a core) and still reports the full proc/loop/mailbox state. The Windows DbgHelp backend is compiled and reviewed but has not yet been runtime-verified on a Windows host.

To abort with that dump on a failed invariant, use the panic/assert macros (always compiled in – they are diagnostics, not debug-only):

XTC_ASSERT(frame->pin > 0);
XTC_ASSERT_F(n <= cap, "overflow: n=%zu cap=%zu", n, cap);
XTC_PANIC("unreachable: state=%d", st);

To dump automatically on a fault, install the crash handler at startup:

xtc_crash_handler_install();   /* SIGSEGV/SIGBUS/SIGABRT -> dump -> core */

The dump reports each proc’s state and mailbox, not the C stack of every parked fiber (those live in saved coroutine contexts, not on a live OS thread – only the faulting thread’s C stack is unwound). This makes it strongest for the hang/wedge class above (“which proc is waiting on whom”); for deep per-fiber stacks attach a debugger to the core. See xtc_dump(3).

The async causal trace (how a fiber GOT here)

The dump above tells you what a fiber’s state is now – parked on a timer, mailbox depth 2. It does not tell you the path of suspensions that led there. The async causal trace adds that: a small fixed-capacity ring (16 entries) per proc records each suspend/resume boundary – a mailbox receive, a timer sleep, an fd wait, and the matching resume – with a static site label. It is libxtc’s analog of Cats Effect’s per-fiber async-stack-trace ring, and it complements the current-state dump instead of replacing it: together they answer both what is this fiber doing and how did it get here at once.

It lives in xtc_trace.h alongside the causal message trace, but is a separate switch – one records the chain of messages between procs, the other the chain of park/resume boundaries within one proc. It is OFF by default and zero-cost when off: a disabled trace is one relaxed-atomic load and a branch at the suspend/resume boundary and touches no memory, so a default build records nothing. Each recorded event is one index bump and a store on the owning fiber – core-private, single-writer, no lock – so it never writes a shared cache line on the hot path even when enabled. Enable it to debug a stuck or mis-scheduled fiber, then turn it back off:

int
main(void)
{
	xtc_loop_t *loop = NULL;
	xtc_proc_opts_t o = { 0 };

	xtc_trace_causal_enable(1);         /* opt in (off by default) */

	if (xtc_loop_init(&loop) != XTC_OK)
		return 1;
	if (xtc_proc_spawn(loop, worker, NULL, &o, &g_worker) != XTC_OK)
		return 1;
	if (xtc_loop_run(loop) != XTC_OK)
		return 1;
	(void)xtc_loop_fini(loop);

	xtc_trace_causal_enable(0);         /* stop paying for it */
	return 0;
}

Tested source: docs/_includes/snippets/08_causal_trace.c

Read one proc’s chain oldest-first with xtc_trace_causal_dump:

/* One record of the fiber's park/resume chain, oldest-first. */
static const char *
kind_name(int k)
{
	switch (k) {
	case XTC_CAUSAL_PARK_MAILBOX: return "park:mailbox";
	case XTC_CAUSAL_PARK_TIMER:   return "park:timer";
	case XTC_CAUSAL_PARK_FD:      return "park:fd";
	case XTC_CAUSAL_RESUME:       return "resume";
	default:                      return "?";
	}
}

static int
on_causal(const xtc_causal_rec_t *rec, void *user)
{
	(void)user;
	printf("  %s @ %s\n", kind_name(rec->kind),
	    rec->site != NULL ? rec->site : "?");
	return 0;                       /* 0 = keep going */
}

Tested source: docs/_includes/snippets/08_causal_trace.c

When the trace is enabled, xtc_dump splices each proc’s recent chain onto its state line – a causal: line of kind@site entries, oldest-first – so a dump reads:

procs:
  <0.0.1> parked    park=mailbox mbox=0/4096 peak=1 recv=1 drop=0
    causal: park:mailbox@__do_recv -> resume@__do_recv -> park:mailbox@__do_recv

The chain reads left-to-right, oldest-first: this fiber parked awaiting a message, resumed when one arrived, and parked again awaiting the next. A proc that has already exited has been reaped, so xtc_trace_causal_dump returns XTC_E_NOTFOUND for it – read a fiber’s chain while it is still alive (or from the dump, which walks the live set). See xtc_trace(3).

When you file a libxtc bug

Include the output of xtc-loops, xtc-procs, and thread apply all bt, plus whether the program holds any lock across a recv/await and whether any proc does CPU work without yielding. Those three artifacts resolve most reports immediately – usually to one of the anti-patterns in docs/guide/transitioning.md.

Debugging and observing

Loading the tools

GDB (and VS Code / CLion, which drive GDB)

LLDB (and VS Code / CLion on macOS)

The five-minute triage

The commands, with example output

xtc-loops – the scheduler at a glance

xtc-procs – the observer process table

xtc-proc ADDR – one process in detail

xtc-mailbox ADDR – who is queued, and from whom

xtc-self – map a thread back to its proc

xtc-trace – the causal message trace

Recipe: the whole program hangs

When a completion seems to have been lost

When a fiber looks like it is never resumed

Recipe: a stuck or growing mailbox (the classic)

Recipe: tracing what caused what (causal message trace)

Recipe: a leak

Recipe: a fiber that ran too long (latency spikes, p99)

Recipe: a contained fault (R1)

Recipe: who holds / waits on a lock

Naming procs in the output

Programmatic observability (no debugger)

In-process dumps without a debugger (xtc_dump / XTC_PANIC)

The async causal trace (how a fiber GOT here)

When you file a libxtc bug

`xtc-loops` – the scheduler at a glance

`xtc-procs` – the observer process table

`xtc-proc ADDR` – one process in detail

`xtc-mailbox ADDR` – who is queued, and from whom

`xtc-self` – map a thread back to its proc

`xtc-trace` – the causal message trace