xtc_tail(3)

---

xtc_tail(3)

runtime microscope (individual scheduler-event recording)

XTC_TAIL(3) Library Functions Manual XTC_TAIL(3)

xtc_tail_enable, xtc_tail_disable, xtc_tail_reset, xtc_tail_read, xtc_tail_dump, xtc_tail_dump_dial9, xtc_tail_from_env, xtc_tail_spill_dial9, xtc_tail_count, xtc_tail_dropped — runtime microscope (individual scheduler-event recording)

library “libxtc”

#include <xtc_tail.h>

unsigned
xtc_tail_enable(unsigned source_mask);

void
xtc_tail_disable(void);

int
xtc_tail_reset(void);

int
xtc_tail_read(xtc_tail_fn cb, void *user);

int
xtc_tail_dump(int fd);

int
xtc_tail_dump_dial9(int fd);

unsigned
xtc_tail_from_env(void);

int
xtc_tail_spill_dial9(const char *dir);

size_t
xtc_tail_count(void);

uint64_t
xtc_tail_dropped(void);

xtc_tail is a runtime microscope: it records every INDIVIDUAL runtime event, tied to a precise instant, into a bounded ring -- so a degraded window can be diffed against a normal one after the fact. It is the event-level complement to xtc_stats(3) (aggregate counters) and xtc_trace(3) (the causal message trace). Inspired by dial9, "a microscope for Tokio".

Recording is OFF by default and costs a single predictable branch when disabled, so it is safe to compile in for production and turn on only while diagnosing.

() turns on the named event sources -- a bitwise OR of XTC_TAIL_SCHED (proc spawn / exit / wake / run), XTC_TAIL_MSG (message send / receive / mailbox depth high-water), XTC_TAIL_IO, and XTC_TAIL_OS (the latter two reserved for the staged build-out). The mask REPLACES the current one and the previous mask is returned. () turns off all recording. () drops all buffered records.

() visits every buffered record, oldest first, via cb (return nonzero to stop early); the snapshot is stable. () returns how many records are buffered. () writes the buffer to fd as a versioned, compact, PORTABLE binary trace: an xtc_tail_hdr_t header (magic, version, endianness flag, count, base timestamp) written little-endian, followed by one varint/delta-encoded frame per event (kind, source, LEB128 timestamp delta, pid, and detail). The delta-encoded timestamps and varint fields keep it to a handful of bytes per event -- and being explicit little-endian with no struct padding, a trace dumps and decodes identically across architectures -- for a separate offline tool to render.

() writes the same buffered records to fd in the dial9 trace wire format (“TRC\0, version 1”), so a libxtc trace opens in the dial9 GUI viewer with no viewer change. The dial9 viewer keys on a schema NAME string, so each libxtc event kind is registered as a self-describing schema under a name the viewer recognizes where a runtime analogue exists (TaskSpawnEvent, TaskTerminateEvent, WakeEventEvent, PollStartEvent) and under an Xtc* name otherwise (the io_uring reap/submit chain), which the viewer shows as a generic custom event pinned to the right worker and task. Use whichever consumer you have: the dial9 GUI reads xtc_tail_dump_dial9() output, while xtc-tail.py reads () output.

For a DEPLOYED application, two helpers make xtc_tail usable in production with no code change and a trace you can retrieve after the fact. () turns recording on from the environment -- XTC_TAIL_ENABLE=1 (or sched) enables the SCHED source, all enables every source, and unset or a falsy value leaves it off -- so an operator arms the microscope on a running service without a rebuild. () writes the current ring to a new, uniquely-named dial9 segment file (dir/xtc-tail-⟨pid⟩-⟨ns⟩.d9) in an existing directory, which a sidecar or scp(1) ships to wherever the dial9 viewer reads. There is deliberately no background thread: the caller spills when it makes sense -- at shutdown, on a signal, on a health-check trigger, or from its own timer -- and a crash-surviving trace is a spill from a fault handler. Rotation and the on-disk byte budget are the deployment's job (a directory plus a cleanup rule), matching how a dial9 disk buffer is ultimately bounded.

Each xtc_tail_rec_t carries a monotonic timestamp, the source and kind, the proc pid, and a kind-specific detail (an EXIT reason, or the wake-to-run latency in nanoseconds for a RUN event -- the signal that exposes lost or late wakeups).

xtc_tail_enable() returns the previously enabled source mask. xtc_tail_from_env() returns the source mask it enabled (0 if the environment left it off). xtc_tail_reset(), xtc_tail_read(), xtc_tail_dump(), and xtc_tail_dump_dial9() return XTC_OK or a negative XTC_E_* code. xtc_tail_spill_dial9() returns XTC_OK after writing a segment, or a negative XTC_E_* code (for example if dir does not exist or is not writable). xtc_tail_count() returns the buffered record count.

xtc_tail_dropped() returns how many records have been OVERWRITTEN because the ring wrapped (total emitted minus what is still buffered).

Check it before drawing any conclusion from the ABSENCE of an event. With a non-zero dropped count, "this pid has no events" and "this loop never polled" are unfalsifiable -- the events may simply have been evicted, especially by a higher-frequency event on the same ring. Conclusions drawn from events that are PRESENT remain valid regardless.

xtc_tail_disable() returns no value.

Phase 1 implements the SCHED source (proc spawn / exit / park / run with park-to-run latency) and the MSG source (send / receive / mailbox-depth high-water), the in-process read, and the compact portable binary dump. The IO and OS sources, an on-disk spill with rotation, and the offline viewer are staged follow-ons; the record layout and the source mask are stable so they extend the format without a break.

Which parks are recorded, precisely, because the coverage is not uniform: a XTC_TAIL_PARK / XTC_TAIL_RUN pair is emitted for a mailbox () park, for the native async-file-I/O park inside (), (), () and (), and for an xtc_proc_wait_fd() readiness park -- which is the one most workloads spend their off-CPU time in, since every xtc_net(3) accept/read/write, () completion, xtc_osproc(3) reap and () parks through it. For a xtc_proc_wait_fd() park the XTC_TAIL_PARK detail carries the file descriptor, so the event joins against the park_fd column of xtc-procs; for the other two sources it carries the aio opcode or zero. In all cases the XTC_TAIL_RUN detail is the park-to-run latency in nanoseconds.

Parks that are still recorded: () (a pure timer park -- its wake instant is already known from the deadline) and the internal condition-variable waits inside xtc_sync(3).

The intended use of the pair is diagnostic: a XTC_TAIL_PARK with no matching XTC_TAIL_RUN for the same pid is a fiber that went to sleep and was never resumed -- a lost wakeup -- while a matching RUN whose detail (the park-to-run latency in nanoseconds) is large is a late one.

The remaining SCHED kinds form a vocabulary for following an I/O wakeup through the runtime, from the request to the resumed fiber. Each names one stage, so an absent event localizes a lost wakeup to the stage that dropped it. They are listed here in that order.

XTC_TAIL_LOOP_POLL is emitted by the SCHED source after each completed poll of a loop's own I/O ring, even when the poll dispatched nothing (detail = the number of events dispatched). Its pid carries only loop_id; a loop is not a proc. Its purpose is liveness: when a XTC_TAIL_PARK has no matching XTC_TAIL_RUN, a continuing stream of XTC_TAIL_LOOP_POLL for that loop_id proves the loop kept turning and that one fiber was skipped, whereas the absence of any proves the loop itself stopped. Those have different causes and different fixes, and without this event the only way to distinguish them is to infer from the absence of other events -- which cannot tell a dead loop apart from the end of the recording window.

XTC_TAIL_WAKE is emitted from the completion dispatch (the waker side). Its pid carries only the dispatching loop_id -- dispatch has a task, not a proc -- and its detail is the task pointer as an integer, which is the join key against the task column of xtc-procs and against XTC_TAIL_PARK_TASK. An aio XTC_TAIL_PARK cannot be the join target: its detail is the OPCODE, not a task pointer, so XTC_TAIL_PARK_TASK is the event that carries the key.

XTC_TAIL_PARK_TASK is emitted immediately before the XTC_TAIL_PARK it describes, from the same fiber, and carries the PARKING proc's pid with that fiber's task pointer in detail. It exists to make XTC_TAIL_WAKE joinable without displacing the aio PARK's opcode: PARK carries the pid and, for an aio park, the opcode that distinguishes an fdatasync park from a read park, so the task pointer needs its own event rather than overwriting the opcode. All three park sites emit it -- aio, () readiness, and mailbox recv.

XTC_TAIL_REAP records that a completion-queue entry was consumed by a reaping loop, and what tag it resolved to. Its pid carries only the reaping loop_id; its detail is the same task pointer the other two carry, or 0 when the entry was consumed WITHOUT being handed to dispatch. A zero is expected for the wakeup pipe and for a poll-remove cancel, and is also the signature of a completion dropped because its registration was already torn down -- so a zero alone is not a fault; the join is what matters.

Together these events localize a lost wakeup to one stage, and the stages have different fixes:

  • a PARK_TASK whose task has no SUBMIT: no SQE was ever queued for it.
  • a PARK_TASK whose task has a SUBMIT but no REAP: the completion never came back from the ring.
  • a REAP with no WAKE: the reaper consumed it and never handed it to dispatch.
  • a WAKE with no RUN: dispatch ran; the loss is downstream of it.

XTC_TAIL_SUBMIT records that an SQE was handed to the kernel, keyed by the task the request is on behalf of, and XTC_TAIL_SUBMIT_FAIL records that a submission did NOT reach it. Its detail carries the negated errno when the submit was refused, or -- for a SHORT submit, which has no errno -- the number of SQEs left behind, offset by XTC_TAIL_SHORT_SUBMIT_BASE. Both shapes matter: the submit call returns a NEGATIVE errno or the NUMBER of entries it consumed, which can be fewer than were queued, so a short submit looks like success to an errno-only check while an SQE never reached the kernel. Together they split the first bucket above: a completion that never came back may never have been ASKED for. XTC_TAIL_SUBMIT_FAIL is the only kind here whose mere presence is a fault -- every other needs a join to mean anything -- because a fiber parked on a refused request can never be woken.

A ring with no exec-relative loop id -- one belonging to a loop created by a bare () rather than as part of an executor -- reports XTC_TAIL_LOOP_NONE (0xFFFF), which xtc-tail.py renders as none. This is distinct from a fallback of 0, which would be indistinguishable from the executor's real loop 0 and would misattribute an unlabelled ring's events to it.

XTC_TAIL_POLL_FULL records a poll that drained its FULL per-call budget and so may have left completions behind (detail = the budget it filled). XTC_TAIL_LOOP_POLL covers only IDLE polls, so without this a poll that reaped some-but-not-all events was invisible -- and "I polled, took what I could, and left the rest" is exactly the state to look for when a ring holds a pinned backlog while its loop is demonstrably alive. It is low-volume by construction: it fires only when one poll hits its ceiling, never on every poll.

xtc-tail.py --strands performs this classification. Every bucket is an ABSENCE claim, so check () first: in a wrapped ring an absence may only mean the event was evicted. XTC_TAIL_REAP is the highest-volume event here, one per completion, so prefer a short capture window.

xtc_tail_enable(XTC_TAIL_SCHED);
/* run the workload */
int fd = open("trace.xtcl", O_WRONLY|O_CREAT|O_TRUNC, 0644);
xtc_tail_dump(fd);          /* versioned binary, for the offline viewer */
close(fd);
xtc_tail_disable();

xtc_stats(3), xtc_trace(3), xtc_inspect(3), xtc(7)

September 12, 2026 Debian

View the mdoc source