xtc_tail(3)

---

xtc_tail(3)

runtime microscope (individual scheduler-event recording)

XTC_TAIL(3) Library Functions Manual XTC_TAIL(3)

xtc_tail_enable, xtc_tail_disable, xtc_tail_reset, xtc_tail_read, xtc_tail_dump, xtc_tail_count, xtc_tail_dropped — runtime microscope (individual scheduler-event recording)

library “libxtc”

#include <xtc_tail.h>

unsigned
xtc_tail_enable(unsigned source_mask);

void
xtc_tail_disable(void);

int
xtc_tail_reset(void);

int
xtc_tail_read(xtc_tail_fn cb, void *user);

int
xtc_tail_dump(int fd);

size_t
xtc_tail_count(void);

uint64_t
xtc_tail_dropped(void);

xtc_tail is a runtime microscope: it records every INDIVIDUAL runtime event, tied to a precise instant, into a bounded ring -- so a degraded window can be diffed against a normal one after the fact. It is the event-level complement to xtc_stats(3) (aggregate counters) and xtc_trace(3) (the causal message trace). Inspired by dial9, "a microscope for Tokio".

Recording is OFF by default and costs a single predictable branch when disabled, so it is safe to compile in for production and turn on only while diagnosing.

() turns on the named event sources -- a bitwise OR of XTC_TAIL_SCHED (proc spawn / exit / wake / run), XTC_TAIL_MSG (message send / receive / mailbox depth high-water), XTC_TAIL_IO, and XTC_TAIL_OS (the latter two reserved for the staged build-out). The mask REPLACES the current one and the previous mask is returned. () turns off all recording. () drops all buffered records.

() visits every buffered record, oldest first, via cb (return nonzero to stop early); the snapshot is stable. () returns how many records are buffered. () writes the buffer to fd as a versioned, compact, PORTABLE binary trace: an xtc_tail_hdr_t header (magic, version, endianness flag, count, base timestamp) written little-endian, followed by one varint/delta-encoded frame per event (kind, source, LEB128 timestamp delta, pid, and detail). The delta-encoded timestamps and varint fields keep it to a handful of bytes per event -- and being explicit little-endian with no struct padding, a trace dumps and decodes identically across architectures -- for a separate offline tool to render.

Each xtc_tail_rec_t carries a monotonic timestamp, the source and kind, the proc pid, and a kind-specific detail (an EXIT reason, or the wake-to-run latency in nanoseconds for a RUN event -- the signal that exposes lost or late wakeups).

xtc_tail_enable() returns the previously enabled source mask. xtc_tail_reset(), xtc_tail_read(), and xtc_tail_dump() return XTC_OK or a negative XTC_E_* code. xtc_tail_count() returns the buffered record count.

xtc_tail_dropped() returns how many records have been OVERWRITTEN because the ring wrapped (total emitted minus what is still buffered).

Check it before drawing any conclusion from the ABSENCE of an event. With a non-zero dropped count, "this pid has no events" and "this loop never polled" are unfalsifiable -- the events may simply have been evicted. A consumer reported two verdicts that were exactly this artifact: 23 of 33 loops appeared to have stopped polling when their events had merely been overwritten by a higher-frequency event. Conclusions drawn from events that are PRESENT remain valid regardless.

xtc_tail_disable() returns no value.

Phase 1 implements the SCHED source (proc spawn / exit / park / run with park-to-run latency) and the MSG source (send / receive / mailbox-depth high-water), the in-process read, and the compact portable binary dump. The IO and OS sources, an on-disk spill with rotation, and the offline viewer are staged follow-ons; the record layout and the source mask are stable so they extend the format without a break.

Which parks are recorded, precisely, because the coverage is not uniform: a XTC_TAIL_PARK / XTC_TAIL_RUN pair is emitted for a mailbox () park, for the native async-file-I/O park inside (), (), () and (), and for an xtc_proc_wait_fd() readiness park -- which is the one most workloads spend their off-CPU time in, since every xtc_net(3) accept/read/write, () completion, xtc_osproc(3) reap and () parks through it. For a xtc_proc_wait_fd() park the XTC_TAIL_PARK detail carries the file descriptor, so the event joins against the park_fd column of xtc-procs; for the other two sources it carries the aio opcode or zero. In all cases the XTC_TAIL_RUN detail is the park-to-run latency in nanoseconds.

Parks that are still recorded: () (a pure timer park -- its wake instant is already known from the deadline) and the internal condition-variable waits inside xtc_sync(3).

The intended use of the pair is diagnostic: a XTC_TAIL_PARK with no matching XTC_TAIL_RUN for the same pid is a fiber that went to sleep and was never resumed -- a lost wakeup -- while a matching RUN whose detail (the park-to-run latency in nanoseconds) is large is a late one.

XTC_TAIL_LOOP_POLL is emitted by the SCHED source after each completed poll of a loop's own I/O ring, even when the poll dispatched nothing (detail = the number of events dispatched). Its pid carries only loop_id; a loop is not a proc. Its purpose is liveness: when a XTC_TAIL_PARK has no matching XTC_TAIL_RUN, a continuing stream of XTC_TAIL_LOOP_POLL for that loop_id proves the loop kept turning and that one fiber was skipped, whereas the absence of any proves the loop itself stopped. Those have different causes and different fixes, and without this event the only way to distinguish them is to infer from the absence of other events -- which cannot tell a dead loop apart from the end of the recording window.

XTC_TAIL_WAKE is emitted from the completion dispatch (the waker side). Its pid carries only the dispatching loop_id -- dispatch has a task, not a proc -- and its detail is the task pointer as an integer, which is the join key against the task column of xtc-procs and against XTC_TAIL_PARK_TASK.

: an earlier version of this page said to join XTC_TAIL_WAKE against the detail of the aio XTC_TAIL_PARK, claiming it carried a task pointer. It does not -- an aio PARK's detail is the OPCODE. The two key spaces do not intersect, and that instruction sent a consumer chasing a join that could not be performed. XTC_TAIL_PARK_TASK exists because of it.

XTC_TAIL_PARK_TASK is emitted immediately before the XTC_TAIL_PARK it describes, from the same fiber, and carries the PARKING proc's pid with that fiber's task pointer in detail. It exists solely to make XTC_TAIL_WAKE joinable: PARK carries the pid and, for an aio park, the opcode a consumer needs to tell an fdatasync park from a read park, so overwriting that opcode with the task pointer would trade one key for the other. Emitting both gives both. All three park sites emit it -- aio, () readiness, and mailbox recv.

XTC_TAIL_REAP records that a completion-queue entry was consumed by a reaping loop, and what tag it resolved to. Its pid carries only the reaping loop_id; its detail is the same task pointer the other two carry, or 0 when the entry was consumed WITHOUT being handed to dispatch. A zero is expected for the wakeup pipe and for a poll-remove cancel, and is also the signature of a completion dropped because its registration was already torn down -- so a zero alone is not a fault; the join is what matters.

Together the four events localize a lost wakeup to one step, and the steps have different fixes:

  • a PARK_TASK whose task has no SUBMIT: no SQE was ever queued for it.
  • a PARK_TASK whose task has a SUBMIT but no REAP: the completion never came back from the ring.
  • a REAP with no WAKE: the reaper consumed it and never handed it to dispatch.
  • a WAKE with no RUN: dispatch ran; the loss is downstream of it.

XTC_TAIL_SUBMIT records that an SQE was handed to the kernel, keyed by the task the request is on behalf of, and XTC_TAIL_SUBMIT_FAIL records that a submission did NOT reach it. Its detail carries the negated errno when the submit was refused, or -- for a SHORT submit, which has no errno -- the number of SQEs left behind, offset by XTC_TAIL_SHORT_SUBMIT_BASE. Both shapes matter: the submit call returns a NEGATIVE errno or the NUMBER of entries it consumed, which can be fewer than were queued, so a short submit looks like success to an errno-only check while an SQE never reached the kernel. Together they split the first bucket above: a completion that never came back may never have been ASKED for. XTC_TAIL_SUBMIT_FAIL is the only kind here whose mere presence is a fault -- every other needs a join to mean anything -- because a fiber parked on a refused request can never be woken.

xtc-tail.py --strands performs this classification. Every bucket is an ABSENCE claim, so check () first: in a wrapped ring an absence may only mean the event was evicted. XTC_TAIL_REAP is the highest-volume event here, one per completion, so prefer a short capture window.

xtc_tail_enable(XTC_TAIL_SCHED);
/* run the workload */
int fd = open("trace.xtcl", O_WRONLY|O_CREAT|O_TRUNC, 0644);
xtc_tail_dump(fd);          /* versioned binary, for the offline viewer */
close(fd);
xtc_tail_disable();

xtc_stats(3), xtc_trace(3), xtc_inspect(3), xtc(7)

July 10, 2026 Debian

View the mdoc source