xtc_trace(3)

---

xtc_trace(3)

causal message tracing (libxtc’s seq_trace) and the per-fiber async causal trace

XTC_TRACE(3) Library Functions Manual XTC_TRACE(3)

xtc_trace_enable, xtc_trace_reset, xtc_trace_dump, xtc_hlc_now, xtc_trace_causal_enable, xtc_trace_causal_dump — causal message tracing (libxtc's seq_trace) and the per-fiber async causal trace

libxtc

#include <xtc.h>
#include <xtc_trace.h>

typedef int (*xtc_trace_fn)(const xtc_trace_rec_t *rec, void *user);

int
xtc_trace_enable(int on);

int
xtc_trace_reset(void);

int
xtc_trace_dump(xtc_trace_fn cb, void *user);

uint64_t
xtc_hlc_now(void);

int
xtc_trace_causal_enable(int on);

int
xtc_trace_causal_dump(xtc_pid_t pid, xtc_causal_fn cb, void *user);

xtc_trace is libxtc's : a bounded, opt-in ring of message-passing events -- each send and receive between procs -- stamped so that the causal order of one request can be reconstructed across procs and cores even when the wall-clock order in which the records arrived lies. It is the analog of Erlang's trace token that rides along message sends, letting you answer “who sent this, and what caused that” in a system where the answer is spread across many mailboxes.

Every traced event is stamped with a hybrid logical clock (): a 64-bit value whose high 48 bits are a monotonic physical-time component (microseconds) and whose low 16 bits are a logical counter. A single global clock ticks once per traced event, so a send's stamp is always strictly less than the stamp of the receive it causes. Sorting the records by HLC therefore yields a true happens-before order, not a thread-local arrival order: when two cores disagree about which message landed first, the HLC is the truth.

The causal edge is made explicit, not merely implied by ordering: a XTC_TRACE_RECV record carries the originating send's HLC stamp in its cause field. Following cause from a receive back to the matching send, and from that send's proc to the receive that triggered it, walks the chain of one request as it fans out across procs.

Tracing is OFF by default. When disabled it costs a single relaxed atomic load on the message hot path, so a production build that never enables it pays essentially nothing. When enabled it serializes ring writes under a lock and overwrites the oldest records once the ring is full. Enable it to debug a specific problem -- as you would reach for or on the BEAM -- not as an always-on production tax.

() turns tracing on when on is non-zero and off when it is zero. It returns the previous state (non-zero if tracing was on), so a caller can save and restore it.

() drops all buffered trace records, leaving the enable state unchanged. It returns XTC_OK. Call it before a workload to ensure the ring holds only the events of interest.

() visits the buffered records in causal (HLC-ascending) order, invoking cb once per record with a pointer to the record and the caller's user pointer. The callback returns 0 to continue or any non-zero value to stop the walk early. The callbacks run the internal trace lock is released, so a callback may freely call back into libxtc -- including (), the inspection API, or xtc_trace_dump() itself -- without risk of self-deadlock. It returns the number of records visited, or a negative XTC_E_* code on error.

() returns the current value of the global HLC. It is intended for tests and for display; an application does not normally need it, since the stamps already ride in the trace records.

A trace record is reported in:

typedef struct xtc_trace_rec {
        uint64_t  hlc;     /* this event's HLC stamp */
        uint64_t  cause;   /* RECV: the originating send's HLC; else 0 */
        int       kind;    /* enum xtc_trace_kind */
        xtc_pid_t self;    /* the proc the event happened in */
        xtc_pid_t peer;    /* the other proc (dest / source / child) */
        uint32_t  detail;  /* SEND/RECV: payload bytes; EXIT: reason */
} xtc_trace_rec_t;

The fields are:

hlc
This event's HLC stamp. Records sort by this value into causal order.
cause
For a XTC_TRACE_RECV record, the HLC stamp of the send that produced the received message; 0 for every other kind. This is the explicit causal edge.
kind
One of the enum xtc_trace_kind values below.
self
The proc in which the event happened: the sender for a send, the receiver for a receive, the parent for a spawn, the exiting proc for an exit.
peer
The other proc involved: the destination of a send, the source of a receive, or the spawned child.
detail
For a send or receive, the message payload size in bytes; for an exit, the exit reason.

enum xtc_trace_kind names the event kinds:

self sent a message to peer.
self received a message from peer; cause holds the originating send's stamp.
self (the parent) spawned peer (the child).
self exited; detail holds the exit reason.

The visit callback has the type

typedef int (*xtc_trace_fn)(const xtc_trace_rec_t *rec, void *user);

where rec points at one record (valid only for the duration of the call) and user is the pointer passed to ().

The per-shard HLC used by the MVCC data path is a separate, later use of the same clock idea; it is not the global trace clock described here.

Where the message trace above records the causal chain of messages procs, the records the suspend/resume chain one proc: the ordered call sites at which a fiber parked and resumed (a mailbox receive, a timer sleep, an fd wait). It answers a different question -- “how did THIS fiber get here” -- and complements the current-state view of xtc_dump(3), which reports what a proc's state is now but not the path of suspensions that led to it. It is the C analog of Cats Effect's per-fiber async-stack-trace ring, which is spliced onto a fault or fiber dump.

Each proc carries a small fixed-capacity ring (16 entries) of records:

typedef struct xtc_causal_rec {
        int          kind;    /* enum xtc_causal_kind */
        const char  *site;    /* static label (e.g. __func__) */
} xtc_causal_rec_t;
where kind is one of XTC_CAUSAL_PARK_MAILBOX, XTC_CAUSAL_PARK_TIMER, XTC_CAUSAL_PARK_FD, or XTC_CAUSAL_RESUME, and site is a static string label (never freed). Once the ring is full the oldest record is overwritten, so the window holds the most recent 16 boundaries.

The async causal trace is OFF by default and ZERO-COST when off: a disabled trace is a single relaxed-atomic load and a branch at the suspend/resume boundary and touches no memory, so a default build records nothing. Each recorded event is one index bump and a store on the owning fiber -- core-private, single-writer, no lock, no atomic on the record, and no allocation after spawn -- so a fiber never writes a shared cache line on the hot path even when the trace is on. The recorder reads no clock and draws no randomness, so a run remains deterministic under the simulation backend with the trace enabled.

() turns the async causal trace on when on is non-zero and off when it is zero, returning the previous state (1 on, 0 off). This is a separate switch from xtc_trace_enable(), which governs the message trace above.

() visits the causal records of the proc pid oldest-first -- the fiber's recent park/resume chain -- invoking cb once per record. The callback returns 0 to continue or non-zero to stop early. It returns the number of records visited, XTC_E_NOTFOUND if no live proc has that pid (a proc that has exited has been reaped), or XTC_E_INVAL on a NULL callback. When the trace is disabled a live proc simply has an empty ring.

xtc_dump(3) splices each proc's recent causal chain onto its state line when the async causal trace is enabled -- a ‘causal:’ line of ‘kind@site’ entries, oldest-first -- so a dump answers both “what is this fiber's state now” and “how did it get here” at once.

xtc_trace_enable() returns the previous enable state: non-zero if tracing was already on, 0 if it was off.

xtc_trace_reset() returns XTC_OK.

xtc_trace_dump() returns the number of records visited, or a negative XTC_E_* code on error. See xtc_strerror(3).

xtc_hlc_now() returns the current global HLC value.

Enable tracing, run a workload, and print the records in causal order:

static int
on_rec(const xtc_trace_rec_t *r, void *user)
{
        (void)user;
        printf("HLC=%llu kind=%d self=%llu peer=%llu cause=%llu det=%u\n",
            (unsigned long long)r->hlc, r->kind,
            (unsigned long long)r->self,
            (unsigned long long)r->peer,
            (unsigned long long)r->cause, r->detail);
        return (0);                 /* 0 = keep going */
}

int was_on = xtc_trace_enable(1);   /* opt in */
xtc_trace_reset();                  /* start from a clean ring */
run_workload();                     /* exercise the procs in question */

int n = xtc_trace_dump(on_rec, NULL);
printf("%d trace records\n", n);

if (!was_on)
        xtc_trace_enable(0);        /* stop paying for it */

Reading the dump, a XTC_TRACE_RECV whose cause equals the hlc of an earlier XTC_TRACE_SEND is the same message; chaining those edges reconstructs a request's path across procs.

xtc_inspect(3), xtc_proc(3), xtc_stats(3)

The xtc project.

June 1, 2026 Debian

View the mdoc source