xtc_trace(3)

---

xtc_trace(3)

causal message tracing (libxtc’s seq_trace)

XTC_TRACE(3) Library Functions Manual XTC_TRACE(3)

xtc_trace_enable, xtc_trace_reset, xtc_trace_dump, xtc_hlc_now — causal message tracing (libxtc's seq_trace)

libxtc

#include <xtc.h>
#include <xtc_trace.h>

typedef int (*xtc_trace_fn)(const xtc_trace_rec_t *rec, void *user);

int
xtc_trace_enable(int on);

int
xtc_trace_reset(void);

int
xtc_trace_dump(xtc_trace_fn cb, void *user);

uint64_t
xtc_hlc_now(void);

xtc_trace is libxtc's : a bounded, opt-in ring of message-passing events -- each send and receive between procs -- stamped so that the causal order of one request can be reconstructed across procs and cores even when the wall-clock order in which the records arrived lies. It is the analog of Erlang's trace token that rides along message sends, letting you answer “who sent this, and what caused that” in a system where the answer is spread across many mailboxes.

Every traced event is stamped with a hybrid logical clock (): a 64-bit value whose high 48 bits are a monotonic physical-time component (microseconds) and whose low 16 bits are a logical counter. A single global clock ticks once per traced event, so a send's stamp is always strictly less than the stamp of the receive it causes. Sorting the records by HLC therefore yields a true happens-before order, not a thread-local arrival order: when two cores disagree about which message landed first, the HLC is the truth.

The causal edge is made explicit, not merely implied by ordering: a XTC_TRACE_RECV record carries the originating send's HLC stamp in its cause field. Following cause from a receive back to the matching send, and from that send's proc to the receive that triggered it, walks the chain of one request as it fans out across procs.

Tracing is OFF by default. When disabled it costs a single relaxed atomic load on the message hot path, so a production build that never enables it pays essentially nothing. When enabled it serializes ring writes under a lock and overwrites the oldest records once the ring is full. Enable it to debug a specific problem -- as you would reach for or on the BEAM -- not as an always-on production tax.

() turns tracing on when on is non-zero and off when it is zero. It returns the previous state (non-zero if tracing was on), so a caller can save and restore it.

() drops all buffered trace records, leaving the enable state unchanged. It returns XTC_OK. Call it before a workload to ensure the ring holds only the events of interest.

() visits the buffered records in causal (HLC-ascending) order, invoking cb once per record with a pointer to the record and the caller's user pointer. The callback returns 0 to continue or any non-zero value to stop the walk early. The callbacks run the internal trace lock is released, so a callback may freely call back into libxtc -- including (), the inspection API, or xtc_trace_dump() itself -- without risk of self-deadlock. It returns the number of records visited, or a negative XTC_E_* code on error.

() returns the current value of the global HLC. It is intended for tests and for display; an application does not normally need it, since the stamps already ride in the trace records.

A trace record is reported in:

typedef struct xtc_trace_rec {
        uint64_t  hlc;     /* this event's HLC stamp */
        uint64_t  cause;   /* RECV: the originating send's HLC; else 0 */
        int       kind;    /* enum xtc_trace_kind */
        xtc_pid_t self;    /* the proc the event happened in */
        xtc_pid_t peer;    /* the other proc (dest / source / child) */
        uint32_t  detail;  /* SEND/RECV: payload bytes; EXIT: reason */
} xtc_trace_rec_t;

The fields are:

hlc
This event's HLC stamp. Records sort by this value into causal order.
cause
For a XTC_TRACE_RECV record, the HLC stamp of the send that produced the received message; 0 for every other kind. This is the explicit causal edge.
kind
One of the enum xtc_trace_kind values below.
self
The proc in which the event happened: the sender for a send, the receiver for a receive, the parent for a spawn, the exiting proc for an exit.
peer
The other proc involved: the destination of a send, the source of a receive, or the spawned child.
detail
For a send or receive, the message payload size in bytes; for an exit, the exit reason.

enum xtc_trace_kind names the event kinds:

self sent a message to peer.
self received a message from peer; cause holds the originating send's stamp.
self (the parent) spawned peer (the child).
self exited; detail holds the exit reason.

The visit callback has the type

typedef int (*xtc_trace_fn)(const xtc_trace_rec_t *rec, void *user);

where rec points at one record (valid only for the duration of the call) and user is the pointer passed to ().

The per-shard HLC used by the MVCC data path is a separate, later use of the same clock idea; it is not the global trace clock described here.

xtc_trace_enable() returns the previous enable state: non-zero if tracing was already on, 0 if it was off.

xtc_trace_reset() returns XTC_OK.

xtc_trace_dump() returns the number of records visited, or a negative XTC_E_* code on error. See xtc_strerror(3).

xtc_hlc_now() returns the current global HLC value.

Enable tracing, run a workload, and print the records in causal order:

static int
on_rec(const xtc_trace_rec_t *r, void *user)
{
        (void)user;
        printf("HLC=%llu kind=%d self=%llu peer=%llu cause=%llu det=%u\n",
            (unsigned long long)r->hlc, r->kind,
            (unsigned long long)r->self,
            (unsigned long long)r->peer,
            (unsigned long long)r->cause, r->detail);
        return (0);                 /* 0 = keep going */
}

int was_on = xtc_trace_enable(1);   /* opt in */
xtc_trace_reset();                  /* start from a clean ring */
run_workload();                     /* exercise the procs in question */

int n = xtc_trace_dump(on_rec, NULL);
printf("%d trace records\n", n);

if (!was_on)
        xtc_trace_enable(0);        /* stop paying for it */

Reading the dump, a XTC_TRACE_RECV whose cause equals the hlc of an earlier XTC_TRACE_SEND is the same message; chaining those edges reconstructs a request's path across procs.

xtc_inspect(3), xtc_proc(3), xtc_stats(3)

The xtc project.

June 1, 2026 Debian

View the mdoc source