Locks and synchronization
Which synchronization primitive to reach for, and when.
---This reference documents every locking and synchronization primitive xtc provides, organized by the layer that hosts it. The audience is a programmer choosing a primitive for a particular use, or porting code from another system into the xtc framework.
The presentation moves bottom-up. Lower-layer primitives are simpler, more constrained, and faster. Higher-layer primitives build on those below them and offer richer semantics, sometimes at the cost of additional state or runtime overhead. Choose the lowest layer that meets the requirement.
The treatment of the lock manager (xtc_lockmgr) in section 4 is
deliberately patterned on the Berkeley DB locking subsystem
documentation, which is concise and accurate. The deadlock
detection material in particular follows the BDB approach
because the algorithm is the BDB algorithm.
Contents
- Layer L0:
__os_*system primitives - Layer L3 sync:
xtc_syncand friends - Layer L3 reader-writer:
xtc_lwlock,xtc_lrlock,xtc_rcu - Layer L3 lock manager:
xtc_lockmgr - Choosing a primitive
- Locking and the coroutine model
Layer L0: __os_* system primitives
Layer L0 wraps the operating system’s native synchronization
primitives. These are the fastest synchronization mechanisms
xtc offers because they translate directly to the kernel’s
implementation: a __os_mutex_lock is one pthread_mutex_lock
call, no scheduler indirection, no fairness queueing, no
deadlock checks. They are also the most dangerous because they
have no awareness of xtc tasks; a thread blocked on a
__os_mutex is blocked at the OS level, and the scheduler
cannot wake other coroutines on the same thread.
__os_mutex_t
Plain mutual-exclusion lock. On POSIX, a pthread_mutex_t; on
Windows, a CRITICAL_SECTION.
int __os_mutex_init (__os_mutex_t *mu);
int __os_mutex_lock (__os_mutex_t *mu);
int __os_mutex_unlock (__os_mutex_t *mu);
int __os_mutex_destroy(__os_mutex_t *mu);
The mutex is non-recursive; locking it from a thread that already holds it is undefined. The mutex is not interruptible; there is no try-lock with a deadline at this layer.
The xtc convention is that calls into L0 mutex routines must
not run inside a coroutine that intends to yield while the lock
is held. Holding __os_mutex_t across an xtc_yield() is a
priority inversion in the making: the scheduler cannot dispatch
other tasks on the thread, and any coroutine that blocks on the
same mutex from a different thread blocks that thread entirely.
Use cases:
- Protecting a small data structure between OS threads when the critical section runs to completion without yielding.
- Inside the implementation of a higher-layer primitive
(
xtc_lwlock,xtc_lockmgr, the slab allocator) where the critical section is bounded and known to be non-yielding. - Protecting state shared between an xtc thread and a non-xtc thread (for example, a callback from a third-party library running outside the xtc loop).
When a critical section is bounded by a few cache-line memory
accesses, __os_mutex_t is the right answer. When the
critical section may yield, raise to xtc_amutex (section 2).
When readers far outnumber writers, raise to xtc_lwlock or
xtc_lrlock (section 3). When the program acquires multiple
locks in arbitrary order under user control, use xtc_lockmgr
(section 4).
__os_rwlock_t
Reader-writer lock. On POSIX, a pthread_rwlock_t; on Windows,
an SRWLock.
int __os_rwlock_init (__os_rwlock_t *rw);
int __os_rwlock_rdlock (__os_rwlock_t *rw);
int __os_rwlock_wrlock (__os_rwlock_t *rw);
int __os_rwlock_unlock (__os_rwlock_t *rw);
int __os_rwlock_destroy (__os_rwlock_t *rw);
The rules are the same as for __os_mutex_t regarding yielding.
The fairness policy is the platform default; xtc does not
enforce a writer-preference policy at this layer.
__os_rwlock_t exists for completeness and for code that ports
from a system that uses pthread rwlocks. In new xtc code it is
almost always the wrong choice. An xtc_lwlock (section 3) is
faster on the read side under contention; an xtc_lrlock
(section 3) admits wait-free reads where the data structure
permits it.
__os_cond_t
Condition variable. pthread_cond_t on POSIX; condition
variable on Windows.
int __os_cond_init (__os_cond_t *c);
int __os_cond_wait (__os_cond_t *c, __os_mutex_t *mu);
int __os_cond_timedwait(__os_cond_t *c, __os_mutex_t *mu,
int64_t deadline_ns);
int __os_cond_signal (__os_cond_t *c);
int __os_cond_broadcast(__os_cond_t *c);
int __os_cond_destroy (__os_cond_t *c);
Standard condition-variable semantics. The mutex must be held when calling wait; it is released atomically with sleep and re-acquired before wait returns. Spurious wake-ups are permitted; the caller must re-test the predicate.
The same caveats apply as for __os_mutex_t: a thread blocked
in __os_cond_wait is blocked at the OS level; an xtc loop on
that thread cannot make progress. Use xtc_notify
(section 2) when the wait is part of a coroutine.
__os_sem_t
Counting semaphore. POSIX sem_t where available, falling back
to a counter behind a mutex+cond pair on platforms without
unnamed semaphore support.
int __os_sem_init (__os_sem_t *s, unsigned initial);
int __os_sem_post (__os_sem_t *s, unsigned n);
int __os_sem_wait (__os_sem_t *s, unsigned n, int64_t deadline_ns);
int __os_sem_destroy (__os_sem_t *s);
__os_sem_t is mostly used as scaffolding. Higher layers use
xtc_sem (section 2) which presents the same API but cooperates
with the coroutine scheduler.
Layer L3 sync: xtc_sync and friends
These primitives sit on the L3 (ptc/) layer and are aware of
the xtc loop. When a coroutine waits on one of them, the loop
parks the coroutine and dispatches another runnable task on the
same thread. The wake path resumes the parked coroutine through
the loop’s normal scheduling.
A coroutine may hold any of these locks across an xtc_yield(),
provided the calling code understands that other coroutines on
the same loop may queue waiting for the lock and that the
critical section therefore is not strictly bounded by wall time.
xtc_amutex_t – async mutex
A mutex that parks the caller on the loop instead of blocking the OS thread.
int xtc_amutex_create (xtc_amutex_t **out);
void xtc_amutex_destroy (xtc_amutex_t *m);
int xtc_amutex_lock (xtc_amutex_t *m, int64_t deadline_ns);
int xtc_amutex_try_lock (xtc_amutex_t *m);
int xtc_amutex_unlock (xtc_amutex_t *m);
The fast path is one compare_exchange against the owner field.
Under contention, the caller registers a waker on the lock’s
queue and yields; the unlocking coroutine wakes the head of the
queue.
Use xtc_amutex_t when the critical section may yield, perform
async I/O, or take a long time. The cost relative to
__os_mutex_t is one extra atomic and a queue-link write on the
slow path; both primitives are within a small constant factor on
the fast path. The cost relative to xtc_lwlock (section 3) is
that there is no separation between readers and writers.
xtc_notify_t – one-shot signalling
A coroutine-aware condition variable. The notify object holds
a single sticky signal-edge: xtc_notify_signal() makes any
subsequent or in-flight xtc_notify_wait() return; once
consumed, the edge is cleared.
int xtc_notify_create (xtc_notify_t **out);
void xtc_notify_destroy (xtc_notify_t *n);
int xtc_notify_signal (xtc_notify_t *n);
int xtc_notify_wait (xtc_notify_t *n, int64_t deadline_ns);
xtc_notify_t is the building block for higher-layer drain-wait
patterns. xtc_lrlock uses it to let a writer wait for the
last reader on the old buffer to finish.
xtc_sem_t – counting semaphore
A counting semaphore that parks waiters on the loop.
int xtc_sem_create (unsigned initial, xtc_sem_t **out);
void xtc_sem_destroy (xtc_sem_t *s);
int xtc_sem_post (xtc_sem_t *s, unsigned n);
int xtc_sem_acquire (xtc_sem_t *s, unsigned n, int64_t deadline_ns);
int xtc_sem_try_acquire(xtc_sem_t *s, unsigned n);
int xtc_sem_count (const xtc_sem_t *s);
Use xtc_sem_t for resource pools (database connections,
buffer slots) and for bounded-concurrency guards where the bound
is greater than one. When the bound is one, an xtc_amutex_t
is faster.
xtc_abort_source_t – cancellation
Not strictly a lock, but listed here because it is the cooperative-cancellation primitive that other locks honour through their deadline arguments.
int xtc_abort_source_create (xtc_abort_source_t **out);
void xtc_abort_source_destroy (xtc_abort_source_t *s);
int xtc_abort_source_fire (xtc_abort_source_t *s, int reason);
int xtc_abort_source_token (xtc_abort_source_t *s, xtc_abort_token_t *t);
Fire an abort source and every wait on a token derived from it
returns XTC_E_ABORTED. The intent is to cancel a tree of
pending operations – a request whose client disconnected, an
RPC whose deadline expired – without leaking blocked
coroutines.
Layer L3 reader-writer: xtc_lwlock, xtc_lrlock, xtc_rcu
These three primitives implement different points on the read-bias / write-cost / consistency-model trade-off.
xtc_lwlock_t – lightweight reader-writer lock
A reader-writer lock with shared and exclusive modes, modelled directly on the PostgreSQL LWLock with the same encoding tricks (HAS_WAITERS, WAKE_IN_PROGRESS, QUEUE_LOCKED in the high-order bits of the state word). The fast paths for shared-acquire and shared-release each compile to a single atomic increment when no exclusive waiter is present; the exclusive paths add a compare-exchange.
int xtc_lwlock_init (xtc_lwlock_t *lw);
int xtc_lwlock_acquire (xtc_lwlock_t *lw, xtc_lwlock_mode_t mode);
int xtc_lwlock_release (xtc_lwlock_t *lw);
int xtc_lwlock_destroy (xtc_lwlock_t *lw);
Modes are XTC_LWLOCK_SHARED and XTC_LWLOCK_EXCLUSIVE.
Use xtc_lwlock_t when reads dominate and the protected region
is short enough to leave a non-yielding critical section. The
contention behaviour matches PG’s LWLock: shared acquirers see
fully concurrent fast paths until an exclusive waiter arrives,
at which point all subsequent acquirers queue.
xtc_lwlock_t is the right choice for ~95% of “this is a
hot read-mostly map” cases.
xtc_lrlock_t – left-right (wait-free reads)
A two-buffer lock that gives readers wait-free access at the cost of doubled storage. Writers update the off-line buffer, then publish (atomic index swap) and wait for readers on the old buffer to finish; the reader-side cost is one atomic load.
int xtc_lrlock_create_ex (const xtc_lrlock_opts_t *opts,
xtc_lrlock_t **out);
void xtc_lrlock_destroy (xtc_lrlock_t *lr);
const void *xtc_lrlock_read_data (xtc_lrlock_t *lr);
void *xtc_lrlock_write_data (xtc_lrlock_t *lr);
void xtc_lrlock_publish_full_sync(xtc_lrlock_t *lr);
Use xtc_lrlock_t when reads are absolutely on the hot path,
the data fits naturally in two copies (or the optional COW mode
that xtc_lrlock supports for sparse updates), and writes are
infrequent compared to reads. Typical examples are configuration
tables, schema caches, and routing tables.
xtc_lrlock_t admits writes from concurrent threads only via
its single writer mutex; it is not lock-free for writers.
When the writer runs inside a fiber, its publish wait yields the
fiber back to the loop rather than spinning the OS thread, so reader
fibers advance and the loop keeps serving other work. This is
automatic; the xtc_alrlock_create / xtc_alrlock_create_ex aliases
name that fiber-awareness explicitly at the call site (an object created
with either family works with either family’s calls).
xtc_rcu_t – read-copy-update with epoch reclamation
Wait-free reads, copy-on-write for updates, deferred reclamation through epochs.
int xtc_rcu_create (xtc_rcu_t **out);
void xtc_rcu_destroy (xtc_rcu_t *r);
void xtc_rcu_read_lock (xtc_rcu_t *r);
void xtc_rcu_read_unlock(xtc_rcu_t *r);
int xtc_rcu_assign_pointer(xtc_rcu_t *r, void *new_ptr);
void xtc_rcu_synchronize(xtc_rcu_t *r);
void xtc_rcu_call (xtc_rcu_t *r, void (*cb)(void *), void *arg);
Use xtc_rcu_t for graph data structures whose updates produce
new versions and whose old versions can be freed once no reader
still holds a pointer into them: relcache-style maps, version
trees, immutable index nodes.
The trade-off relative to xtc_lrlock_t is that RCU has no
internal copy management (the user produces and assigns the new
version) but admits multiple writer threads producing updates
concurrently with readers.
When NOT to use any of these
- When the workload is balanced (reads ~ writes), a plain
xtc_amutex_toutperforms all three of these. - When acquisition order is dynamic and could induce a cycle, the lock-manager (section 4) is the only deadlock-safe answer.
- Inside a coroutine that may yield while holding the lock,
xtc_lwlockandxtc_lrlockare correct but the writer queue can grow unbounded; considerxtc_amutexor the lock-manager.
Layer L3 lock manager: xtc_lockmgr
The xtc lock manager is the most general of the locking primitives. It admits arbitrary lock-acquisition orders, detects and resolves deadlocks, supports timeouts and cancellation, and manages large numbers of fine-grained locks identified by 64-bit keys. It is also the most expensive: each lock acquire takes one or two cache-line writes plus a hash lookup, plus the cost of honouring the deadlock detector’s victim selection policy.
The lock manager is a port of the Berkeley DB locking subsystem to the xtc framework. It uses the same conflict matrix parameterization, the same notion of lockers and lock objects, and the same deadlock-detection algorithm (waits-for-graph search). Code that ported between BDB and Cassandra by way of DataStax’s lock manager will recognize the surface immediately.
Conceptual model
A locker is an entity that holds locks. In xtc this is most
often an xtc_proc – one entity per coroutine – but a locker
can be any 64-bit handle the application chooses to allocate.
A lock object is what the locker holds the lock on. Lock objects are identified by a 64-bit key and an optional namespace tag, both selected by the application. In a database context, the key might be a (database-id, table-id, page-id, row-id) tuple folded down to 64 bits; in a higher-level application, the key might be the address of the protected structure.
A lock mode describes the access being requested. The
default conflict matrix is the BDB nine-mode matrix (NL, IS, IX,
S, SIX, U, X, R, W); applications can install a custom matrix
through xtc_lockmgr_create_ex.
API
int xtc_lockmgr_create (xtc_lockmgr_t **out);
int xtc_lockmgr_create_ex (const xtc_lockmgr_opts_t *opts,
xtc_lockmgr_t **out);
void xtc_lockmgr_destroy (xtc_lockmgr_t *lm);
int xtc_lockmgr_locker_alloc(xtc_lockmgr_t *lm,
xtc_locker_t *out);
int xtc_lockmgr_locker_free (xtc_lockmgr_t *lm,
xtc_locker_t l);
int xtc_lockmgr_acquire (xtc_lockmgr_t *lm,
xtc_locker_t l,
uint64_t object_key,
xtc_lock_mode_t mode,
int64_t deadline_ns,
xtc_lock_t *out);
int xtc_lockmgr_release (xtc_lockmgr_t *lm, xtc_lock_t lk);
int xtc_lockmgr_release_all(xtc_lockmgr_t *lm, xtc_locker_t l);
int xtc_lockmgr_stats (xtc_lockmgr_t *lm,
xtc_lockmgr_stats_t *out);
Deadlock detection
A deadlock is a cycle in the waits-for graph: a set of lockers each waiting on a lock held by another locker in the set, with the cycle closing back on the original locker. Without intervention, a deadlock is permanent: each locker waits for the next, and none can release anything because none can proceed.
The lock manager runs a periodic deadlock detector that walks
the waits-for graph looking for cycles. When it finds one, it
breaks the cycle by aborting one of the lockers in it – the
victim. The victim’s xtc_lockmgr_acquire returns
XTC_E_DEADLK; the application is responsible for reacting,
typically by releasing every lock the locker holds and retrying
the operation that triggered the cycle.
The detector runs in its own thread. Detection frequency is
configurable through xtc_lockmgr_opts_t.detect_interval_ns;
the default is 100 ms. Setting the interval to zero disables
the detector and reduces the lock manager to a deadlock-prone
fast path; this is appropriate only when the application
guarantees a global lock ordering and uses the lock manager for
its other features (timeouts, statistics, fine-grained
acquisition).
Victim selection
Once the detector finds a cycle, it must choose which locker in
the cycle to abort. The application chooses the policy through
xtc_lockmgr_opts_t.victim:
XTC_LOCK_VICTIM_RANDOM -- pick uniformly from the cycle
(the BDB default; called
XTC_LOCK_VICTIM_DEFAULT for
historical compatibility)
XTC_LOCK_VICTIM_OLDEST -- pick the locker with the
longest existence
XTC_LOCK_VICTIM_YOUNGEST -- pick the most recently created
locker
XTC_LOCK_VICTIM_MIN_LOCKS -- pick the locker with the
fewest locks held; minimizes
the rollback cost
XTC_LOCK_VICTIM_MAX_LOCKS -- pick the locker with the most
locks; aborts the largest
transaction
XTC_LOCK_VICTIM_MIN_WRITE -- pick the locker holding the
fewest write locks
XTC_LOCK_VICTIM_MAX_WRITE -- pick the locker holding the
most write locks
XTC_LOCK_VICTIM_EXPIRE -- pick only lockers whose own
deadlines have already passed;
if no expired locker exists,
abort no one and let the
detector try again at the next
tick
XTC_LOCK_VICTIM_CUSTOM -- the application supplies the
selection function in
opts.victim_pick_fn
The choice of policy depends on the workload. RANDOM avoids starvation and is a defensible default. YOUNGEST tends to abort short transactions and let long ones complete, reducing total work lost; this is the wound-wait style. OLDEST is the wait-die style and is biased the other way. MIN_LOCKS is a useful policy when the application can cheaply reproduce the work of a small transaction.
Timeouts
Independently of the deadlock detector, an acquisition can fail
because its own deadline has expired. The deadline_ns
argument to xtc_lockmgr_acquire is an absolute deadline in
xtc_now_ns() units. When the deadline elapses without the
lock being granted, the call returns XTC_E_TIMEDOUT.
Timeouts are independent of deadlock detection but interact
with it. The XTC_LOCK_VICTIM_EXPIRE policy makes the
detector wait for natural timeouts before resorting to abort:
if a deadlock cycle exists but no locker has its own deadline
yet, the detector takes no action; once a locker’s deadline
passes, it becomes the natural victim and the cycle resolves.
This policy is appropriate when timeouts are short relative to
the detection interval and when application code already has
robust deadline handling.
Performance and tuning
The detector walks the waits-for graph each tick. The walk is
linear in the number of waiting lockers; idle lockers (those
not currently waiting) cost nothing. In a workload with many
idle lockers and few contention points the detector’s overhead
is negligible. In a workload with many contention points the
overhead can be observed; tune detect_interval_ns upward to
amortize.
xtc_lockmgr_stats_t exposes:
n_acquires -- successful acquisitions, lifetime
n_releases -- releases, lifetime
n_waits -- acquisitions that had to wait
n_timeouts -- acquisitions that returned TIMEDOUT
n_deadlocks_found -- cycles found by the detector
n_aborts -- victim aborts performed
detect_iterations -- detector ticks elapsed
waits_for_edges -- size of the current waits-for graph
When to use the lock manager
Use xtc_lockmgr_t when:
- The set of lockable objects is dynamic and large. An
application protecting individual rows of a million-row
table cannot reasonably allocate a
xtc_amutex_tper row; the lock manager indexes by 64-bit key and folds collisions onto a single hash bucket worth of state. - The acquisition order is dictated by the user (a SQL query plan, a graph traversal where edges are followed in response order) and a global ordering cannot be enforced. Deadlock detection is the only correct response to such workloads.
- Multiple lock modes participate: shared, exclusive, intention-shared, intention-exclusive, update. The full BDB conflict matrix is supported out of the box.
- Statistics and observability matter. The lock manager tracks contention, wait queues, and conflict counts at low cost.
Do not use the lock manager when:
- The number of locks is small and known statically; an
array of
xtc_amutex_tis faster per-acquisition. - The workload reads more than it writes; one of the reader-writer primitives in section 3 is faster.
- The critical section is so short that the hash-table
lookup itself dominates the cost; an
__os_mutex_tis cheaper.
Choosing a primitive
The decision tree below covers the common cases; each leaf is the recommended primitive.
flowchart TD
Start(["I need mutual exclusion"]) --> Y{"Does the critical section<br/>yield / await / block<br/>on async I/O?"}
Y -->|Yes| YM{"Multiple lock objects<br/>with user-driven<br/>ordering?"}
Y -->|No| NM{"Multiple lock objects<br/>with user-driven<br/>ordering?"}
YM -->|Yes| LM1["xtc_lockmgr"]:::leaf
YM -->|No| RH{"Read-heavy<br/>workload?"}
RH -->|Yes| RW["xtc_lwlock<br/>or xtc_lrlock"]:::leaf
RH -->|No| AM["xtc_amutex"]:::leaf
NM -->|Yes| LM2["xtc_lockmgr"]:::leaf
NM -->|No| OM["__os_mutex_t"]:::leaf
classDef leaf fill:#e6f6ec,stroke:#2e9e57,stroke-width:2px;
For signalling rather than mutual exclusion:
flowchart TD
S(["I need to signal, not lock"]) --> K{"What shape?"}
K -->|one-shot edge| N["xtc_notify"]:::leaf
K -->|counting| SE["xtc_sem"]:::leaf
K -->|cancellation tree| AB["xtc_abort_source"]:::leaf
classDef leaf fill:#e6f6ec,stroke:#2e9e57,stroke-width:2px;
For the special case of “a pointer that updates rarely, where readers must never block”:
flowchart TD
P(["rarely-updated shared data,<br/>never-blocking readers"]) --> Q{"Shape of the data?"}
Q -->|fits in two buffers| LR["xtc_lrlock"]:::leaf
Q -->|graph needing reclamation| RC["xtc_rcu"]:::leaf
classDef leaf fill:#e6f6ec,stroke:#2e9e57,stroke-width:2px;
Locking and the coroutine model
xtc coroutines are cooperative: a coroutine continues to run on its thread until it yields, awaits, or returns. The locking implications of this design:
-
Two coroutines on the same loop never contend on a lock at the same instant; they are serialized by the loop. An
xtc_amutex_tbetween them is essentially a flag check. -
Two coroutines on different loops contend through the L3 primitive’s normal mechanism; the difference is that the waiting coroutine yields to its own loop instead of blocking the thread.
-
Holding any lock across an
xtc_yield()is permitted but must be reasoned about. The yield gives other coroutines on the same loop the chance to run; if any of them tries to acquire the same lock, it joins the queue. Long-held locks effectively serialize the loop. -
Holding an
__os_mutex_t(L0) across anxtc_yield()is a correctness bug: the loop cannot dispatch coroutines that don’t need the mutex either, because the OS-level lock blocks the thread. Usextc_amutex_tinstead. -
Lock manager waits respect cancellation. An
xtc_lockmgr_acquirewhose caller’s abort token is fired returnsXTC_E_ABORTEDpromptly; the locker is removed from the wait graph and the deadlock detector no longer sees its waits-for edges.
See also
xtc_lwlock(3)– LWLock manual page.xtc_lrlock(3)– LRLock manual page.xtc_lockmgr(3)– Lock manager manual page.xtc_sync(3)– Sync primitives manual page.xtc_rcu(3)– RCU manual page.
The Berkeley DB locking subsystem documentation, available in the BDB source tree, is the canonical reference for the deadlock-detection model adopted here.