Dynomite

This is the reference manual and getting-started guide for the Rust implementation of Dynomite. It is written to be read two ways: front to back, as a guided tour that takes you from "what is this and why would I use it" to a running, replicated cluster and an embedded engine in your own program; and as a reference, dipped into by section when you need the exact semantics of a consistency level, a wire frame, or a configuration knob.

What it is

Dynomite is available as two things from one codebase:

• dynomited -- a standalone server binary. You point it at a backend (a local Valkey or Memcached, or its own embedded store) and a list of peers, and it presents the backend's own wire protocol to clients while replicating writes across the cluster behind the scenes. Existing Redis and Memcached clients talk to it unmodified.

• dynomite-engine -- a library crate (imported as dynomite), published on crates.io, that embeds the same distribution layer directly in your Rust program through a stable, documented API. You supply a backend by implementing one trait; Dynomite supplies the ring, gossip, quorum, hinted handoff, and repair.

On top of the engine sits Dyniak, a Riak-compatible layer backed by the transactional Noxu storage engine. Dyniak adds buckets and objects, convergent data types (CRDTs), cross-node XA transactions, links, secondary indexes, MapReduce, and full-text / vector / regex search. It has its own set of chapters.

How a request flows

flowchart LR
  C[Redis / Memcache / Riak client] -->|native wire protocol| P(dynomited node)
  P -->|owns this key| L[(local backend)]
  P -->|replica lives elsewhere| G{{ring lookup}}
  G -->|DNODE peer frame| R1(peer node)
  G -->|DNODE peer frame| R2(peer node)
  R1 --> LB1[(backend)]
  R2 --> LB2[(backend)]

A client speaks its backend's protocol to any node. That node routes the key over the ring, serving it locally or forwarding it to the owning peers over the DNODE peer plane.

The client never learns the topology. It connects to any node, speaks the protocol it already knows, and Dynomite does the routing, replication, and reconciliation.

What the engine gives you

Sharding

Consistent-hash partitioning over a token ring, shared-nothing, no single point of failure.

Replication

Multi-datacenter, rack-aware, with tunable quorum reads and writes.

Membership

Gossip-based discovery and failure detection; auto-eject and auto-rejoin.

Durability under failure

Hinted handoff for writes to briefly-absent peers; read repair and Merkle-tree anti-entropy for divergent replicas.

Transports

TCP (the default, matching upstream) or QUIC, both over IPv4 and IPv6.

Dyniak extras

CRDTs, XA transactions, links, 2i, MapReduce (optional WASM phases), and durable FT.* search.

The philosophy: distribution as a layer, not a rewrite

The central bet of Dynomite -- inherited from the Amazon Dynamo paper and Netflix's original C implementation -- is that availability and cross-datacenter replication can be added as a thin layer in front of a storage engine, rather than baked into it. A single-node store stays simple and fast; Dynomite wraps it with the distributed-systems machinery, and the two concerns stay separate.

This manual is explicit about the trade-offs that bet implies, and about the designs we considered and deliberately did not choose. Those are collected in Design Decisions (Roads Not Taken) and called out inline wherever a choice matters.

Conventions used in this manual

• Shell commands assume you have run nix develop first, which pins every tool the project uses.
• Rust snippets that appear inline are illustrative teaching examples. Complete, compilable programs live under crates/*/examples/ and each has its own walk-through in the Examples part.
• Links written like ServerBuilder point into this manual; API links point at the generated rustdoc; and manual-page references such as dynomited(8) point at the shipped man pages.

Why Dynomite?

This page is the "should I use this at all" chapter. It states the problem Dynomite solves, the bet it makes to solve it, the things it deliberately does not do, and how it compares to the obvious alternatives. Read it before you commit to the design; the trade-offs here shape every later choice.

The problem

A single-node key/value store is a single point of failure. When the process dies, the machine reboots, the rack loses power, or the datacenter goes dark, the data it held is unavailable until something brings it back. For a cache that is an annoyance; for a session store or a shopping cart it is an outage.

The usual answers each have a cost:

• Replicate inside the store. Redis replication and Redis Cluster bolt replication and sharding onto the store itself. That works, but it couples your availability story to one vendor's clustering model, and it does not help the Memcached deployment two teams over.
• Put a proxy in front. A proxy such as twemproxy shards across backends, but a classic proxy has no notion of replicas, no gossip, no failure detection, and no reconciliation. It spreads load; it does not make you highly available.
• Move to a distributed database. Cassandra, ScyllaDB, DynamoDB, and friends are distributed from the ground up. If you want their data model and are willing to migrate to it, that is a fine answer -- but it is a rewrite, not a layer, and you give up the exact single-node store you already tuned and trust.

The Dynomite bet

Dynomite -- following the Amazon Dynamo paper and Netflix's original C implementation -- makes one central bet:

Availability, sharding, and cross-datacenter replication can be added as a thin layer in front of a storage engine, rather than baked into it.

The store stays simple and fast. Dynomite wraps it with the distributed-systems machinery -- a consistent-hash token ring, gossip membership, tunable quorum, hinted handoff, read repair, and anti-entropy -- and the two concerns stay cleanly separated. Your clients keep speaking the store's own wire protocol; they never learn the topology.

flowchart TB
  subgraph before[Before: single point of failure]
    C1[client] --> S1[(Valkey)]
  end
  subgraph after[After: Dynomite layer]
    C2[client] -->|same RESP protocol| D1(dynomited)
    D1 --> B1[(Valkey)]
    D1 -.gossip + replication.-> D2(dynomited)
    D1 -.gossip + replication.-> D3(dynomited)
    D2 --> B2[(Valkey)]
    D3 --> B3[(Valkey)]
  end

The client's contract does not change. It speaks the backend's protocol to any node; Dynomite handles routing, replication, and recovery behind that unchanged wire.

The payoff of the bet is separation of concerns. The store team keeps optimizing single-node throughput; the distribution layer evolves on its own schedule; and a Memcached shop and a Valkey shop get the same availability story from the same binary.

What Dynomite is not

Being explicit about the ceiling matters more than the pitch. Dynomite is a Dynamo-style, availability-first system. That has hard consequences you must accept going in:

Not a consensus system

There is no RAFT, no Paxos, no single leader ordering writes. Dynomite does not give you linearizable, strictly-serializable writes on plain key/value operations. If you need "the write either happened everywhere in one order or nowhere", this is not that tool for that key class.

Tunable, not strong, by default

Consistency is a knob per read and per write: DC_ONE, DC_QUORUM, DC_SAFE_QUORUM, DC_EACH_SAFE_QUORUM. You trade latency for overlap. Even at quorum this is the Dynamo tradition of eventual consistency with quorum overlap, not a linearizability guarantee. See Replication and Consistency.

Last-write-wins on plain keys

Concurrent writes to the same plain key reconcile by last-writer-wins. If you need conflict-free merges you want the Dyniak CRDT layer, not raw RESP.

Not a query engine on the RESP path

The RESP and Memcache surfaces are the backend's own commands. Rich queries, secondary indexes, MapReduce, and full-text/vector search live in Dyniak, the optional Riak-compatible layer.

If any of those constraints is a dealbreaker for your workload, stop here and reach for a consensus store. If they are acceptable -- and for caches, sessions, feature flags, counters, and most "mostly-available beats occasionally-perfect" data they are -- read on.

When to reach for Dynomite (and when not)

Two comparisons deserve a closer look because they are the ones people actually agonize over.

Dynomite vs Redis Cluster / Valkey native clustering. Both give you sharding and replicas. The native clustering is tighter with the store and needs no extra process. Dynomite's edge is that it is store-agnostic (the same layer fronts Memcached), it is topology-opaque to clients (a plain Redis client works, no cluster-aware client required), and its cross-datacenter story is a first-class design goal rather than an add-on. If you run one Valkey deployment, in one region, with cluster-aware clients, native clustering may be less to operate. If you run several stores, across regions, with a fleet of dumb clients, the layer earns its keep.

Dynomite vs a bespoke proxy. If your proxy already does consistent-hash sharding and you are tempted to "just add replication", you are about to reimplement gossip, failure detection, hinted handoff, read repair, and anti-entropy. That is the body of this manual. Reusing it is cheaper than rebuilding it.

Server or library: two ways to run it

Dynomite ships as one codebase with two front doors, and the choice is orthogonal to everything above.

• dynomited, the server. A standalone binary. Point it at a backend and a peer list; it presents the backend's wire protocol to clients and does the distribution behind the scenes. Existing Redis and Memcached clients connect unmodified. This is the operational default and the subject of Your First Cluster. See also dynomited(8).

• dynomite-engine, the library. The same distribution layer as a Rust crate (imported as dynomite). You embed it in your own program, supply a backend by implementing one trait, and Dynomite supplies the ring, gossip, quorum, hinted handoff, and repair through a documented, SemVer-governed API. This is the subject of Your First Embedded Engine.

Where to next

• Concepts in Ten Minutes -- the vocabulary: token ring, rack, datacenter, replica, consistency level, gossip, hinted handoff, read repair, anti-entropy. Read this before either hands-on chapter.
• Your First Cluster -- stand up a small local dynomited cluster in front of Valkey and watch a write on one node become readable from another.
• Your First Embedded Engine -- build the smallest embedded engine, then grow it to a real backend and a second peer.

Concepts in Ten Minutes

This chapter is the shared vocabulary. Every later page assumes you know these terms, so it is worth ten minutes now to save an hour later. Each concept gets a tight definition, a sentence on why it exists, and a link to the chapter that treats it in full. Read it top to bottom the first time; come back to it as a glossary afterward.

The mental model is a chain. A key hashes to a point on a ring. The ring is owned by nodes, grouped into racks, grouped into datacenters. A write lands on several replicas; how many must agree is the consistency level. Nodes learn each other's health through gossip. When a replica is briefly unreachable, hinted handoff holds its writes; when replicas drift apart, read repair and anti-entropy pull them back together. That chain is the whole system.

The token ring

Dynomite partitions the key space with consistent hashing. Imagine the output of the hash function laid out on a circle -- the token ring. Every node claims one or more tokens, which are just positions on that circle. To place a key, hash it to a point and walk clockwise to the first token; the node that owns that token owns the key.

flowchart LR
  K["key 'user:42'"] -->|hash| P["point on ring"]
  P -->|walk clockwise| T["first token >= point"]
  T --> N["owning node"]

A key is placed by hashing it to a point and walking the ring clockwise to the first token. The owner of that token owns the key.

Consistent hashing is the reason adding or removing a node only reshuffles the keys near that node's tokens, not the entire key space. The full treatment -- token arithmetic, the continuum, and how replicas are chosen by continuing the walk -- is in The Ring and the Token Space.

Node, rack, datacenter

These three form the physical hierarchy, and replication is aware of all three levels.

Node

One dynomited process (or one embedded Server) fronting one backend datastore. Nodes are symmetric -- there is no special coordinator node. A client may connect to any node.

Rack

A named group of nodes that, together, hold a complete copy of the datacenter's key space. A rack is a replica set: each rack covers the whole ring exactly once. Racks usually map to failure domains (a physical rack, an availability zone).

Datacenter (DC)

A named group of racks. Multiple racks in one DC means multiple local copies; multiple DCs means geographic replication. Consistency levels are expressed relative to the DC (hence the DC_ prefix).

The key invariant: one rack = one full copy of the ring. If a datacenter has three racks, the DC holds three copies of every key, one per rack. That is how the number of replicas is controlled -- by how many racks you deploy, not by a per-key setting. See The Ring and the Token Space for how the per-rack continuum is built.

flowchart TB
  subgraph dc1[Datacenter dc1]
    r1[rack1: full ring copy]
    r2[rack2: full ring copy]
  end
  subgraph dc2[Datacenter dc2]
    r3[rack1: full ring copy]
  end
  r1 -. cross-DC replication .- r3
  r2 -. cross-DC replication .- r3

Two racks in dc1 hold two local copies; dc2 holds a third across the wire. Replica count is a function of how many racks exist, not a per-key knob.

Replica

A replica is a copy of a key held on a distinct rack. Because each rack covers the whole ring, the replicas of a key are "the node owning that key's token, in each rack." A read or write fans out to the replicas that the consistency level requires, and their answers are coalesced into the single reply the client sees. When replicas disagree, reconciliation kicks in (read repair, below).

Consistency level

The consistency level is the knob that trades latency against overlap. It says how many replicas must acknowledge before Dynomite answers the client. It is set per pool, and can be overridden per request class through bucket types. There are four, and they apply independently to reads (read_consistency) and writes (write_consistency):

In the embedded API these are the ConsistencyLevel variants (DcOne, DcQuorum, DcSafeQuorum, DcEachSafeQuorum); in YAML they are the DC_ONE / DC_QUORUM / DC_SAFE_QUORUM / DC_EACH_SAFE_QUORUM strings shown above. The full semantics -- how a quorum is counted, how answers are coalesced, what happens when it cannot be met -- are in Replication and Consistency.

Gossip

Nodes do not have a central registry telling them who is alive. Instead they gossip: on a fixed interval, each node exchanges a small digest of what it knows about every peer's state and token assignment with a few others. Over a handful of rounds, a change (a node joined, a node went quiet) propagates to the whole cluster. Gossip is what turns a list of seed addresses into a live, self-healing membership view.

flowchart LR
  A[node A] <-->|digest| B[node B]
  B <-->|digest| C[node C]
  C <-->|digest| D[node D]
  A <-->|digest| D

Each round, a node swaps state digests with a few peers. A membership change reaches everyone in a few rounds without any central coordinator.

From gossip come two operational behaviors: a peer that stops responding is marked down and auto-ejected from routing, and when it comes back it auto-rejoins. The state machine, the digest format, and the failure detector live in Membership and Gossip.

Hinted handoff

A write's target replica might be down for a moment -- a restart, a brief network blip. Dropping the write would weaken durability; blocking on it would hurt availability. Hinted handoff takes the third path: the coordinating node stores the write locally as a hint tagged with the intended peer, keeps serving, and a background drainer replays the hint once gossip reports the peer back to Normal.

Hinted handoff is off by default and applies to writes only. Turn it on and size its store, TTL, and drain cadence per pool -- the knobs (enable_hinted_handoff, hint_ttl_seconds, hint_store_max_bytes, hint_drain_interval_ms, hint_dir) are documented under Configuration. Failure behavior in full is in Failure Handling.

Read repair

Replicas drift. A hint has not drained yet; a write missed a replica that was briefly down. Read repair fixes the divergence on the read path: when a read fans out to several replicas and their answers disagree, Dynomite returns the freshest to the client and, in the background, writes it back to the stale replicas. It is opportunistic -- it only repairs the keys that are actually read -- and cheap, which is why it is the first line of reconciliation. Read repair applies to GET-style reads. See Failure Handling.

Anti-entropy

Read repair only heals keys that get read. Cold keys can stay divergent indefinitely. Active anti-entropy (AAE) is the background sweep that closes that gap: peers exchange Merkle trees (compact hashes of key ranges), spot the ranges whose hashes differ, and reconcile only those ranges -- without shipping the whole dataset. It is the slow, thorough backstop behind the fast, opportunistic read repair.

flowchart LR
  P1["peer 1 Merkle root"] -->|compare| X{hashes differ?}
  P2["peer 2 Merkle root"] -->|compare| X
  X -->|no| Done[in sync, stop]
  X -->|yes| Descend[descend into subtree]
  Descend --> Reconcile[exchange only divergent ranges]

Merkle-tree comparison narrows reconciliation to just the divergent key ranges, so anti-entropy scales with the drift, not the dataset.

For plain RESP/Memcache pools AAE is the reconciliation backstop; the richer, scheduled AAE (full sweeps, segment ticks, per-bucket trees) belongs to Dyniak and is configured under the riak: block. See Failure Handling for how the pieces fit.

How it all fits

flowchart TB
  C[client] -->|any node| Coord(coordinating node)
  Coord -->|hash key -> ring| Ring{token ring}
  Ring --> R1[replica rack1]
  Ring --> R2[replica rack2]
  Ring --> R3[replica rack3]
  R1 & R2 & R3 -->|coalesce at consistency level| Coord
  Coord -->|reply| C
  Coord -.peer down: store hint.-> HH[(hint store)]
  R1 -.disagree on read.-> RR[read repair]
  R1 -.cold divergence.-> AAE[anti-entropy sweep]

The full request path: hash to the ring, fan out to replicas, coalesce at the consistency level, and reconcile via hinted handoff, read repair, and anti-entropy when things go wrong.

With the vocabulary in hand, pick your path:

• Running the server: Your First Cluster.
• Embedding the library: Your First Embedded Engine.
• The deep dives: Architecture and its subpages on the ring, consistency, gossip, and failure handling.

Your First Cluster

This chapter takes you from a fresh checkout to a running two-node Dynomite cluster fronting Valkey, and then proves the point of the whole project: a value written through one node is readable through the other. Everything here uses dynomited, the server binary. If you want the library instead, see Your First Embedded Engine.

We build a two-datacenter cluster: one node in dc1, one in dc2, each fronting its own local Valkey. The two nodes share a token, so each is a full replica of the other -- which is exactly what makes the cross-node read at the end work. This is the shape of the shipped node1.yml / node2.yml sample pair, and you can start from those files directly.

Step 1: enter the dev shell

$ cd dynomite
$ nix develop

You now have valkey-server, valkey-cli, and a way to run dynomited (cargo run -p dynomited -- ...) on your PATH.

Step 2: start two backends

Each node fronts its own backend. This is the single most common first-time mistake, so it gets an admonition:

Start two Valkey instances on distinct ports:

$ valkey-server --port 11371 --daemonize yes
$ valkey-server --port 11370 --daemonize yes

Sanity-check them:

$ valkey-cli -p 11371 ping
PONG
$ valkey-cli -p 11370 ping
PONG

Step 3: write the node configs

A dynomited config is one YAML pool stanza. Here is node1.yml, in dc1, fronting the Valkey on 11371:

dyn_o_mite:
    datacenter: dc1
    rack: rack1
    listen: 127.0.0.1:8102
    dyn_listen: 127.0.0.1:8101
    dyn_seeds:
        - 127.0.0.1:8113:rack1:dc2:101134286
    dyn_seed_provider: simple_provider
    tokens: '101134286'
    servers:
        - 127.0.0.1:11371:1
    data_store: 0
    stats_listen: 127.0.0.1:33331
    preconnect: true

And node2.yml, in dc2, fronting the Valkey on 11370:

dyn_o_mite:
    datacenter: dc2
    rack: rack1
    listen: 127.0.0.1:8114
    dyn_listen: 127.0.0.1:8113
    dyn_seeds:
        - 127.0.0.1:8101:rack1:dc1:101134286
    dyn_seed_provider: simple_provider
    tokens: '101134286'
    servers:
        - 127.0.0.1:11370:1
    data_store: 0
    stats_listen: 127.0.0.1:33333
    preconnect: true

These are the shipped samples under crates/dynomited/conf/ (node1.yml, node2.yml); you can copy them verbatim.

What each knob does

The pool key (dyn_o_mite) is just the pool name; it can be anything. Inside it:

datacenter / rack

Where this node sits in the physical hierarchy from Concepts. One rack per DC here means one full copy of the ring per DC -- so the two nodes are replicas of each other across the two DCs.

listen

The client plane: the address your Valkey clients connect to. Node 1 serves clients on 8102, node 2 on 8114. Clients speak plain RESP here.

dyn_listen

The peer plane: the address other dynomited nodes connect to for gossip and replication (the DNODE protocol). Node 1 listens on 8101, node 2 on 8113. Clients never touch this port.

dyn_seeds

The bootstrap peer list: host:dyn_port:rack:dc:token. Node 1 seeds node 2's peer address and vice versa. Once gossip converges, the full membership is learned from these seeds.

tokens

This node's position(s) on the ring. Both nodes claim the same token (101134286), so each owns the same slice in its own rack -- i.e. they are replicas. Different tokens would make them shard partners instead.

servers

The backend datastore: host:port:weight [name]. Node 1 fronts Valkey on 11371, node 2 on 11370 -- distinct backends, per the warning above.

data_store

0 selects the Valkey / RESP backend (alias valkey/redis). 1 is Memcache, 2 is Dyniak. See the dynomited(8) backend list.

stats_listen

The HTTP stats endpoint for this node. Distinct per node so both can run on one host.

preconnect

Open the backend connection at startup rather than on first request, so a misconfigured backend fails loudly and early.

The exhaustive field list, plus consistency levels, bucket types, and hinted handoff, is in Configuration.

Step 4: validate before you launch

Always dry-run the config first. The --test-conf flag parses and validates, then exits without binding anything:

$ cargo run -p dynomited -- --test-conf --conf-file node1.yml
$ cargo run -p dynomited -- --test-conf --conf-file node2.yml

A clean exit means the YAML is well-formed and internally consistent (no duplicate bucket-type names, valid consistency strings, resolvable listen addresses, and so on). Fix anything it reports before moving on.

Step 5: launch both nodes

In two terminals (both inside nix develop):

$ cargo run -p dynomited -- --conf-file node1.yml

$ cargo run -p dynomited -- --conf-file node2.yml

Each logs its bound client and peer addresses and then a gossip round as the two discover each other. Give it a second or two to converge.

Step 6: write to one node, read from the other

This is the payoff. Set a key through node 1's client port (8102):

$ valkey-cli -p 8102 set greeting "hello from dc1"
OK

Now read it through node 2's client port (8114) -- a different node, in a different datacenter, fronting a different backend:

$ valkey-cli -p 8114 get greeting
"hello from dc1"

The value crossed the cluster. Node 1 coordinated the write, routed it over the ring, and replicated it to node 2's replica over the peer plane; node 2 served the read from its own backend. The client did none of that -- it spoke plain RESP to whichever node was nearest.

You can confirm the two backends really do each hold a copy:

$ valkey-cli -p 11371 get greeting
"hello from dc1"
$ valkey-cli -p 11370 get greeting
"hello from dc1"

Two distinct Valkey instances, same value -- that is replication, not a shared backend.

Step 7: watch the stats

Each node exposes an HTTP stats endpoint on its stats_listen address:

$ curl -s http://127.0.0.1:33331/ | head
$ curl -s http://127.0.0.1:33333/ | head

You will see per-pool and per-server counters -- request rates, forward counts, peer states. These are the same metrics the Metrics chapter describes, and the fields are enumerated by dynomited --describe-stats.

What just happened

flowchart LR
  Cli1[valkey-cli :8102] -->|SET| N1(node1 dc1)
  N1 --> B1[(Valkey :11371)]
  N1 -->|DNODE replicate| N2(node2 dc2)
  N2 --> B2[(Valkey :11370)]
  Cli2[valkey-cli :8114] -->|GET| N2

A write on node 1 replicated across the peer plane to node 2; a read on node 2 served the replicated value from its own backend.

You built a shared-nothing, cross-DC replicated cluster in front of an unmodified Valkey, and your client never learned the topology. That is the Dynomite bet from Why Dynomite? made concrete.

Where to next

• Turn consistency up: set read_consistency / write_consistency in the pool stanza and re-run. Start with DC_ONE (what these samples use by default) and move to DC_QUORUM. See Configuration and Replication and Consistency.
• Add a third node in a second rack of dc1 for a local replica; the shipped redis_rack1_node.yml / redis_rack2_node.yml / redis_rack3_node.yml samples show a three-rack single-DC layout with DC_SAFE_QUORUM already configured.
• Break something on purpose: stop one node and watch gossip eject it, then restart it and watch it rejoin. The mechanics are in Membership and Gossip and Failure Handling.
• Front a different backend: set data_store: 1 and point servers at a memcached instance from the flake.
• For the operator's view -- daemonizing, log formats, pid files, admin operations -- read dynomited(8) and Running dynomited.

Your First Embedded Engine

This chapter is the library counterpart to Your First Cluster. Instead of running the dynomited server, you embed the same distribution layer directly in a Rust program through the dynomite crate. You supply a backend by implementing one trait; Dynomite supplies the ring, gossip, quorum, hinted handoff, and repair.

We start with the smallest engine that runs, then grow it in two steps: a real Valkey backend, then a second in-process peer. Every snippet here mirrors a complete program under crates/dynomite/examples/; the inline code is the teaching version, the examples are the runnable version.

The smallest engine that runs

The minimal embed is a five-call build chain plus a start/shutdown handshake. No external store, no peers, no gossip: the in-crate MemoryDatastore stands in for the backing store, so this compiles and runs with nothing else installed.

use dynomite::embed::{Server, ServerBuilder};

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let handle = Server::start_with(
        ServerBuilder::new("dyn_o_mite")
            .listen("127.0.0.1:0".parse()?)
            .dyn_listen("127.0.0.1:0".parse()?),
    )
    .await?;

    eprintln!(
        "embedded dynomite up; client listen={:?} dnode listen={:?}",
        handle.listen_addr(),
        handle.dyn_listen_addr()
    );

    handle.shutdown().await?;
    Ok(())
}

That is the whole program. It is examples/embedded_minimal.rs verbatim. Run it with cargo run --example embedded_minimal.

The chain, call by call

ServerBuilder::new("dyn_o_mite")

Start a builder for a pool named dyn_o_mite. The pool name is the same one you would put as the top-level key in a YAML config; it labels the pool in stats and events.

.listen("127.0.0.1:0")

The client plane -- where RESP/Memcache clients connect. Port 0 asks the OS for an ephemeral port; you read the assigned port back from the handle. Bind a fixed port in production.

.dyn_listen("127.0.0.1:0")

The peer plane -- where other nodes connect for gossip and replication. Also ephemeral here.

Server::start_with(builder)

A convenience that calls .build() then .start() in one step. It spawns the background tasks (stats, metrics, and -- if enabled -- gossip and the accept loops) on the current tokio runtime and returns a ServerHandle.

Server::start_with(builder) is exactly builder.build()?.start().await folded into one call. When you need to inspect or hold the configured Server before starting it, split them:

#![allow(unused)]
fn main() {
use dynomite::embed::{Server, ServerBuilder};
async fn f() -> Result<(), Box<dyn std::error::Error>> {
let server: Server = ServerBuilder::new("dyn_o_mite")
    .listen("127.0.0.1:0".parse()?)
    .dyn_listen("127.0.0.1:0".parse()?)
    .build()?;                 // validate config, no tasks yet
let handle = server.start().await?;   // spawn background tasks
handle.shutdown().await?;
Ok(())
}
}

.build() validates the configuration and can fail with an EmbedError; nothing binds or spawns until .start(). That split is the in-process equivalent of dynomited --test-conf followed by launch.

The handle

ServerHandle is the control surface. It is Clone + Send + Sync, so several parts of your program can hold it. The calls you meet first:

• handle.listen_addr() / handle.dyn_listen_addr() -- the bound addresses (with the real port resolved when you asked for :0).
• handle.stats() -- a Snapshot of the current counters.
• handle.shutdown().await? -- graceful shutdown: cancel every background task, deregister, drain, return. It is idempotent.

Step up: a real Valkey backend

The minimal engine used the in-memory stand-in. To front a real Valkey, add the backend and topology to the chain. This mirrors examples/embedded_single_node.rs:

use std::time::Duration;

use dynomite::conf::{ConfServer, DataStore};
use dynomite::embed::{Server, ServerBuilder};

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let server: Server = ServerBuilder::new("dyn_o_mite")
        .listen("127.0.0.1:18102".parse()?)
        .dyn_listen("127.0.0.1:18101".parse()?)
        .data_store(DataStore::Valkey)
        .servers(vec![ConfServer::parse("127.0.0.1:6379:1 backend")?])
        .datacenter("dc-local")
        .rack("rack-local")
        .tokens_str("0")
        .timeout(Duration::from_secs(5))
        .enable_gossip(false)
        .build()?;

    let handle = server.start().await?;
    eprintln!(
        "up; client={:?} dnode={:?}",
        handle.listen_addr(),
        handle.dyn_listen_addr()
    );

    let snap = handle.stats();
    eprintln!("pool={} uptime={}s", snap.pool.name, snap.uptime);

    handle.shutdown().await?;
    Ok(())
}

The new calls, in order:

.data_store(DataStore::Valkey)

Select the backend protocol. The parallel to data_store: 0 in YAML. DataStore::Memcache and DataStore::Dyniak are the other two.

.servers(vec![ConfServer::parse("127.0.0.1:6379:1 backend")?])

The backend endpoints, parsed from the same host:port:weight [name] string the YAML servers: list uses. Here one Valkey on 6379, weight 1, named backend.

.datacenter(...) / .rack(...)

This node's place in the hierarchy, exactly as in Concepts.

.tokens_str("0")

This node's ring position(s), parsed from the same string the YAML tokens: key takes. One node owning token 0 owns the whole ring.

.timeout(Duration::from_secs(5))

Backend request timeout.

.enable_gossip(false)

Single node, no peers to gossip with, so gossip is off. Turn it on once you add peers.

Point a plain client at the bound listen address and it talks RESP to your embedded engine:

$ valkey-cli -p 18102 set k v
OK
$ valkey-cli -p 18102 get k
"v"

A connection accepted on the listen socket is served end to end -- parsed, routed through the dispatcher, and answered by the backend -- the same path the standalone dynomited proxy uses. See Server Lifecycle for the client-plane and in-process traffic contract.

Bring your own backend: the Datastore hook

You are not limited to Valkey and Memcache. Implement the Datastore trait and register it with .datastore(...), and Dynomite routes requests to your code instead of an external store. This is the single most important extension point; the full trait set is in Hooks and Traits.

A tiny in-memory Datastore (the shape used by examples/embedded_cluster3.rs):

#![allow(unused)]
fn main() {
use std::collections::HashMap;
use std::sync::Arc;

use parking_lot::Mutex;

use dynomite::embed::hooks::{BoxFuture, Datastore, DatastoreError, Protocol};
use dynomite::msg::{Msg, MsgType};

#[derive(Default, Clone)]
struct SharedKv {
    inner: Arc<Mutex<HashMap<u64, MsgType>>>,
}

impl Datastore for SharedKv {
    fn protocol(&self) -> Protocol {
        Protocol::Custom
    }

    fn dispatch(&self, req: Msg) -> BoxFuture<'_, Result<Msg, DatastoreError>> {
        let inner = self.inner.clone();
        Box::pin(async move {
            let mut g = inner.lock();
            if matches!(req.ty(), MsgType::ReqRedisSet) {
                g.insert(req.id(), MsgType::RspRedisStatus);
            }
            let stored = g.get(&req.id()).copied();
            drop(g);
            let mut rsp =
                Msg::new(req.id(), stored.unwrap_or(MsgType::RspRedisStatus), false);
            rsp.set_parent_id(req.id());
            Ok(rsp)
        })
    }
}
}

dispatch is the whole contract: take a parsed request Msg, return a response Msg (or a DatastoreError). protocol() tells the engine how to frame replies. Register it on the builder:

#![allow(unused)]
fn main() {
use dynomite::embed::ServerBuilder;
use dynomite::conf::DataStore;
#[derive(Default)] struct SharedKv;
impl dynomite::embed::hooks::Datastore for SharedKv {
  fn protocol(&self) -> dynomite::embed::hooks::Protocol { dynomite::embed::hooks::Protocol::Custom }
  fn dispatch(&self, req: dynomite::msg::Msg) -> dynomite::embed::hooks::BoxFuture<'_, Result<dynomite::msg::Msg, dynomite::embed::hooks::DatastoreError>> {
    Box::pin(async move { Ok(req) })
  }
}
fn f() -> Result<(), Box<dyn std::error::Error>> {
let builder = ServerBuilder::new("p")
    .listen("127.0.0.1:0".parse()?)
    .dyn_listen("127.0.0.1:0".parse()?)
    .data_store(DataStore::Valkey)
    .tokens_str("0")
    .datastore(Box::new(SharedKv::default()));   // your backend
let _ = builder;
Ok(())
}
}

Datastore is one of several hooks. The others -- SeedsProvider (where peer addresses come from), CryptoProvider (peer-plane encryption), and MetricsSink (where counters flush) -- each ship a default and are documented, with examples, in Hooks and Traits.

Step up again: a second peer

To make it a cluster, build more than one Server and give each the other's peer address as a seed. The shape below follows examples/embedded_cluster3.rs, trimmed to two nodes:

#![allow(unused)]
fn main() {
use std::time::Duration;
use dynomite::conf::{ConfDynSeed, ConfServer, ConsistencyLevel, DataStore};
use dynomite::embed::{Server, ServerBuilder, ServerHandle};
async fn spawn_node(
    rack: &str,
    listen: &str,
    dyn_listen: &str,
    tokens: &str,
    seeds: Vec<ConfDynSeed>,
) -> ServerHandle {
    let server: Server = ServerBuilder::new("p")
        .listen(listen.parse().unwrap())
        .dyn_listen(dyn_listen.parse().unwrap())
        .data_store(DataStore::Valkey)
        .servers(vec![ConfServer::parse("127.0.0.1:6379:1").unwrap()])
        .datacenter("dc-local")
        .rack(rack)
        .tokens_str(tokens)
        .read_consistency(ConsistencyLevel::DcOne)
        .write_consistency(ConsistencyLevel::DcOne)
        .dyn_seeds(seeds)
        .build()
        .unwrap();
    server.start().await.unwrap()
}
}

Each node takes a distinct rack, distinct listen/peer ports, its own token, and the other node(s) as dyn_seeds. The consistency-level calls map one-to-one to the read_consistency / write_consistency YAML keys and take the ConsistencyLevel variants from Concepts: DcOne, DcQuorum, DcSafeQuorum, DcEachSafeQuorum.

For in-process clusters you can drive requests without a socket hop via handle.inject_request(msg).await -- the dispatcher computes a routing plan and, if it targets a co-located peer, forwards through the in-process registry. That is how the cluster example verifies a write on one node reads back through another entirely in one process.

Start-then-park for a long-running service

The examples shut down immediately to keep them short. A real embedder starts the engine, then parks until a signal arrives. Use join(), which waits for the tasks without requesting cancellation, and drive shutdown() from a separate task:

#![allow(unused)]
fn main() {
use dynomite::embed::ServerBuilder;
use dynomite::conf::DataStore;
async fn f() -> Result<(), Box<dyn std::error::Error>> {
let handle = ServerBuilder::new("p")
    .listen("127.0.0.1:0".parse()?)
    .dyn_listen("127.0.0.1:0".parse()?)
    .data_store(DataStore::Valkey)
    .servers(vec![dynomite::conf::ConfServer::parse("127.0.0.1:6379:1")?])
    .tokens_str("0")
    .build()?
    .start()
    .await?;

let shutdown = handle.clone();
tokio::spawn(async move {
    tokio::signal::ctrl_c().await.ok();
    shutdown.shutdown().await.ok();
});

handle.join().await;   // parks until shutdown() drains the tasks
Ok(())
}
}

Where to next

• Embedding API Overview -- the two-tier API and the component diagram.
• Server Lifecycle -- the full ServerBuilder chain, ServerHandle control surface, events, snapshots, and the SemVer policy.
• Hooks and Traits -- Datastore, SeedsProvider, CryptoProvider, MetricsSink, each with a default impl and an example.
• Cookbook -- task-oriented recipes.
• The runnable examples, each with a walk-through: embedded_minimal, embedded_single_node, embedded_cluster3.

Tutorial: Vector + Text + Regex Search via valkey-cli

This walkthrough takes a fresh checkout to a running search stack on your laptop in about ten minutes. Every command in this chapter is copy-paste runnable against the binary you build below; the responses you see here are the verbatim wire output captured while writing the chapter.

By the end you will:

• have a single dynomited node listening on 127.0.0.1:18402, backed by a local Redis on 127.0.0.1:22122;
• have run k-NN vector search, trigram substring search, and TRE approximate regex search via valkey-cli;
• understand the wire-protocol shape of each FT.* response and how to encode the binary float blob a vector query needs.

1. Prerequisites

You need:

• Rust 1.90 or newer (cargo --version should report >= 1.90).
• valkey-cli and valkey-server (any 6.x, 7.x, or 8.x release works).
• python3 (used as a small encoder for binary float blobs).
• On Linux, libopenblas-dev (Debian/Ubuntu) or the equivalent openblas package on your distro. The vector engine pulls in turbovec, which links against OpenBLAS at the system layer.
• The tre-sys git submodule, which the FT.REGEX path uses for approximate-regex matching.

If you have Nix, nix develop from the repo root pins all of the above. Otherwise, install the system packages explicitly:

# Debian / Ubuntu
sudo apt-get install -y valkey-server redis-tools libopenblas-dev python3

# macOS (Homebrew)
brew install redis openblas python3

Pull the submodule the first time you build:

git submodule update --init crates/tre-sys/vendor/tre

2. Build and start dynomited

Build the server with the riak feature enabled. The feature pulls in the storage and admin paths the FT.* commands need:

cargo build --release -p dynomited --features riak

Start a local Redis on the default backing port (22122). Dynomite proxies non-FT.* keys to this Redis; FT.* commands are answered from the in-process vector + text registry, so the backing Redis only sees your raw HSET / HGET / DEL traffic:

mkdir -p /tmp/dynomite-tutorial
cd /tmp/dynomite-tutorial
valkey-server --port 22122 --daemonize yes \
    --dir /tmp/dynomite-tutorial \
    --pidfile /tmp/dynomite-tutorial/redis.pid \
    --logfile /tmp/dynomite-tutorial/redis.log
valkey-cli -p 22122 PING

Expected output:

PONG

Drop a minimal dynomited config in the same scratch directory:

cat > /tmp/dynomite-tutorial/dynomite.yml <<'EOF'
tutorial:
  listen: 127.0.0.1:18402
  dyn_listen: 127.0.0.1:18401
  stats_listen: 127.0.0.1:18422
  data_store: 0
  servers:
  - 127.0.0.1:22122:1
  tokens: '437425602'
  dyn_seed_provider: simple_provider
EOF

The keys are:

• listen: is the client-facing RESP port. Your valkey-cli connects here.
• dyn_listen: is the inter-node DYN_O_MITE port. We are not running a cluster yet but the field is required.
• stats_listen: is the HTTP stats endpoint (/stats, /cluster-info.txt, Prometheus metrics).
• data_store: 0 selects the Redis backing family.
• servers: lists the backing-store endpoints. We point at the local Redis from the previous step.

Start dynomited from the workspace root, with the absolute path to the config file. It runs in the foreground; background it with & so the shell stays usable:

cd /home/<you>/ws/dynomite        # the repo root
nohup ./target/release/dynomited \
    -c /tmp/dynomite-tutorial/dynomite.yml \
    > /tmp/dynomite-tutorial/dynomited.log 2>&1 &
sleep 2
valkey-cli -p 18402 PING

Expected output:

PONG

If the PONG does not come back, inspect /tmp/dynomite-tutorial/dynomited.log. The most common cause is one of the three ports above being in use; pick a different number for the listener that conflicts and re-edit the YAML.

3. Vector search: FT.CREATE, HSET, FT.SEARCH KNN

Create a four-dimensional cosine HNSW index against the docs: prefix. title is a TEXT field and vec is the vector field:

valkey-cli -p 18402 FT.CREATE myidx \
    ON HASH PREFIX 1 docs: \
    SCHEMA \
        title TEXT \
        vec VECTOR HNSW 6 TYPE FLOAT32 DIM 4 DISTANCE_METRIC COSINE

Expected output:

OK

The HNSW 6 token reads as "HNSW algorithm with six attribute pairs following": TYPE FLOAT32, DIM 4, and DISTANCE_METRIC COSINE. RediSearch chose this counted-attribute shape; we accept it unchanged.

Confirm the index registered, then peek at its metadata:

valkey-cli -p 18402 FT.LIST
valkey-cli -p 18402 FT.INFO myidx

Expected output:

myidx
index_name
myidx
algorithm
HNSW
distance_metric
COSINE
vector_field
vec
vector_type
FLOAT32
dim
4
prefixes
docs:
schema_fields
title
TEXT
num_docs
0
tracked_rows
0

Encode a vector

A FLOAT32 DIM 4 vector is sixteen bytes: four IEEE-754 single precision floats, each four bytes, all in little-endian order. For [1.0, 0.0, 0.0, 0.0] the bytes are:

python3 -c "import struct; print(struct.pack('<4f', 1.0, 0.0, 0.0, 0.0).hex())"

0000803f000000000000000000000000

Read left to right: 0x3f800000 is the IEEE-754 encoding of 1.0, written little-endian as 00 00 80 3f; the next twelve bytes are three zero floats.

If you do not have Python handy, printf works for fixed values:

printf '\x00\x00\x80\x3f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' | xxd

00000000: 0000 803f 0000 0000 0000 0000 0000 0000  ...?............

The mistake to avoid: native byte order is not always little-endian. Use Python's < format prefix or struct.pack('<4f', ...); >4f will silently produce big-endian bytes that decode as different floats, and the index will store and search the wrong values.

Insert documents

valkey-cli -x reads its last argument from standard input. Pipe the binary blob through it so the shell never has to handle the embedded NUL bytes:

python3 -c "import struct,sys; sys.stdout.buffer.write(struct.pack('<4f', 1.0, 0.0, 0.0, 0.0))" \
    | valkey-cli -p 18402 -x HSET docs:1 title "first hello world" vec

python3 -c "import struct,sys; sys.stdout.buffer.write(struct.pack('<4f', 0.9, 0.1, 0.0, 0.0))" \
    | valkey-cli -p 18402 -x HSET docs:2 title "second hello there" vec

python3 -c "import struct,sys; sys.stdout.buffer.write(struct.pack('<4f', 0.0, 1.0, 0.0, 0.0))" \
    | valkey-cli -p 18402 -x HSET docs:3 title "third orthogonal" vec

Expected output (one line per HSET):

2
2
2

The 2 is the count of fields written by HSET (title and vec), not a status code. If you see -ERR dimension mismatch: index=4, payload=N your packed vector was not exactly sixteen bytes; double-check the <4f format and that you sent four floats.

Run a k-NN query

The query vector goes on the wire the same way: sixteen bytes of little-endian f32. Place PARAMS 2 blob at the end of the command so the binary blob piped via -x lands in the right argument slot:

python3 -c "import struct,sys; sys.stdout.buffer.write(struct.pack('<4f', 1.0, 0.0, 0.0, 0.0))" \
    | valkey-cli -p 18402 -x FT.SEARCH myidx '*=>[KNN 3 @vec $blob]' PARAMS 2 blob

Expected output:

3
docs:1
__vec_score
0.000000
title
first hello world
docs:2
__vec_score
0.006116
title
second hello there
docs:3
__vec_score
1.000000
title
third orthogonal

The wire shape is RESP *7: total count, then three (doc-id, fields) pairs. Each fields array contains the implicit __vec_score (cosine distance, smaller is closer) plus every metadata field the index stores. Hits are returned closest-first.

Add a RETURN <n> <field>... clause to project a subset, and a LIMIT <off> <cnt> clause to paginate. Always keep PARAMS last when you pipe the vector via -x:

python3 -c "import struct,sys; sys.stdout.buffer.write(struct.pack('<4f', 1.0, 0.0, 0.0, 0.0))" \
    | valkey-cli -p 18402 -x FT.SEARCH myidx '*=>[KNN 3 @vec $blob]' \
        RETURN 1 title LIMIT 0 5 PARAMS 2 blob

Expected output (only title is projected; __vec_score is omitted):

3
docs:1
title
first hello world
docs:2
title
second hello there
docs:3
title
third orthogonal

4. Text search: trigram substring lookup

The title field above was already declared TEXT. Substring queries use the @field:substring form. Insert a few more rows so the result sets are interesting:

for i in 10 11 12 13; do
    case $i in
      10) t="hello world" ;;
      11) t="helo world" ;;
      12) t="helllo world" ;;
      13) t="hxllo world" ;;
    esac
    python3 -c "import struct,sys; sys.stdout.buffer.write(struct.pack('<4f', 0.${i}, 0.0, 0.0, 0.0))" \
        | valkey-cli -p 18402 -x HSET docs:$i title "$t" vec
done

Expected output (one 2 per row).

Run the substring query:

valkey-cli -p 18402 FT.SEARCH myidx '@title:hello'

Expected output:

3
docs:1
title
first hello world
docs:10
title
hello world
docs:2
title
second hello there

Notice that helo, helllo, and hxllo are not matches: the text path is exact substring, not approximate. For that, jump to FT.REGEX in the next section.

Inspect the trigram funnel via FT.EXPLAIN

dynomited does not implement Redis MONITOR (the proxy semantics make that command impossible to replay faithfully across the cluster). Instead, FT.EXPLAIN prints the trigrams the planner extracted from the query string. That is the same trigram set the funnel uses to walk the postings list:

valkey-cli -p 18402 FT.EXPLAIN myidx '@title:hello'

Expected output:

@title: SUBSTRING
  field: title
  query: hello
  index: trigram+bloom
  trigrams: ["hel", "ell", "llo"]

Three trigrams of three bytes each. Each trigram has a postings list of doc IDs that contain it, the planner intersects them, and a final exact-substring recheck filters the candidate set.

Compare with a vector EXPLAIN:

valkey-cli -p 18402 FT.EXPLAIN myidx '*=>[KNN 5 @vec $blob]'

Expected output:

VECTOR KNN
  index: myidx
  algorithm: HNSW
  metric: COSINE
  field: vec
  k: 5
  dim: 4
  param: $blob

5. Regex search: FT.REGEX with K=0, K=1, K=2

FT.REGEX <idx> <field> <pattern> [K=<n>] is a Dynomite extension on top of the RediSearch surface. K=0 is exact match, K=1 allows one edit (insert, delete, or substitute), K=2 allows two. The underlying engine is TRE (the same library that powers agrep(1)).

K=0: exact match. Only the four documents that literally contain hello come back.

valkey-cli -p 18402 FT.REGEX myidx title 'hello' K=0

Expected output:

3
docs:1
title
first hello world
docs:2
title
second hello there
docs:10
title
hello world

K=1: one typo. The result set picks up helo, helllo, and hxllo:

valkey-cli -p 18402 FT.REGEX myidx title 'hello' K=1

Expected output:

6
docs:1
title
first hello world
docs:2
title
second hello there
docs:10
title
hello world
docs:11
title
helo world
docs:12
title
helllo world
docs:13
title
hxllo world

K=2: two typos. With this corpus the result set is the same as K=1; on a larger corpus you would see additional matches drift in:

valkey-cli -p 18402 FT.REGEX myidx title 'hello' K=2

Real regex syntax works too. The pattern is anchored by the underlying TRE engine; group, alternation, and character class metacharacters all behave the way regex(7) documents:

valkey-cli -p 18402 FT.REGEX myidx title 'h(e|x)l*o' K=0

Expected output:

6
docs:1
title
first hello world
docs:2
title
second hello there
docs:10
title
hello world
docs:11
title
helo world
docs:12
title
helllo world
docs:13
title
hxllo world

K must be a non-negative integer. A negative value is a parse error:

valkey-cli -p 18402 FT.REGEX myidx title 'hello' K=-1

Expected output:

ERR syntax error: FT.REGEX: invalid K= value -1

6. Combined queries: text plus vector ranking

A document with both a TEXT body and a VECTOR field exposes two query shapes against the same row. The current build keeps the two queries separate at the wire layer; the application combines them.

Add a body-bearing document:

valkey-cli -p 18402 FT.ALTER myidx ADD body TEXT
python3 -c "import struct,sys; sys.stdout.buffer.write(struct.pack('<4f', 0.5, 0.5, 0.0, 0.0))" \
    | valkey-cli -p 18402 -x HSET docs:30 \
        title "with body" \
        body "this body talks about machine learning" \
        vec

Expected output:

OK
3

Filter by body keyword:

valkey-cli -p 18402 FT.SEARCH myidx '@body:machine'

Expected output:

1
docs:30
body
this body talks about machine learning

Rank the same set by vector distance to a query point. The doc matching the body filter should come back first if its vector is closest:

python3 -c "import struct,sys; sys.stdout.buffer.write(struct.pack('<4f', 0.5, 0.5, 0.0, 0.0))" \
    | valkey-cli -p 18402 -x FT.SEARCH myidx '*=>[KNN 3 @vec $blob]' PARAMS 2 blob

Expected output (truncated; closer matches first):

3
docs:30
__vec_score
0.000000
body
this body talks about machine learning
title
with body
...

To compose the two: run the text query, take its doc IDs, then run KNN and filter the KNN result down to that doc-ID set client-side. A native hybrid query syntax (RediSearch's (@body:machine)=>[KNN ...] pre-filter form) is on the roadmap; track it via the stage/ft-search-distributed worktree. Until that lands, the two- phase application pattern above is the correct shape.

7. Operations: FT.INFO, FT.LIST, FT.DROPINDEX, FT.ALTER

FT.INFO <name> prints schema metadata, index counters, and the configured prefixes:

valkey-cli -p 18402 FT.INFO myidx

The output is a flat key/value array. The integer fields you will care about most:

• num_docs: how many documents the underlying engine considers live.
• tracked_rows: rows the registry has indexed so far. Drift between the two indicates stale entries from a partial flush; the AAE scheduler reconciles this on the next pass.

FT.LIST (or its alias FT._LIST) returns every registered index name:

valkey-cli -p 18402 FT.LIST

Expected output:

myidx

FT.ALTER <idx> ADD <field> <type> adds a metadata field to a live index. Subsequent HSETs that supply the new field get indexed; rows written before the ALTER do not retroactively pick it up. Supported types are TEXT and TAG:

valkey-cli -p 18402 FT.ALTER myidx ADD genre TAG

FT.DROPINDEX <idx> [DD] removes the index. With DD, the underlying documents are deleted from the backing store; without it, the rows survive and you can re-index them with another FT.CREATE:

valkey-cli -p 18402 FT.CREATE empty ON HASH PREFIX 1 emp: \
    SCHEMA body TEXT vec VECTOR HNSW 6 TYPE FLOAT32 DIM 4 \
    DISTANCE_METRIC COSINE
valkey-cli -p 18402 FT.DROPINDEX empty
valkey-cli -p 18402 FT.LIST

Expected output:

OK
OK
myidx

8. Common gotchas

The patterns below have caught everyone porting their first RediSearch app to dynomite:

• Vector bytes are little-endian f32, packed end-to-end. A DIM 4 TYPE FLOAT32 index expects exactly sixteen bytes per vector, four bytes per dim, low byte first. The Python idiom is struct.pack('<4f', a, b, c, d). The < is mandatory; the default is host byte order.

• Wrong vector length is a clean -ERR, not a corruption. An HSET that ships fifteen, seventeen, or zero bytes for a four-dim index produces:

  -ERR dimension mismatch: index=4, payload=N
  

  Nothing is written. Fix the encoder and retry.

• FT.SEARCH on an empty index returns zero results, not an error. A freshly-created index responds with *0 to any query. The error path is reserved for parse and schema problems.

• K must be >= 0 in FT.REGEX. K=-1, K=foo, and K= all parse-fail with a structured -ERR.

• Text fields are byte-level, with UTF-8 implicit. Trigrams are three raw bytes, not three Unicode codepoints. ASCII substrings match exactly. Multi-byte sequences (latin-1 high bytes, CJK codepoints, emoji) work, but only when the query bytes are an exact contiguous prefix of the encoded form. Match cafe, not the U+00E9 character; the latter would split across two trigrams the user did not type.

• valkey-cli -x reads stdin into the LAST argument. Always put the binary blob's PARAMS clause at the end of your FT.SEARCH command line. Putting RETURN, LIMIT, or SORTBY after PARAMS 2 blob will not work because the blob lands one slot too early.

• MONITOR is not implemented. The proxy layer cannot replay a monitor stream faithfully across the cluster. Use FT.EXPLAIN to inspect the planner output, the stats endpoint (http://127.0.0.1:18422/stats) for counters, and dyn-admin metrics for the Prometheus surface.

9. Cleanup

When you are done, stop the server and the backing Redis:

pkill -f "dynomited.*tutorial"
valkey-cli -p 22122 SHUTDOWN NOSAVE 2>/dev/null || true
rm -rf /tmp/dynomite-tutorial

10. Multi-node preview

A single dynomite node already accepts every FT.* command. Three dynomited processes form a cluster by sharing a dyn_seeds: list, so writes hash to the right node and reads return from a quorum.

Cluster topology is documented in docs/book/src/operations/distribution.md; example three-node configurations live in crates/dynomited/conf/redis_rack{1,2,3}_node.yml. In the current build each node maintains its own FT.* registry: KNN, substring, and regex queries answer from the local node only. The cluster- wide broadcast that fans an FT.SEARCH across every primary peer and merges the top-K hits is in flight on the stage/ft-search-distributed branch. Watch docs/dynvec/fold-into-redis-path.md for the design overview and the corresponding journal entry for status.

11. Where to go from here

• Embedding cookbook: embedding/cookbook.md covers spinning a dynomite server up inside another Rust binary, with full FT.* support against a custom datastore trait.
• Architecture: architecture.md walks the request lifecycle from the listener through the dispatcher, the cluster routing layer, and the backend pool.
• Vector design: docs/dynvec/architecture.md documents the dynvec engine that backs HSET indexing, HNSW graph layout, and the encoding/distance modules.
• Text design: docs/dyntext/design.md documents the trigram + bloom funnel that backs @field:substring and FT.REGEX.
• Production-readiness evidence: the chaos-test reports under dist/chaos-reports/v0.1.0/ contain the full failure-injection results for the multi-host pre-tag pass. multi-host-pass-8-redis.md is the most recent Redis-backend run.

Architecture

This page is populated by the stage that ports the corresponding subsystem (see PLAN.md). Until then it intentionally stays brief so the manual structure is visible.

The Ring and the Token Space

This chapter covers how a key becomes a token, how a token maps to an owning node, how tokens are assigned to nodes, and how the rack-as-replica model layers full copies of the ring across fault domains. It closes with how the ClusterDispatcher decides between serving a request locally and forwarding it over the DNODE protocol.

Keys, hashes, and tokens

A client request carries a key -- the bytes between the command verb and its value. Dynomite feeds those bytes through the pool's configured hash function and interprets the result as a token: a coordinate on a one-dimensional ring.

flowchart LR
  K["key bytes<br/>(user:1042)"] --> H{{hash function}}
  H --> T["token<br/>(ring coordinate)"]
  T --> R{{ring lookup}}
  R --> N["owning node"]

A key is hashed to a token; the token is looked up on the ring; the lookup names the owning node. Each arrow is a pure function -- no I/O, no coordination.

The hash function is selected per pool from the configured hash knob. The runtime enum HashType covers the full upstream set (Murmur, Murmur3, the FNV family, CRC16/32, Jenkins, MD5, and the default one-at-a-time). The configuration-layer enum is mapped onto the runtime enum by map_hash in cluster/dispatch.rs; that mapping is the single seam so the dispatcher, the reaper, and the Dyniak replica router all agree on which hash a pool uses.

The token as a signed big integer

Tokens are not u64s. They are stored as a signed-magnitude big integer, DynToken, holding up to four little-significance-first 32-bit words plus a sign (Negative, Zero, Positive). This representation exists for one reason: to compare and parse tokens bit-identically across peers.

Comparison

Sign first, then word length, then word-by-word from the most significant word. A negative token always sorts below a positive one.

Textual parsing

An optional leading -, then base-10 digit groups folded by a fixed radix constant. The constant is deliberately the on-the-wire value peers expect, not a clean power of ten, so a Rust node and a peer agree on every token string.

Hashing into the token

The hash output is loaded into the token's magnitude words; a 32-bit hash occupies one word.

Mapping a token to a node: the continuum

Each rack stores its slice of the ring as a continuum: a vector of (token, peer_idx) points sorted ascending by token. Building the continuum is a rebuild pass over the pool's peer list; rebuild_continuums clears every rack's continuum, appends each peer's tokens onto the owning rack, then sorts each touched rack once.

Lookup is a left-leaning binary search in vnode::dispatch. Given a search token t:

• if t is greater than the largest continuum token, wrap to the first point;
• if t is less than or equal to the first continuum token, also return the first point;
• otherwise, return the smallest continuum entry whose token is greater than or equal to t (upper-bound, (a, b] semantics).

flowchart TD
  S["search token t"] --> C1{"t > last token?"}
  C1 -->|yes, wrap| F["first point owns t"]
  C1 -->|no| C2{"t <= first token?"}
  C2 -->|yes| F
  C2 -->|no| B["binary search:<br/>smallest point with token >= t"]

The dispatch function reproduces the reference ring semantics exactly: a key past the end of the ring wraps to the first owner, and every other key is owned by the next point clockwise.

The ring is conceptually circular, but the storage is a sorted array; the wraparound branch is what makes the last segment of the array and the first point share responsibility for the arc that crosses the ring's zero point.

flowchart LR
  subgraph ring["one rack's continuum"]
    A["token 10<br/>peer 0"] --> B["token 20<br/>peer 1"]
    B --> Cc["token 30<br/>peer 2"]
    Cc -.wrap.-> A
  end

Three peers, three tokens. A key hashing to 15 is owned by peer 1 (upper bound of 20); a key hashing to 35 wraps to peer 0. This is exactly the behaviour pinned by the dispatch doctests.

Token assignment and vnodes

Each peer carries a token list. In the simplest deployment a peer owns a single token -- its position on the ring -- and the nodes in a rack partition the ring into as many arcs as there are nodes. A node's token is the end of the arc it owns (upper-bound semantics), so the node with token 20 owns the half-open arc (10, 20] when its neighbour holds 10.

A peer may hold more than one token. Each additional token is an independent continuum point pointing back at the same peer index, so a single physical node can occupy several positions on the ring. This is the vnode (virtual node) idea: more, smaller arcs per node smooth out the load imbalance that a handful of large arcs would produce, and they make rebalancing on membership change move less data per moved token.

One token per node

Simple, and the classic single-token-per-node deployment. Arc boundaries are the configured tokens; the seed list carries them.

Many tokens per node (vnodes)

Each token is a separate continuum point for the same peer index. Smoother load, cheaper rebalancing, larger continuum.

Racks are replicas

Replication topology in Dynomite is expressed through the datacenter and rack hierarchy, not through a separate replica-count parameter attached to each key.

• A datacenter owns one or more racks.
• Within a datacenter, each rack holds a full copy of the ring. The number of racks in a DC is the replication factor (n_val) for that DC.
• Within a rack, nodes partition the token space. No two nodes in the same rack own the same arc; together they cover the whole ring exactly once.

Put differently: to place a key with a replication factor of three inside one datacenter, you run three racks, and the key lands on one node in each rack -- the node whose arc contains the key's token.

flowchart TB
  subgraph DC["datacenter dc1 (n_val = 3 racks)"]
    subgraph r1["rack r1 (full ring copy)"]
      a1["node A1<br/>arc (max,10]"]
      a2["node A2<br/>arc (10,20]"]
      a3["node A3<br/>arc (20,max]"]
    end
    subgraph r2["rack r2 (full ring copy)"]
      b1["node B1"]
      b2["node B2"]
      b3["node B3"]
    end
    subgraph r3["rack r3 (full ring copy)"]
      c1["node C1"]
      c2["node C2"]
      c3["node C3"]
    end
  end

One datacenter, three racks, three nodes per rack. The ring is copied across racks (replication) and partitioned across nodes within a rack (sharding). A key's replica set is one node from each rack: the node whose arc owns the key's token.

Each rack's continuum is built and searched independently. When the dispatcher plans a request it walks every rack in every datacenter, runs the ring lookup once per rack, and the union of the per-rack owners is the key's replica set. Because each rack is a full copy, the per-rack lookup always finds exactly one owner (or none, when the rack is empty).

flowchart LR
  K["key -> token t"] --> R1["rack r1 lookup"]
  K --> R2["rack r2 lookup"]
  K --> R3["rack r3 lookup"]
  R1 --> P1["node A2"]
  R2 --> P2["node B1"]
  R3 --> P3["node C3"]
  P1 & P2 & P3 --> RS["replica set for t"]

The same token is resolved once per rack. The three owners -- one per rack -- form the replica set. This is the input to the quorum machinery described in Replication and Consistency.

Cross-datacenter placement

Multiple datacenters each hold their own set of racks, so a key is replicated in every datacenter that has racks. When a write must be replicated into a remote DC, Dynomite does not fan out to every rack in that DC; it preselects one rack per remote DC to receive the cross-DC copy. The preselection (ServerPool::preselect_remote_racks) sorts each DC's racks by name and, for each remote DC, chooses the rack at local_rack_index % remote_rack_count. This spreads cross-DC replication traffic evenly across the remote racks instead of hammering one.

Routing: local versus remote

Once the dispatcher has the replica set, it splits it into local-DC and remote-DC targets and decides, per consistency level, which of them the request must actually reach. The full decision table is in Replication and Consistency; here is the routing mechanic that sits underneath it.

flowchart TD
  REQ["parsed request"] --> KEY{"has a key?"}
  KEY -->|no| LOCAL["route to local datastore"]
  KEY -->|yes| PLAN["hash to token, walk racks,<br/>collect routable replicas"]
  PLAN --> EMPTY{"any routable replica?"}
  EMPTY -->|no| NQ["no-quorum error"]
  EMPTY -->|yes| PART["partition into local-DC / remote-DC"]
  PART --> SELF{"target is<br/>the local node?"}
  SELF -->|yes| LOCAL
  SELF -->|no| FWD["forward over DNODE<br/>to the owning peer"]

A request either terminates at the local datastore or is framed and forwarded to the owning peer over the DNODE peer plane. The client never learns which; it always speaks to one node and gets one answer.

A peer is a routing candidate only when its PeerState is routable -- Normal or Joining. A Joining peer stays in the continuum until it transitions to Down or Leaving, so it keeps receiving traffic while it bootstraps. A Down peer is filtered out of the routable set for reads and for writes when hinted handoff is off; when hinted handoff is on, Down write targets are kept in the set so the dispatcher can record a hint (see Failure Handling).

When the owning peer is the local node, the request short-circuits to the local datastore rather than making a network hop to itself. This is the DispatchPlan::LocalDatastore branch, and it is also the plan for keyless commands and for requests explicitly tagged local-node-only.

Forwarding to a remote peer is a DNODE frame: the request bytes are relayed verbatim to the peer's dyn_listen socket, the peer serves them against its local backend, and the reply comes back through the per-request responder channel. A request forwarded to a remote peer is tagged as a forward so the receiver hands it straight to its datastore instead of re-hashing and re-planning it -- which would, in the worst case, bounce the request back. See the DNODE protocol chapter for the frame layout.

Rebalancing on membership change

The ring is a pure function of the topology, and the topology changes only through gossip. When a peer joins, leaves, or is marked down, gossip updates the peer table and the continuum is rebuilt from scratch by rebuild_continuums. Because the rebuild clears and repopulates every touched rack deterministically, two nodes that have converged on the same gossip view compute byte-identical continua and therefore route every key identically.

Where to go next

• Replication and Consistency -- how the replica set this chapter produces is turned into a client-visible answer under each tunable consistency level.
• Membership and Gossip -- how the topology that feeds the ring is discovered and kept converged.
• Failure Handling -- what happens to routing when a replica is down or partitioned away.
• DNODE protocol -- the peer-plane wire format used when a key's owner is remote.
• Configuration -- the hash, distribution, tokens, and per-bucket n_val knobs referenced here.

Replication and Consistency

This chapter defines the four consistency levels, spells out read and write semantics for each, walks a quorum write and a quorum read as sequence diagrams, explains what a no-quorum error means, and describes read repair on divergence. It is explicit that none of this is linearizable consensus, and it says why.

The model: replicate wide, agree loosely

Every key has a replica set: one node per rack in each datacenter, as described in The Ring and the Token Space. A write is sent to those replicas; a read gathers answers from them. The consistency level decides how many answers count as "enough" and what to do when they disagree.

flowchart LR
  C["client"] --> CO["coordinator node<br/>(any node)"]
  CO -->|fan out| R1["replica (rack r1)"]
  CO --> R2["replica (rack r2)"]
  CO --> R3["replica (rack r3)"]
  R1 & R2 & R3 -->|replies| CO
  CO -->|coalesced answer| C

The node the client happens to connect to is the coordinator for that request. It fans the request to the replica set, coalesces the replies according to the consistency level, and returns one answer.

The coordinator is stateless with respect to the key -- any node can play the role because any node can compute the replica set from the ring. There is no primary replica, no leader, and no log. This is the Amazon Dynamo lineage: available and partition-tolerant, with consistency as a tunable rather than a guarantee.

The four consistency levels

Consistency is expressed by the ConsistencyLevel enum, resolved per request from the pool's read_consistency / write_consistency (or a bucket-type override) in cluster/dispatch.rs. The per-reply coalescing rules live in proto/redis/coalesce.rs.

DC_ONE

One replica's answer suffices. Reads pick the rack-closest local replica and return its first reply; writes fan to every local-DC replica and ack on the first success. The default.

DC_QUORUM

A majority of local-DC replicas must agree. Quorum is floor(N/2)+1 over the local DC's replica count. Divergent replicas are recorded for read repair.

DC_SAFE_QUORUM

Every received local-DC reply must agree. Waits for all local replies; any divergence is a hard error, not a majority vote.

DC_EACH_SAFE_QUORUM

Per-DC unanimity. Every datacenter's replicas must internally agree; the local DC's agreed answer is returned to the client, and any replica in any DC that diverges from its DC's answer is marked for repair.

The scope of "quorum" is always the local datacenter's replica set, except for DC_EACH_SAFE_QUORUM, which imposes agreement in every DC. The quorum count uses the local replica count, and racks are the replicas, so N is the number of racks in the local DC.

DC_ONE

The weakest and fastest level.

• Read. The coordinator picks the single rack-closest local-DC replica -- lowest RackDistance cost, so same-rack beats same-DC beats remote -- and returns its first reply. No divergence is reported; read repair is a quorum-or-stronger feature.
• Write. The coordinator fans out to every local-DC replica for durability (each rack is a replica), and the coalescer acks the client on the first successful reply. The write still reaches all local replicas; the client just does not wait for all of them.

DC_QUORUM

The classic tunable-quorum level.

• Quorum is local_count / 2 + 1. With three racks that is two.
• The coordinator fans out to every local-DC replica, tallies votes keyed by reply payload, and declares the request done as soon as one payload gathers at least quorum votes. The most-voted payload is the winner; ties break toward the first-arrived majority.
• Replicas whose payload differs from the winner are reported as divergent targets and scheduled for read repair.
• If all local replies are in and no payload reached quorum, the coalescer picks the plurality winner (highest vote count, lowest peer index as tiebreak) and marks the rest divergent.

DC_SAFE_QUORUM

Like DC_QUORUM but it does not settle for a majority: it waits for every local-DC reply and requires them all to agree. If they do, the agreed reply is returned. If any local reply diverges once all are in, the request is a hard error rather than a majority decision. This trades availability for a stronger read guarantee within the DC.

DC_EACH_SAFE_QUORUM

The strictest level, spanning datacenters.

• Replicas are grouped by DC. Each DC must be internally unanimous once fully populated; an intra-DC divergence is an immediate error.
• The local DC's unanimous answer is the one returned to the client.
• Any replica -- in the local DC or a remote one -- whose payload differs from its DC's agreed answer is added to the divergent set for repair.
• Writes fan out per-DC (the plan walks the preselected rack in each remote DC in addition to the local-DC replicas).

A quorum write, step by step

sequenceDiagram
  participant Cl as client
  participant Co as coordinator
  participant R1 as replica r1
  participant R2 as replica r2
  participant R3 as replica r3
  Cl->>Co: SET k v
  Note over Co: hash k -> token,<br/>walk racks -> {r1, r2, r3},<br/>DC_QUORUM, quorum = 2
  par fan-out to all local replicas
    Co->>R1: forward SET
    Co->>R2: forward SET
    Co->>R3: forward SET
  end
  R1-->>Co: +OK
  R2-->>Co: +OK
  Note over Co: 2 matching acks >= quorum,<br/>declare Ready
  Co-->>Cl: +OK
  R3-->>Co: +OK (late)
  Note over Co: tracker already decided,<br/>late reply is dropped

A DC_QUORUM write with three replicas. The coordinator returns as soon as two replicas agree; the third reply, even though it arrives, does not produce a second answer -- the coalescer is one-shot.

The one-shot property matters: the per-request coalescer (CoalesceTracker) pins its decision the moment quorum is reached and reports every later reply as pending, so a straggler can be drained (and inspected for repair) without ever emitting a duplicate answer to the client.

A quorum read with divergence

sequenceDiagram
  participant Cl as client
  participant Co as coordinator
  participant R1 as replica r1
  participant R2 as replica r2
  participant R3 as replica r3
  Cl->>Co: GET k
  par fan-out
    Co->>R1: forward GET
    Co->>R2: forward GET
    Co->>R3: forward GET
  end
  R1-->>Co: v1 (stale)
  R2-->>Co: v2
  R3-->>Co: v2
  Note over Co: v2 has 2 votes >= quorum,<br/>r1 is divergent
  Co-->>Cl: v2
  Note over Co,R1: schedule read-repair write of v2 to r1

A DC_QUORUM read where one replica is stale. The majority value v2 is returned to the client, and the stale replica r1 is scheduled for a read-repair write so the next read finds it consistent.

Reply equivalence is by exact wire-byte comparison, with the error flag folded into the key: a successful reply and an error reply never coalesce even if their bytes match. This keeps an error on one replica from being mistaken for agreement.

No-quorum errors

When the topology cannot satisfy a request, the dispatcher returns a DynomiteNoQuorumAchieved error rather than a stale or partial answer. This happens when:

• the pool has no peers, or the key resolves to no routable replica (DispatchPlan::NoTargets);
• under a strict level, replies diverge and no agreement is possible (DC_SAFE_QUORUM divergence, DC_EACH_SAFE_QUORUM intra-DC divergence);
• every replica fan-out failed to send and no hint could be recorded.

flowchart TD
  P["plan request"] --> T{"routable replicas?"}
  T -->|none| NQ["DynomiteNoQuorumAchieved"]
  T -->|some| F["fan out, coalesce"]
  F --> D{"consistency satisfied?"}
  D -->|yes| OK["return coalesced reply"]
  D -->|strict level diverged| NQ

A no-quorum error is a deliberate refusal to answer, not a crash. It tells the client that the requested consistency level could not be met with the replicas currently reachable.

A no-quorum error is a safety response: the system would rather refuse than return an answer that violates the requested level. The failure-cause metrics count each no-quorum branch so an operator can distinguish "nobody home" from "replicas disagreed under a strict level".

Read repair

Read repair is how divergence discovered during a quorum read gets healed without a separate scan. When the coalescer reports divergent targets, the dispatcher schedules a repair write of the winning value to each stale replica through the same per-peer channels it used for the fan-out. The repair plumbing lives in proto/redis/repair/:

• reconcile picks a single response from the per-replica set and, where supported, produces the read-repair side effect;
• make builds the repair message when responses disagree;
• rewrite handles command-level rewrites (for example, turning SMEMBERS into a deterministic SORT ALPHA under DC_SAFE_QUORUM so set ordering does not read as divergence, and rewriting a write into a timestamped Lua script so a later reconcile can pick the newer value);
• clear emits metadata cleanup after a delete.

sequenceDiagram
  participant Co as coordinator
  participant St as stale replica
  Note over Co: quorum read found St divergent,<br/>winner = v2
  Co->>St: repair write (v2)
  St-->>Co: +OK
  Note over St: next read of k on St returns v2

Read repair is opportunistic: it heals exactly the replicas that a real read touched and found stale. Replicas nobody reads are healed by anti-entropy instead (see below).

Read repair only heals what reads observe. Keys that are written but rarely read, or replicas that were down during the read, are reconciled by the background anti-entropy path -- the Merkle-tree repair described in Failure Handling and, for the Dyniak layer, in Dyniak AAE.

Choosing a level

Latency-sensitive, tolerant of staleness

DC_ONE. One local reply, no waiting on the slow rack.

Balanced read-your-writes within a DC

DC_QUORUM read and write. A quorum write plus a quorum read overlaps in at least one replica, so a quorum read after a quorum write observes the write.

Strong intra-DC agreement, willing to error on divergence

DC_SAFE_QUORUM.

Cross-DC agreement

DC_EACH_SAFE_QUORUM, at the cost of waiting on every DC.

Levels are set per pool and can be overridden per bucket type, so a single deployment can serve a latency-sensitive cache at DC_ONE and a correctness-sensitive dataset at DC_QUORUM from the same nodes. The knobs are documented in Configuration.

Where to go next

• The Ring and the Token Space -- how the replica set that quorum operates over is computed.
• Membership and Gossip -- how the coordinator learns which replicas exist and are up.
• Failure Handling -- hinted handoff, and the anti-entropy path that heals what read repair does not.
• DNODE protocol -- the frames the coordinator sends to replicas.

Membership and Gossip

This chapter covers how nodes find each other through seeds, how gossip propagates state and converges, the shape of a peer's identity and why an advertised address matters, and how the gossip plane feeds the failure detector described in Failure Handling.

Why gossip and not a coordinator

A central membership service is simpler to reason about -- one authority, one answer -- but it is also a single point of failure and a single point of partition. If the coordinator is unreachable, either the cluster stops accepting membership changes or it forks into disagreeing halves.

Seeds: the bootstrap list

A brand-new node knows nothing about the cluster except its seeds: a list of host:port:rack:dc:tokens entries supplied through the seeds provider (the static dyn_seeds list, or a dynamic provider). Parsing is done by parse_seed_node / parse_seed_blob: entries are separated by |, and each entry's fields are split from the right so a host containing colons parses correctly. The token field may be a single big-integer or a comma-separated list (for a vnode node).

10.0.0.1:8101:rackA:dc1:1383429731|10.0.0.2:8101:rackB:dc1:2147483648

Seeds are a bootstrap hint, not the authority. A node contacts its seeds to enter the gossip mesh, but once it has gossiped it knows about peers that were never in its seed list, and it keeps functioning if a seed is down -- as long as some reachable seed lets it join the mesh. The seeds provider is re-queried at most once per seeds_check_interval (30s by default), so a changing seed list is picked up without restarting the node.

flowchart LR
  N["new node"] -->|contact seeds| S1["seed 10.0.0.1"]
  S1 -->|"gossip: here is the whole cluster"| N
  N -.learns.-> P2["peer 10.0.0.3<br/>(not in seed list)"]
  N -.learns.-> P3["peer 10.0.0.4<br/>(not in seed list)"]

Seeds get a node into the mesh; gossip does the rest. After the first round the new node knows peers that were never in its seed list.

The gossip round

Gossip runs on a fixed interval (gos_interval, 1000 ms by default). Each round the node:

1. Queries the seeds provider if the seeds-check interval has elapsed, and reconciles the returned entries against its peer and gossip tables.
2. Applies its per-node add-or-update state machine to each observed node.
3. Forwards state to one randomly chosen peer: a GOSSIP_SYN if the local node is still joining, or its local state digest if it is normal.

The reconciliation is the interesting part. It is driven by GossipState::add_or_update, keyed on (dc, rack, primary-token) and on (dc, rack, host):

flowchart TD
  OBS["observed node<br/>(dc, rack, token, host, ts)"] --> K{"token known?"}
  K -->|no| ADD["insert new node"]
  K -->|yes| H{"same host?"}
  H -->|no| REP["replace IP,<br/>re-index by name"]
  H -->|yes| TS{"newer timestamp?"}
  TS -->|no| UNC["ignore (stale/dup)"]
  TS -->|yes| ST{"state changed?"}
  ST -->|yes| SC["update state + ts"]
  ST -->|no| TU["update ts only"]

The add-or-update state machine. A node is inserted when new, has its IP replaced when the token is known but the host moved, and is timestamp- or state-updated when only the clock or the lifecycle moved forward. Stale updates are dropped by the timestamp check.

Two properties fall out of this design:

Token is identity

A node is identified on the ring by its (dc, rack, primary token). If the same token reappears under a new host, that is an IP change for the same logical node, not a new node -- the old name index is dropped and re-created.

Timestamps break ties

Every update carries an epoch-seconds timestamp. An update with a timestamp no newer than what is stored is ignored, so out-of-order or duplicate gossip cannot roll state backward.

Convergence

Because each round pushes state to one random peer, information spreads epidemically: a fact known to one node reaches roughly all connected nodes in O(log N) rounds. There is no barrier, no acknowledgement of global receipt, and no notion of "the membership is now final" -- the cluster is always converging toward the latest observed state, and under a stable topology it reaches a fixed point where every node's view agrees.

sequenceDiagram
  participant A as node A (knows: A up)
  participant B as node B
  participant C as node C
  participant D as node D
  Note over A: A learns "C is up" from a round
  A->>B: gossip digest (incl. C up)
  Note over B: B now knows C up
  B->>D: gossip digest (incl. C up)
  A->>C: gossip digest
  Note over C,D: after a few rounds every node<br/>holds the same view: A,B,C,D up

Epidemic propagation. Each node forwards to one random peer per round; a new fact reaches the whole connected cluster in a logarithmic number of rounds. Convergence is eventual, not instantaneous.

When the topology stops changing, gossip converges and the per-rack continua rebuilt from that converged view are byte-identical across nodes -- which is precisely the determinism that makes ring routing coordination-free (see The Ring and the Token Space).

Peer identity and the advertised address

A peer is matched by its endpoint's pname -- the host:port string. The gossip handler records inbound heartbeats against the peer whose PeerEndpoint::pname() matches the sender, and the failure detector is keyed the same way. This makes the advertised address load-bearing.

The reason is mechanical: gossip carries a node's own claimed address, and every other node stores and matches on that address. A wildcard is not an address any peer can send to or reconcile against. The advertised address must be the one peers actually reach the node on.

flowchart LR
  subgraph node["node binding 0.0.0.0:8101"]
    L["listener<br/>0.0.0.0:8101"]
  end
  node -->|"advertises 10.0.0.7:8101"| GOS["gossip identity"]
  GOS -->|peers match on| PN["pname 10.0.0.7:8101"]
  PN --> FD["failure detector<br/>keyed by pname"]

Bind address and advertised address are different things. The listener may be wildcard; the gossip identity must be a routable host:port, because that is the key every peer matches on.

From gossip to peer state

Once gossip is wired, the gossip handler (GossipHandler) is the single owner of peer-state transitions. It does two things with each inbound heartbeat and each periodic tick:

• On heartbeat. It feeds the peer's phi-accrual failure detector and, if the peer's suspicion level is below threshold and it is not already Normal, promotes it to Normal immediately. This gives a just-contacted peer a snappy first-contact transition instead of waiting a full tick.
• On tick. It re-evaluates every non-local peer's suspicion level and toggles between Normal and Down: a peer is Normal once at least one heartbeat has been recorded and its phi is at or below threshold, and Down when no heartbeat has ever arrived or its phi exceeds threshold.

flowchart LR
  GH["gossip heartbeat<br/>(pname match)"] --> FD["phi-accrual detector<br/>record_heartbeat"]
  FD -->|"phi <= threshold"| UP["promote to Normal"]
  TICK["periodic tick"] --> EVAL["evaluate phi(now)<br/>per non-local peer"]
  EVAL -->|"phi > threshold<br/>or no heartbeat"| DOWN["mark Down"]
  EVAL -->|"phi <= threshold<br/>and heartbeat seen"| UP

Gossip feeds the failure detector; the failure detector decides liveness. Membership (who exists, where on the ring) and liveness (who is up) are separate concerns tracked by the same handler.

There is a second, coarser detector on the raw gossip table (GossipState::run_failure_detector) that ages a node to Down when its last-seen timestamp is older than 40 * gos_interval. The phi-accrual detector is the primary, adaptive mechanism; the timestamp aging is a backstop for nodes that stopped gossiping entirely. Both feed the same PeerState, and the mechanics of that state machine are the subject of Failure Handling.

Shutdown and departure

A node leaving the cluster announces its own departure through gossip. The handler's mark_down_pname marks the departing peer Down without consulting the failure detector, so the dispatcher stops routing to it immediately rather than waiting for phi to accrue. When a peer is removed and later re-added, its failure detector is reset so historical jitter does not bias the fresh suspicion value.

Where to go next

• Failure Handling -- the phi-accrual detector, the peer state machine, auto-eject / auto-rejoin, and what happens during a partition.
• The Ring and the Token Space -- how the converged membership view becomes the routing ring.
• DNODE protocol -- the wire format carrying GOSSIP_SYN and the state digests between peers.
• Configuration -- the dyn_seeds, gos_interval, and gossip enable knobs.

Failure Handling

This chapter walks the phi-accrual failure detector and the peer state machine, auto-eject and auto-rejoin, durable hinted handoff, read repair, the anti-entropy overview, and the guarantees that hold across a partition.

Detecting failure: phi-accrual

A naive detector flips a peer between "alive" and "dead" on a missed-heartbeat count. That is brittle: a network hiccup trips it, and a genuinely slow link never does. Dynomite uses a phi-accrual detector (PhiAccrual), the same family Cassandra, Akka, and Riak deploy, which produces a continuous suspicion level phi(t) instead of a boolean.

Phi is the negative log-probability that a heartbeat would not have arrived yet given the historical inter-arrival distribution. Modelling arrivals as exponential gives the closed form the detector computes:

    phi(t) = elapsed_ms / (mean_interval_ms * ln(10))

phi = 1.0

~10% chance the heartbeat is merely late.

phi = 2.0

~1% chance.

phi = 8.0

~10^-8 -- "almost certainly dead". This is the default threshold (DEFAULT_THRESHOLD), matching Cassandra's phi_convict_threshold.

The detector keeps a sliding window of the last ~100 inter-arrival times per peer, so it adapts to each link's real cadence. A jittery link that normally varies wildly will not be convicted by a one-second gap that would convict a metronome-steady link -- higher observed variance widens the tolerated gap. Two guards keep the math honest: the mean interval is clamped at a floor (default 1s) so burst arrivals cannot make phi spike, and phi is 0.0 when no heartbeat has ever been recorded (no data is not the same as dead) or when fewer than two heartbeats give no inter-arrival sample.

The peer state machine

Each peer carries a PeerState. Only Normal and Joining are routable; Down and Leaving are not. A remote peer starts Down and is promoted only after its first below-threshold heartbeat; the local node starts Joining.

stateDiagram-v2
  [*] --> Down: remote peer created
  [*] --> Joining: local node created
  Joining --> Normal: bootstrap complete,<br/>heartbeats flowing
  Down --> Normal: first heartbeat,<br/>phi <= threshold
  Normal --> Down: phi > threshold<br/>(silence) or announced departure
  Down --> Normal: heartbeats resume,<br/>phi <= threshold
  Normal --> Leaving: graceful departure
  Leaving --> [*]

Peer lifecycle. The routable states are Normal and Joining; the failure detector drives the Normal/Down toggle, and a graceful departure moves a peer to Leaving so it stops receiving traffic before it goes.

The gossip handler is the single owner of these transitions once gossip is wired. On each inbound heartbeat it records the arrival and, if phi is below threshold, promotes the peer to Normal at once. On each periodic tick it evaluates every non-local peer and toggles Normal / Down from the current phi. A peer that flaps -- goes silent, is convicted, then resumes -- produces exactly one Normal -> Down and one Down -> Normal transition per cycle, each surfaced as a metric and a structured event.

sequenceDiagram
  participant P as peer
  participant FD as phi detector
  participant GH as gossip handler
  loop steady state
    P->>FD: heartbeat (1 Hz)
    Note over FD: phi stays < 1.0
  end
  Note over P: peer goes silent
  GH->>FD: tick: phi(now)?
  FD-->>GH: phi = 12 (> 8)
  GH->>GH: mark peer Down, emit PeerDown
  Note over P: peer recovers
  P->>FD: heartbeat resumes
  GH->>GH: phi <= 8, promote Normal, emit PeerUp

Silence raises phi past the threshold on the next tick and convicts the peer; a resumed heartbeat promotes it back. The detector's window means recovery is judged on the same adaptive baseline as conviction.

Auto-eject and auto-rejoin

Eviction and readmission are gossip-driven, not operator-driven.

• Auto-eject. When a peer crosses the phi threshold (or announces its own departure), the handler marks it Down. The next continuum rebuild drops it from the routable set, and the dispatcher stops planning requests to it. No manual intervention, no config edit.
• Auto-rejoin. When a Down peer starts gossiping again and its phi falls below threshold, the handler promotes it back to Normal on the next heartbeat, and it re-enters the routable set on the next rebuild. Its failure detector is reset on re-add so old jitter does not bias the fresh baseline.

The whole loop is closed by gossip and the detector; the operator's role is to fix the underlying fault, not to click a node back in.

Hinted handoff

A write whose target replica is Down would normally just miss that replica. Hinted handoff makes the write durable anyway: the coordinator records a hint -- the on-the-wire request bytes, the intended peer index, and an expiry deadline -- and a background drainer ships it to the peer once the peer returns to Normal.

sequenceDiagram
  participant Cl as client
  participant Co as coordinator
  participant R1 as replica r1 (up)
  participant R2 as replica r2 (DOWN)
  participant HS as hint store
  Cl->>Co: SET k v (DC_QUORUM)
  Co->>R1: forward SET
  Co->>HS: record hint for r2
  Note over Co: synthetic +OK on r2's behalf<br/>counts toward quorum
  R1-->>Co: +OK
  Co-->>Cl: +OK
  Note over R2: r2 recovers -> Normal
  HS->>R2: drainer ships hinted SET
  R2-->>HS: +OK, hint cleared

Hinted handoff keeps a write durable across a replica outage. The hint counts toward the consistency threshold at write time and is replayed to the replica when it returns, so the down replica catches up without a full repair.

Handoff is only active when the hint store is wired and the pool sets enable_hinted_handoff. When active, Down write targets are kept in the routable set so the dispatcher can hint them; a synthetic +OK is fed to the coalescer on the hinted target's behalf so the surviving replicas plus the hint can meet the consistency threshold. Without handoff, a Down target is simply skipped.

Durability of hints

The hint store (HintStore) has two modes:

RAM-only (HintStore::new)

Hints live only in per-peer in-memory queues and are lost if the coordinator restarts.

Durable (HintStore::open, with hint_dir)

One append-only segment file per peer under <dir>/peer-<idx>.hints. Each record is framed with a length, an IEEE CRC-32 of the body, a wall-clock deadline, and the payload. Hints survive a coordinator restart -- replay re-anchors each deadline to the current clock and drops any already-expired hint.

The durable format is torn-tail safe: a crash mid-append leaves at most a trailing partial record, and replay detects it two ways -- a short read before the body completes, or a body whose CRC does not match -- and stops cleanly at the first damaged record, keeping every intact record before it. A torn tail never panics and never surfaces an error from open.

Hints are bounded by max_bytes and expire after hint_ttl_seconds (default one day). Over-capacity or zero-TTL enqueues are rejected so the store cannot grow without bound; when the store is full the coordinator falls back to its no-quorum error path rather than silently dropping the write.

Read repair

Read repair heals divergence that a quorum read observes: the majority value is returned to the client and the stale replicas are written back through the same channels. It is covered in full in Replication and Consistency; the important boundary is that read repair only heals replicas a real read touched. Keys that are written but rarely read, and replicas that were down during the read, are left for anti-entropy.

Anti-entropy: Merkle-tree repair

Anti-entropy is the background reconciliation that does not depend on a client reading a key. Replicas periodically compare compact Merkle-tree digests of their key ranges; where the trees differ, only the divergent sub-ranges are exchanged and reconciled, so a full replica comparison costs a tree walk rather than a full data transfer.

flowchart TD
  R1["replica r1<br/>Merkle root"] --> CMP{"roots equal?"}
  R2["replica r2<br/>Merkle root"] --> CMP
  CMP -->|yes| DONE["ranges agree,<br/>nothing to do"]
  CMP -->|no| DESC["descend into<br/>differing subtrees"]
  DESC --> EX["exchange only the<br/>divergent key ranges"]
  EX --> REC["reconcile, write back"]

Merkle-tree anti-entropy narrows a whole-range comparison to just the sub-ranges that actually differ. Equal roots mean the replicas agree and no data moves.

This is the safety net beneath read repair and hinted handoff: it catches divergence that neither of the other two mechanisms reached -- writes that were never read back, hints that expired before the peer recovered, or data that drifted during a long partition. The full anti-entropy design, including the transactional Dyniak layer's reconciliation, is documented in Dyniak AAE.

What happens during a partition

Put the three mechanisms together and the partition story is:

sequenceDiagram
  participant A as DC-A side
  participant B as DC-B side
  Note over A,B: link between DCs drops
  Note over A: A's detector convicts B's peers (phi > 8)
  Note over B: B's detector convicts A's peers
  Note over A,B: each side keeps serving with its<br/>own reachable replicas
  Note over A: writes to B's replicas -> hinted (if enabled)
  Note over A,B: link heals
  A->>B: gossip reconverges membership
  A->>B: hints drained to recovered replicas
  A->>B: read repair + anti-entropy reconcile divergence

A cross-DC partition. Each side stays available on its own replicas, records hints for the unreachable side, and reconciles via gossip, hint drain, read repair, and anti-entropy once the link heals.

Availability

Both sides of a partition keep serving reads and writes against the replicas they can still reach. There is no leader to lose, so neither side stalls.

Durability within quorum

A write that meets its consistency level on the reachable side is not lost: it is on that side's replicas, and -- if hinted handoff is enabled -- queued for the unreachable replicas. When the partition heals, gossip readmits the peers, hints drain, and anti-entropy plus read repair close any remaining gap.

Consistency

Divergent writes on the two sides are reconciled after the heal, not prevented during the split. Under the strict consistency levels a request that cannot meet its level returns a no-quorum error rather than a divergent answer (see Replication and Consistency).

Where to go next

• Membership and Gossip -- how the failure detector is fed and how ejection / readmission propagate.
• Replication and Consistency -- read repair, the consistency levels, and what a no-quorum error means.
• The Ring and the Token Space -- how a Down peer drops out of routing on the next continuum rebuild.
• Dyniak AAE -- the full Merkle-tree anti-entropy and the transactional reconciliation path.
• Configuration -- the enable_hinted_handoff, hint_dir, hint_ttl_seconds, and failure-detector knobs.

Configuration

This page covers configuration knobs that go beyond the basic pool stanza. Refer to the inline rustdoc on dynomite::conf::ConfPool for the exhaustive field list; the pages here describe the operator-facing surface in more detail.

Endpoints and Unix sockets

The listen: (client plane), dyn_listen: (peer plane), and stats_listen: (HTTP stats) directives take a host:port address (IPv4 or IPv6). A value that begins with / is treated as a Unix domain socket path instead:

dyn_o_mite:
  listen: /var/run/dynomite/client.sock
  dyn_listen: 127.0.0.1:8101
  stats_listen: 127.0.0.1:22222

The servers: entries (the backend datastore) accept the same shape: host:port:weight [name] or, for a Unix-socket backend, /path/to/socket:weight [name]. The port is reported as 0 for Unix-socket entries.

  servers:
  - /var/run/valkey/valkey.sock:1 valkey-local

Backend authentication

• redis_requirepass (string, default unset): when set, the password is sent as AUTH <pw> on every backend connection immediately after the TCP handshake. It mirrors the Valkey / Redis requirepass server option. Memcache backends are not authenticated (AUTH is RESP-specific; memcache binary SASL is not implemented), so the knob is ignored for a memcache data_store.

Dyniak store

• noxu_path (path): filesystem directory the in-process Noxu DB environment opens at. Required when data_store: dyniak is selected and ignored otherwise. The directory must be writable; an existing environment is reused, otherwise one is created. A dyniak pool serving the Riak surface needs a binary built with --features riak.

Bucket types

A bucket type is a named bundle of routing properties that applies to every key whose on-the-wire form starts with the bucket prefix. Operators use bucket types to give different key classes different SLAs from inside a single pool: cache-style keys can sit on DC_ONE while transactional keys ride DC_EACH_SAFE_QUORUM, and the dispatcher swaps in the right settings on a per-request basis without needing more pools, more listeners, or client-side routing.

The wire convention is intentionally simple. The bucket name is the byte sequence before the first / in the key; everything else (including the slash itself) is the user-visible key body. A key with no / has no bucket, and the dispatcher falls back to the pool defaults (or to default_bucket_type, when one is named).

dyn_o_mite:
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  tokens: '101134286'
  servers:
  - 127.0.0.1:22122:1
  data_store: 0
  read_consistency: DC_ONE
  write_consistency: DC_ONE
  bucket_types:
  - name: sessions
    read_consistency: DC_QUORUM
    write_consistency: DC_EACH_SAFE_QUORUM
    n_val: 3
  - name: cache
    read_consistency: DC_ONE
    write_consistency: DC_ONE
    n_val: 1
  default_bucket_type: cache

With this stanza:

• GET sessions/abc123 reads with DC_QUORUM; writes use DC_EACH_SAFE_QUORUM and may fan out across DCs.
• GET cache/u/9000 uses DC_ONE reads and writes.
• GET plain-key (no slash) falls through to default_bucket_type: cache and inherits its DC_ONE routing.
• Removing default_bucket_type makes the slashless and unknown-prefix cases inherit the pool-level defaults.

n_val caps the number of replicas a single request fans out to. 0 means "no cap" (the default; the consistency level alone decides fan-out). A positive n_val truncates the plan to its first n_val targets, where rack-local replicas are already ordered first.

Validation enforces unique bucket-type names, valid consistency strings, and that default_bucket_type (when set) names an entry in bucket_types. Invalid stanzas are caught by dynomited --test-conf before the daemon starts.

Hinted handoff

Hinted handoff is the data-availability tier that lets writes succeed even when one of their target peers is unreachable. The dispatcher records the on-the-wire request bytes plus the intended peer index in a node-local hint store; a background drainer periodically replays the hints once the peer returns to the gossip-defined Normal state.

The feature is off by default. Enable it per-pool in YAML:

dyn_o_mite:
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  tokens: '101134286'
  servers:
  - 127.0.0.1:22122:1
  data_store: 0
  read_consistency: DC_QUORUM
  write_consistency: DC_QUORUM
  enable_hinted_handoff: true
  hint_ttl_seconds: 86400
  hint_store_max_bytes: 67108864
  hint_drain_interval_ms: 30000
  hint_dir: /var/lib/dynomite/hints

Knobs:

• enable_hinted_handoff (bool, default false): master switch. When false the dispatcher's behaviour is unchanged and a Down or unreachable target is silently skipped.
• hint_ttl_seconds (uint, default 86400 / 24 hours): per-hint expiry. Hints older than this are dropped during the next drainer sweep so the store stays bounded.
• hint_store_max_bytes (uint, default 67108864 / 64 MiB): upper bound on the cumulative payload bytes of pending hints. Once the store is full the dispatcher falls back to its non-handoff error path (DynomiteNoQuorumAchieved) and the next drainer sweep reclaims space when peers come back online.
• hint_drain_interval_ms (uint, default 30000): cadence of the drainer sweep. Every tick (a) drops expired hints and (b) replays the queued hints for every peer that has transitioned to Normal.
• hint_dir (path, default unset): when set, the hint store is durable. It keeps one append-only segment file per peer under this directory and replays them at startup, so hints queued for a temporarily down peer survive a coordinator restart. When unset the hint store is RAM-only and queued hints are lost on restart. Ignored when enable_hinted_handoff is false.

Operational notes:

• Hinted handoff applies only to writes. Reads against a Down peer remain a no-data-to-hint situation and continue to follow the legacy "skip the target and fall through to the consistency check" path.
• The hint store is RAM-only by default. Set hint_dir to make it durable: the store then writes one append-only segment file per peer under that directory and replays them at startup, so pending hints survive a node restart. Without hint_dir, a node restart drops every pending hint, so size hint_store_max_bytes and the drainer cadence conservatively.
• The synthesised reply that the dispatcher feeds the coalescer on a hinted target's behalf is +OK\r\n. This is the correct shape for SET-style writes (the dominant case) but may show up as a "divergent target" for request types whose reply is an integer or a multibulk (e.g. DEL). The hint is still recorded and replayed; the surviving real replies determine the coalesced answer the client sees, and read-repair (which only fires on GET) is not affected.
• The hint_drain_interval_ms value should be small enough to notice peer recoveries promptly but large enough that the drainer does not dominate the per-tick scheduling cost when many peers come back at once. The default 30 seconds matches the gossip period.

Search index persistence

The optional RediSearch FT.* surface (crates/dynomite-search) is wired into dynomited behind the search Cargo feature. By default the FT.* index registry is purely in-memory: index definitions, indexed documents, text fields, and suggestion dictionaries are lost on a process restart and the client must recreate them.

• search_index_dir (path, default unset): when set, the search registry snapshots its full state (index schemas, indexed documents, per-TEXT-field contents, and FT.SUG* suggestion dictionaries) to a CBOR file under this directory and reloads it on restart. A clean checkout, a process kill, or a chaos restart no longer drops the FT.* surface: the snapshot survives on disk and the next startup reloads every index without the client re-issuing FT.CREATE or re-feeding data.

Operational notes:

• The snapshot is written atomically (write to a sibling *.tmp, flush + fsync, rename over the live file), so a crash mid-write never leaves a half-written snapshot; the prior good snapshot survives. A stray *.tmp from an interrupted write is ignored on load and overwritten on the next save.
• Snapshots are taken periodically (every five seconds) and once more on clean shutdown. A crash between snapshots loses at most the un-snapshotted delta; the FT.* workload tolerates re-creation, so the recreate-on-miss path becomes a fallback rather than the primary recovery once persistence is on.
• When search_index_dir is unset the registry never touches disk; behaviour is identical to releases before this knob existed.

Riak mode

The optional Riak protocol surface (crates/dyniak) is wired into dynomited behind the riak Cargo feature. When the binary is built with --features riak, the riak: block of the pool body controls the PBC listener, the HTTP gateway, and the optional active-anti-entropy (AAE) scheduler. The block is parsed and validated even under the default build so YAML files authored against the Riak-enabled binary still validate without the feature flag; under the default build the fields are inert at run time.

my_pool:
  ...
  riak:
    pbc_listen: 127.0.0.1:8087
    http_listen: 127.0.0.1:8098
    quic_listen: 127.0.0.1:8089
    aae_enabled: true
    aae_full_sweep_interval_seconds: 86400
    aae_segment_interval_seconds: 60
    tls_cert: /etc/dynomited/riak.crt
    tls_key: /etc/dynomited/riak.key
    tls_ca: /etc/dynomited/ca.pem
    wasm_modules:
    - id: wordcount
      path: /etc/dynomited/wasm/wordcount.wasm

Every field is optional. Setting only pbc_listen enables the PBC listener with no HTTP gateway; setting only http_listen runs HTTP without PBC. The two listeners share a single backing Datastore so request accounting accumulates in one place.

The CLI offers three matching overrides for the same knobs: --riak-pbc-listen=HOST:PORT, --riak-http-listen=HOST:PORT, and --riak-aae-enabled. They are visible in dynomited --help only when the binary was built with --features riak. See Riak mode for operator-facing details.

Distribution modes

The pool's distribution: directive selects the algorithm that maps a hashed key to one of the rack's peers. Two modes are first-class and supported indefinitely:

random_slicing is the recommended mode for new deployments that do not need byte-identical compatibility with an existing operator-managed token plan. The technique is described in detail in docs/design/random-slicing-integration.md; the operator-facing pitch is "you cannot accidentally configure a hole in the ring", which is the failure mode that bit chaos pass-3 (3-of-4 hosts running with a 4-host token plan, 25% of the ring orphaned).

The legacy aliases ketama, modula, and random are accepted for backward compatibility with the C reference's vocabulary. They emit a tracing::warn! line at config-load time and collapse to vnode at runtime.

Default per binary build

• Default builds keep vnode as the default. Existing YAML files validate and run unchanged.
• --features riak builds that configure a Riak listener (riak.pbc_listen or riak.http_listen) flip the default to random_slicing. Riak-shaped deployments inherit Riak's full-coverage partition invariant by default. Operators can still set distribution: vnode explicitly to override.

Migration: shadow mode

A --distribution-shadow=<vnode|random_slicing> CLI flag (and the matching distribution_shadow: YAML key) lets an operator run both modes simultaneously: routing follows the live distribution:, but the dispatcher also computes the would-be peer for the shadow mode and bumps the distribution_shadow_disagreement_total Prometheus counter when they disagree. Use this to validate a migration before flipping the YAML.

The dyn-admin distribution-dump --node <host:stats-port> subcommand pretty-prints the configured live and shadow modes plus the cumulative disagreement counter, so an operator can verify each node's view from one place.

dyn_o_mite:
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  tokens: '101134286'
  servers:
  - 127.0.0.1:6379:1
  data_store: 0
  distribution: random_slicing       # gap-free partition table
  distribution_shadow: vnode         # validate the new mode
                                     # against the old one

Protocols

A Dynomite node fronts one backend data store, chosen by the data_store: configuration key. Each backend has its own client wire protocol:

• Valkey (RESP) -- data_store: valkey (alias redis). The vendor-neutral RESP request/response protocol.
• Memcache -- data_store: memcache. The Memcached ASCII text protocol.
• Dyniak (Riak PBC / HTTP) -- data_store: dyniak. The built-in Riak-compatible store, exposed over Riak Protocol Buffers (PBC) and an HTTP gateway. Requires a binary built with --features riak.

Independently of the client protocol, nodes talk to one another over the DNODE framing on the peer plane.

Valkey (RESP)

The valkey data store speaks RESP, the request/response protocol that Valkey and Redis share. The wire format is vendor neutral: the engine parses and routes RESP without any Valkey- or Redis-specific assumptions, so the historical data_store: redis alias and a data_store: valkey value select the exact same code path.

The implementation lives in dynomite::proto::redis.

Wire format

RESP frames are length-prefixed. A request is an array of bulk strings; the first element is the command name and the remainder are its arguments:

*2\r\n$3\r\nGET\r\n$3\r\nfoo\r\n

The engine consumes the stream with two byte-driven state machines: one for requests (redis_parse_req) and one for responses (redis_parse_rsp). The parser is total: any byte sequence either yields a complete message or a parse error, never a panic.

Command handling

Each parsed request is classified into a MsgType (for example ReqRedisGet, ReqRedisSet, ReqRedisDel). Classification drives:

• Routing -- the key (or hash tag, when hash_tag: is set) selects the target peers through the configured distribution.
• Multi-key fragmentation -- commands that carry several keys (MGET, MSET, DEL, ...) are split into per-peer fragments and the per-fragment replies are coalesced back into one client reply.
• Read repair -- on a quorum read whose replicas disagree, the repair surface issues corrective writes. Read repair fires on GET-class reads.
• Verification -- request shape is validated before dispatch.

Backend authentication

When redis_requirepass: is set, the engine sends AUTH <pw> on every backend connection immediately after the TCP (or Unix-socket) handshake. See Configuration.

Search extension (FT.*)

When dynomited is built with the search feature, the dynomite-search extension layers the RediSearch FT.* command family on top of the RESP dispatcher: FT.CREATE, FT.SEARCH, FT.REGEX, FT.AGGREGATE, and the FT.SUG* suggestion commands, plus vector KNN. Indexes are cluster-coordinated: a query fans out to every primary peer and the per-peer hits are merged and ranked. Set search_index_dir: to make the index registry durable across restarts. See the search tutorial for an end-to-end walk-through.

Memcache

The memcache data store speaks the Memcached ASCII (text) protocol. The implementation lives in dynomite::proto::memcache.

Wire format

Memcached commands are newline-terminated ASCII. The engine consumes the stream with one byte-driven state machine for requests (memcache_parse_req) and one for responses (memcache_parse_rsp). Like the RESP parser, the memcache parser is total: arbitrary input either yields a complete message or a parse error.

get user:42\r\n
set user:42 0 0 3\r\nabc\r\n

Command handling

Parsed requests are classified into MsgType values covering the storage, retrieval, arithmetic, and lifecycle commands: get, gets, set, add, replace, append, prepend, cas, delete, incr, decr, touch, and quit.

Classification drives the same machinery as the RESP path:

• Routing by key (or hash tag) through the configured distribution.
• Multi-key fragmentation for the multi-key get / gets forms, with per-peer reply coalescing.

Limitations

• The binary protocol is not implemented; only the ASCII protocol.
• Memcache backends are not authenticated. AUTH is RESP-specific, and memcache binary SASL is not implemented, so redis_requirepass: is ignored for a memcache data_store.
• Read repair is a placeholder for memcache: memcache has no versioned reconciliation the way the RESP path does.

DNODE wire protocol

DNODE is the framing Dynomite peers use to talk to one another. Every peer-to-peer message - datastore requests, datastore responses, gossip frames, and the AES key handshake - travels inside a DNODE envelope.

This page documents the framing semantically. The Rust implementation lives in dynomite::proto::dnode.

Frame layout

A DNODE frame is an ASCII-only header followed by an opaque payload:

   $2014$ <msg_id> <type> <flags> <version> <same_dc> *<mlen> <data> *<plen>\r\n
   <payload of <plen> bytes>

• The leading three spaces are part of the magic literal as it appears on the wire; the parser tolerates them as initial whitespace.
• $2014$ is a fixed magic literal that anchors header recovery.
• <msg_id> is the 64-bit unsigned message id, in decimal.
• <type> is the DmsgType discriminator, in decimal.
• <flags> is the bit field. Bit 0 is the encryption flag, bit 1 is the compression flag. The high nibble is reserved.
• <version> is the protocol version. The current version is 1.
• <same_dc> is 1 when the sender and receiver share a datacenter and 0 otherwise.
• *<mlen> is the byte length of the inline data field, expressed as *<decimal>.
• <data> is the inline data: either the single-byte placeholder d (data path) or a (gossip path), or the RSA-wrapped AES key during the crypto handshake.
• *<plen> is the byte length of the payload that follows, expressed as *<decimal>.
• \r\n (CRLF) terminates the header.
• The payload follows immediately after the CRLF and contains exactly plen bytes.

The header fields are always ASCII-decimal even when they encode binary values (the encryption flag, the type tag). The inline data field is the only header location that may contain arbitrary bytes; its length is fixed by the preceding *<mlen> so the parser can copy mlen bytes verbatim.

Message types

The reference engine and the Rust port share the following set of type discriminators:

Crypto handshake

The first frame on a freshly secured peer connection is a CRYPTO_HANDSHAKE envelope with the encryption flag set. The inline <data> field contains the AES-128 session key wrapped with the peer's RSA public key. The receiver unwraps the key with its private RSA key and stores it on the connection state; subsequent frames on the same connection encrypt their payload with that AES key.

Parser state machine

The Rust parser exposes a public state alphabet (DynParseState) that mirrors the reference engine's enum. The states correspond to the header fields in declaration order plus a trailing Done / PostDone / Unknown triple used by the recovery path:

Start
  -> MagicString    (after $2014$)
  -> MsgId
  -> TypeId
  -> BitField       (flags)
  -> Version
  -> SameDc
  -> DataLen        (after *)
  -> Data           (mlen bytes copied)
  -> SpacesBeforePayloadLen
  -> PayloadLen     (after *)
  -> CrlfBeforeDone
  -> Done           (after \n)

PostDone is the state the receiver enters after consuming and decrypting an encrypted handshake frame; the next bytes on the connection feed the datastore parser instead of the DNODE parser. Unknown is the recovery state used when the parser hits a byte that is not valid in the current state.

Encoding

The encoder produces a single contiguous header. The dmsg_write flavour emits the data-path placeholder (d) when no AES key payload is supplied; the dmsg_write_mbuf flavour emits the gossip placeholder (a) instead. Both encoders accept an optional RSA-wrapped AES key as the inline data field; when supplied, the encoder writes the wrapped bytes verbatim and updates <mlen> to match the wrapped key length.

Transports

A Dynomite node moves bytes over two transports. Both support IPv4 and IPv6, and both are available on the client plane, the peer (DNODE) plane, and the dyniak Riak listeners.

• TCP -- the historical default, always available.
• QUIC -- available when the engine is built with the quic Cargo feature.

The transport is transport-agnostic above the byte layer: the same connection state machines and protocol parsers run regardless of which transport carries the bytes. Only the socket plumbing differs.

TCP

TCP is the default transport and is always compiled in. It carries:

• the client plane (the listen: address serving RESP or memcache),
• the peer plane (the dyn_listen: address speaking DNODE between nodes), and
• the dyniak Riak PBC (riak.pbc_listen) and HTTP (riak.http_listen) listeners.

Addresses

listen:, dyn_listen:, and stats_listen: take an IPv4 or IPv6 host:port. A value beginning with / is a Unix domain socket path instead of a TCP address; see Configuration.

TLS

The peer plane runs over TLS when peer_tls_cert: and peer_tls_key: are set (with optional mutual-TLS via peer_tls_ca: and per-DC material via peer_tls_profiles:). The dyniak listeners terminate TLS when riak.tls_cert: and riak.tls_key: are set. In every case, setting one half of a cert/key pair without the other is rejected at configuration validation time.

When no TLS material is configured, the corresponding plane runs in plaintext, matching the historical default.

QUIC

QUIC is available when the engine is built with the quic Cargo feature, which pulls in the quiche crate. It is offered on the client plane and on the dyniak Riak PBC listener.

cargo build -p dynomited --features quic

Wire shape

The QUIC transport is intentionally thin. It wraps a single quiche::Connection in a tokio-driven event loop and serves the configured protocol over one bidirectional stream per accepted connection (the lowest client-initiated bidirectional stream). The transport-agnostic connection state machines and protocol parsers run unchanged on top; only the UDP socket and packet pump are QUIC-specific. Multi-stream multiplexing is left to future revisions.

TLS is mandatory

QUIC mandates TLS, so a QUIC listener always needs a certificate and key.

• Client plane: select QUIC with transport: quic and supply quic_cert_file: and quic_key_file:. The listener binds a UDP socket.
• dyniak PBC plane: set riak.quic_listen: to a host:port. It reuses the riak.tls_cert: / riak.tls_key: pair; setting quic_listen without both is rejected at validation time.

If a QUIC address is configured but the binary was built without the quic feature, dynomited fails fast at startup with a clean configuration error rather than silently ignoring the directive.

dyn_o_mite:
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  tokens: '0'
  data_store: 0
  servers:
  - 127.0.0.1:6379:1
  transport: quic
  quic_cert_file: /etc/dynomited/server.crt
  quic_key_file: /etc/dynomited/server.key

Crypto

This page is populated by the stage that ports the corresponding subsystem (see PLAN.md). Until then it intentionally stays brief so the manual structure is visible.

Embedding API

This chapter is the design contract for the dynomite::embed module landing in Stage 13. It pins the public surface that the implementer must hit so that callers can plan against it today.

Two ways to use this crate

Dynomite ships both a binary (dynomited) and a library (dynomite).

• As a server, dynomited parses a YAML file, builds a dynomite::embed::Server, and runs it under its own #[tokio::main]. The YAML schema is the same one documented in the reference C version; nothing about it is binary-only.
• As a library, an embedding program owns a tokio runtime of its own and constructs a Server directly. The same configuration fields are exposed as typed fluent setters. In addition, slots that cannot be expressed in YAML (custom transports, custom datastores, custom seed providers, custom crypto, custom metrics sinks) are available as typed setters.

Both paths share the same core: dynomited is a thin wrapper around the embedding API, not a parallel code path.

Two-tier API

The surface is deliberately split into a builder and a handle:

1. Server::builder() -> ServerBuilder - typed, fluent, validated at build() time. The builder owns no runtime state.
2. Server::start() -> ServerHandle - returns immediately. The tokio runtime owns all background work. The caller drives the server through the handle.

This split lets configuration errors surface synchronously while the running server is reduced to an async, observable, controllable object.

Component diagram

           +---------------------+
           | Embedding program   |
           +----------+----------+
                      |
                      | Server::builder()...build()
                      v
           +---------------------+
           |   ServerBuilder     |  (typed config + hooks)
           +----------+----------+
                      | .start()
                      v
           +---------------------+        +-----------------+
           |      Server         |------->|  tokio runtime  |
           +----------+----------+        +--------+--------+
                      | ServerHandle               |
                      v                            v
           +---------------------+   +-------------------------+
           |  control surface    |   |    background tasks     |
           |  shutdown / reload  |   |  conn pools (TCP/QUIC)  |
           |  stats / events     |   |  proxy + dnode listeners|
           |  inject_request     |   |  gossip + seeds refresh |
           |  peers / ring       |   |  proto parsers          |
           +---------------------+   |  stats aggregator       |
                                     |  entropy / repair       |
                                     +-------------------------+

Each block is documented in its own chapter:

• Server lifecycle - the ServerBuilder, ServerHandle, and ServerEvent shapes.
• Hooks and traits - the five pluggable traits, their default implementations, and when to write a custom one.
• Examples -- runnable, documented example programs that compose the surface above.

Stability

Until 0.1 is cut, every public-API change in dynomite::embed is recorded in docs/journal/ with an api-change: entry and a migration note. 0.1 is cut when the conformance suite is green; 1.0 is cut when SemVer is locked and cargo public-api shows no diffs across two PRs. See AGENTS.md Section 13.

Server lifecycle

dynomite::embed::Server is the primary entry point of the embedding API. This page pins the shape of the builder, the running handle, and the event stream.

ServerBuilder

impl Server {
    pub fn builder() -> ServerBuilder;
}

ServerBuilder is a fluent, owned builder. Every YAML-visible field on ConfPool (see crates/dynomite/src/conf/pool.rs) gets a matching setter. Setter names use Rust style; the YAML key is the documented mapping.

Representative YAML-mirroring setters (the full set is one-to-one with ConfPool fields):

fn listen(self, addr: SocketAddr) -> Self;
fn dyn_listen(self, addr: SocketAddr) -> Self;
fn stats_listen(self, addr: SocketAddr) -> Self;

fn hash(self, h: HashType) -> Self;
fn hash_tag(self, tag: [u8; 2]) -> Self;
fn distribution(self, d: Distribution) -> Self;
fn data_store(self, d: DataStore) -> Self;

fn timeout(self, ms: u32) -> Self;
fn backlog(self, n: u32) -> Self;
fn client_connections(self, n: u32) -> Self;
fn datastore_connections(self, n: u8) -> Self;
fn local_peer_connections(self, n: u8) -> Self;
fn remote_peer_connections(self, n: u8) -> Self;

fn preconnect(self, on: bool) -> Self;
fn auto_eject_hosts(self, on: bool) -> Self;
fn server_retry_timeout_ms(self, ms: u32) -> Self;
fn server_failure_limit(self, n: u32) -> Self;

fn datacenter(self, dc: impl Into<String>) -> Self;
fn rack(self, rack: impl Into<String>) -> Self;
fn tokens(self, t: TokenList) -> Self;
fn dyn_seeds(self, seeds: Vec<ConfDynSeed>) -> Self;
fn dyn_seed_provider(self, name: impl Into<String>) -> Self;

fn read_consistency(self, c: ConsistencyLevel) -> Self;
fn write_consistency(self, c: ConsistencyLevel) -> Self;
fn read_repairs_enabled(self, on: bool) -> Self;

fn secure_server_option(self, opt: SecureServerOption) -> Self;
fn pem_key_file(self, path: PathBuf) -> Self;
fn recon_key_file(self, path: PathBuf) -> Self;
fn recon_iv_file(self, path: PathBuf) -> Self;

fn mbuf_size(self, n: usize) -> Self;
fn max_msgs(self, n: usize) -> Self;
fn conn_msg_rate(self, n: u32) -> Self;
fn gos_interval_ms(self, ms: u32) -> Self;
fn stats_interval_ms(self, ms: u32) -> Self;
fn enable_gossip(self, on: bool) -> Self;

Two convenience constructors short-circuit the field-by-field path:

fn from_config(cfg: Config) -> ServerBuilder;
fn from_yaml_file(path: impl AsRef<Path>) -> Result<ServerBuilder, ConfError>;

In addition to the YAML-mirroring fields, the builder exposes typed setters for slots that have no YAML form. These are the only way to inject a custom hook implementation:

fn transport(self, t: Box<dyn Transport>) -> Self;
fn seeds_provider(self, p: Box<dyn SeedsProvider>) -> Self;
fn datastore(self, d: Box<dyn Datastore>) -> Self;
fn crypto(self, c: Box<dyn CryptoProvider>) -> Self;
fn metrics_sink(self, s: Box<dyn MetricsSink>) -> Self;

Each defaults to the in-crate implementation when unset. See Hooks and traits for the trait contracts.

build() validates the assembled config (the same ConfPool::validate used for YAML) and returns:

fn build(self) -> Result<Server, BuildError>;

Server::start

impl Server {
    pub fn start(self) -> ServerHandle;
}

start is non-blocking. It spawns the background tasks (listeners, gossip, stats, entropy) onto the current tokio runtime and returns. The returned ServerHandle is Clone + Send + Sync; multiple consumers can hold one. Dropping the last ServerHandle does not shut the server down - only shutdown() does.

ServerHandle

impl ServerHandle {
    pub async fn shutdown(&self) -> Result<(), ShutdownError>;
    pub async fn reload(&self, cfg: Config) -> Result<(), ReloadError>;

    pub fn stats(&self) -> Snapshot;
    pub fn describe_stats(&self) -> StatsManifest;

    pub fn subscribe_events(&self) -> EventStream;

    pub async fn inject_request(&self, req: Msg) -> Result<Msg, InjectError>;

    pub fn peers(&self) -> Vec<PeerSnapshot>;
    pub fn datacenters(&self) -> Vec<DatacenterSnapshot>;
    pub fn ring(&self) -> RingSnapshot;
}

• shutdown is the graceful path: drain listeners, close peer connections, flush stats, then return.
• reload is the in-process equivalent of SIGHUP. The new Config is validated before any state is touched; on failure the running config is left untouched.
• stats returns the live dynomite::stats::Snapshot (already used by the REST endpoint).
• describe_stats returns a machine-readable manifest of every stat the engine exposes (name, kind, unit), suitable for plugging into a MetricsSink.
• subscribe_events returns an async Stream<Item = ServerEvent> built on tokio::sync::broadcast. Lagging consumers receive a ServerEvent::Lagged { missed } marker and resume.
• inject_request hands a parsed Msg to the dispatcher as if it had arrived from a client, skipping the proxy listener. The returned future resolves when the response message is ready. This is the entry point used by the in-process tests in tests/embed_*.rs.
• peers, datacenters, and ring return point-in-time snapshots of cluster topology for inspection and dashboards.

ServerEvent

pub enum ServerEvent {
    PeerUp(PeerId),
    PeerDown { peer: PeerId, reason: PeerDownReason },
    ConfigReloaded { generation: u64 },
    GossipRound { round: u64, peers: u32 },
    AutoEjected { peer: PeerId, failures: u32 },
    RepairTriggered { key_hash: u64, dc: String },
    ConnectionAccepted { conn_id: ConnId, role: ConnRole },
    ConnectionClosed { conn_id: ConnId, reason: CloseReason },
    Lagged { missed: u64 },
}

Variants are non-exhaustive at the type level; consumers must use a wildcard arm.

Snapshot types

pub struct PeerSnapshot {
    pub id: PeerId,
    pub addr: SocketAddr,
    pub dc: String,
    pub rack: String,
    pub state: PeerState,
    pub tokens: Vec<DynToken>,
    pub last_seen_ms: Msec,
}

pub struct DatacenterSnapshot {
    pub name: String,
    pub racks: Vec<RackSnapshot>,
}

pub struct RingSnapshot {
    pub entries: Vec<RingEntry>, // (token, owner peer id)
    pub generation: u64,
}

These are owned, Clone, and free of internal locks. Holding one across an await is safe.

SemVer policy

Pre-1.0, every change in this module produces a docs/journal/ entry titled api-change: <summary> with a migration note and the cargo public-api diff. 1.0 is cut when the Stage 14 conformance suite is green, the Stage 15 fuzz soak is clean, and cargo public-api reports zero diffs over two consecutive PRs.

Implementation

Stage 13 lands the surface above in:

• crates/dynomite/src/embed/mod.rs - re-exports.
• crates/dynomite/src/embed/builder.rs - ServerBuilder.
• crates/dynomite/src/embed/server.rs - Server, ServerHandle.
• crates/dynomite/src/embed/events.rs - ServerEvent, EventStream.
• crates/dynomite/src/embed/snapshots.rs - PeerSnapshot, DatacenterSnapshot, RingSnapshot.
• crates/dynomite/src/embed/error.rs -- the EmbedError type.

The runtime tasks (gossip, stats, metrics, accept loops) live in server.rs next to ServerHandle; they are spawned by Server::start and observe a single tokio_util::sync::CancellationToken.

Hooks and traits

Five public traits expose every cross-cutting concern that an embedder may want to override. Each has at least one default implementation shipped in-crate so a vanilla embedding works with zero hook boilerplate.

The traits are object-safe (Box<dyn Trait> works) and Send + Sync so they can be shared across tokio tasks. All async methods return boxed futures (Pin<Box<dyn Future + Send + 'a>>) to keep the trait object-safe; in practice, callers use the #[async_trait] macro.

Datastore

The datastore trait is the abstraction over Redis / Memcached / anything-else that a Dynomite node fronts.

#[async_trait]
pub trait Datastore: Send + Sync + 'static {
    async fn connect(&self) -> Result<Box<dyn DatastoreConn>, DatastoreError>;
    fn protocol(&self) -> Protocol;        // Redis | Memcache | Custom
    fn supports(&self, cmd: MsgType) -> bool;
}

#[async_trait]
pub trait DatastoreConn: Send {
    async fn dispatch(&mut self, req: Msg) -> Result<Msg, DatastoreError>;
    async fn close(self: Box<Self>) -> Result<(), DatastoreError>;
}

When to implement. Front any storage with the Dynomite gossip / ring / quorum layer. The typical case is keeping the default Redis or Memcache implementation; custom impls are useful when the backing store lives in-process (no socket round-trip) or speaks a private protocol.

Default impls. dynomite::embed::hooks::RedisDatastore and dynomite::embed::hooks::MemcacheDatastore ship in-crate (plus MemoryDatastore for in-process use), each selected by data_store: in the YAML schema.

Custom example. An in-process B-Tree backend (InMemoryBTreeDatastore) that implements DatastoreConn::dispatch by calling directly into a BTreeMap<Bytes, Bytes> guarded by an async_lock::RwLock. Used in tests to remove socket and protocol overhead, and in single-process embeddings where the data lives next to the application logic.

SeedsProvider

Service-discovery integration. The seeds provider produces the list of peer Dynomite nodes at startup and on a configurable refresh interval.

#[async_trait]
pub trait SeedsProvider: Send + Sync + 'static {
    async fn fetch(&self) -> Result<Vec<Seed>, SeedsError>;
    fn refresh_interval(&self) -> Duration { Duration::from_secs(30) }
}

pub struct Seed {
    pub addr: SocketAddr,
    pub dc: String,
    pub rack: String,
    pub tokens: Vec<DynToken>,
}

When to implement. Any cluster topology that does not fit a static seeds list. K8s, Consul, and AWS-tagged-EC2 are the common cases.

Default impls (all under dynomite::embed::seeds):

• SimpleSeedsProvider - reads dyn_seeds: from the YAML.
• DnsSeedsProvider - resolves an A / AAAA record and treats each resolved address as a seed.
• FloridaSeedsProvider - HTTP poller for the Florida service that the reference C version integrates with.

Custom example. A K8sSeedsProvider that uses the kube-rs Api::watch stream over a Service selector. Each event mutates an internal Vec<Seed>; fetch returns a clone. The refresh interval is irrelevant when the watch stream pushes updates, so refresh_interval returns Duration::MAX.

Transport

Transport lives in dynomite::io::reactor. It captures any AsyncRead + AsyncWrite + Send + Unpin byte stream the engine should accept connections on.

pub trait Transport: AsyncRead + AsyncWrite + Send + Unpin {
    fn role(&self) -> ConnRole;
    fn peer_addr(&self) -> Option<SocketAddr>;
}

The embedding builder accepts a transport factory rather than a single transport, because every accepted connection needs its own transport instance:

#[async_trait]
pub trait TransportListener: Send + Sync + 'static {
    async fn accept(&self) -> Result<Box<dyn Transport>, TransportError>;
    fn local_addr(&self) -> Option<SocketAddr>;
}

When to implement. Anything beyond TCP and QUIC: Unix domain sockets, mutually authenticated TLS, in-memory pipes for tests, and embedded message-bus integrations.

Default impls. TcpTransport and the QUIC variant (QuicTransport / QuicListener) both ship in-crate, with their listener equivalents wired into the embedding builder.

Custom example. A UnixDomainTransport whose role is ConnRole::Client and whose peer_addr is None. Useful for co-located producer / consumer processes where TCP loopback overhead is unwanted. A second use case is MutualTlsTransport, which wraps a tokio_rustls::TlsStream after a client cert verification step that calls into an internal CA.

CryptoProvider

HSM / KMS abstraction. The default in-crate implementation is the existing dynomite::crypto::Crypto, which uses OpenSSL with a PEM-loaded RSA private key.

pub trait CryptoProvider: Send + Sync + 'static {
    fn rsa_size(&self) -> usize;
    fn rsa_encrypt(&self, plaintext: &[u8]) -> Result<Vec<u8>, CryptoError>;
    fn rsa_decrypt(&self, ciphertext: &[u8]) -> Result<Vec<u8>, CryptoError>;
    fn aes_encrypt(&self, plaintext: &[u8]) -> Result<Vec<u8>, CryptoError>;
    fn aes_decrypt(&self, ciphertext: &[u8]) -> Result<Vec<u8>, CryptoError>;
    fn aes_key(&self) -> &[u8; 32];
}

When to implement. Any deployment where the RSA private key must not leave a hardware boundary. The provider holds a handle to the HSM / KMS session; rsa_decrypt calls the device.

Default impl. dynomite::embed::crypto::OpensslCryptoProvider is a thin newtype around the existing dynomite::crypto::Crypto. Selected automatically when the YAML schema configures pem_key_file.

Custom example. An AwsKmsCryptoProvider that stores only the KMS key ARN locally and forwards rsa_decrypt calls to the Decrypt API. AES operations stay local on the AES key derived by the DNODE handshake; only RSA is offloaded.

MetricsSink

Exporter abstraction. The default sink is the built-in REST endpoint already implemented in dynomite::stats::rest.

#[async_trait]
pub trait MetricsSink: Send + Sync + 'static {
    async fn emit(&self, snapshot: &Snapshot) -> Result<(), MetricsError>;
    fn flush_interval(&self) -> Duration { Duration::from_secs(10) }
    fn manifest(&self) -> StatsManifest { StatsManifest::default() }
}

The aggregator periodically calls emit with the latest snapshot. Implementations are responsible for any wire-format conversion.

When to implement. Any monitoring backend that is not the built-in JSON REST endpoint.

Default impl. dynomite::embed::metrics::RestMetricsSink - mounts the JSON snapshot at the address from stats_listen:.

Custom examples:

• PrometheusMetricsSink - converts each Snapshot to the Prometheus exposition format and serves it on /metrics. Counters in the snapshot become _total; histograms are exported as _bucket series using the bucket boundaries from Snapshot.
• OtelMetricsSink - sends the snapshot through an opentelemetry-otlp exporter. flush_interval is set to match the OTel collector's expected push cadence (typically 60s).

Discoverability

Each trait lives in crates/dynomite/src/embed/hooks.rs and is re-exported at dynomite::embed. A typical embedder writes use dynomite::embed::SimpleSeedsProvider; without reaching into the internals.

Default implementations shipped in-crate:

• dynomite::embed::MemoryDatastore - in-process datastore for examples and tests.
• dynomite::embed::RedisDatastore / MemcacheDatastore - protocol-tagged datastores that carry the backend target; the wire protocol bridge runs in the dispatcher path.
• dynomite::embed::SimpleSeedsProvider / DnsSeedsProvider / FloridaSeedsProvider - wrappers around the in-crate providers from dynomite::seeds.
• dynomite::embed::RustCryptoProvider - wraps dynomite::crypto::Crypto.
• dynomite::embed::LoggingMetricsSink - emits one tracing::info event per flush.

The PrometheusMetricsSink mentioned in the design above is not shipped by Stage 13 to avoid adding a new workspace dependency; the LoggingMetricsSink is the default sink. Recorded as a Deviation in docs/parity.md.

Embedding cookbook

This page is the mainstream reference for embedding Dynomite as a library inside another Rust program. It complements Server lifecycle, Hooks and traits, and Examples: the lifecycle and hooks pages pin the API surface, the examples page sketches three end-to-end scenarios, and the cookbook below answers the common questions embedders ask in order.

When to embed vs daemon

Dynomite ships in two equally first-class shapes:

• dynomited (the binary, in crates/dynomited/) - reads a YAML file, runs the same dynomite::embed::Server engine under its own #[tokio::main], exposes a stats HTTP endpoint and a SIGHUP reload hook, and is what a typical operations team installs as a service. Choose the daemon when you want observability and lifecycle isolation from your application process and when you are happy proxying through a TCP socket.
• dynomite (the library, in crates/dynomite/) - exposes the same engine through the typed embed API. The host program owns the tokio runtime, plugs custom hooks, drives the engine through a ServerHandle, and reads metrics either by holding the live Arc<Stats> or by plugging a custom MetricsSink. Choose embedding when:
  • the storage layer is already in-process (an in-memory B-Tree, an sled, a RocksDB, ... ) and you want Dynomite-style ring routing and quorum without serialising to a wire format;
  • you want to drive Dynomite from your application's own lifecycle / shutdown signals (and to avoid double-running tokio runtimes);
  • you want bespoke metrics integration (a Prometheus registry your application already exposes, an OTLP pipeline, a custom Grafana dashboard adapter) without mounting a second HTTP listener;
  • your transport is non-TCP: Unix domain sockets, in-memory pipes for tests, mutual-TLS, or the upcoming QUIC variant.

The library and the daemon share the same engine and the same public API; choosing one over the other is purely an operational decision. This entire page assumes the library shape.

Smallest possible embedded server

The five-line example below is the canonical smallest embedding. It binds two ephemeral ports (the :0 syntax asks the kernel for a free port), spawns the engine, prints the post-bind addresses, and shuts down. The cookbook references this exact body as the "five-line embedded server"; the runnable form lives in embedded_minimal.rs.

use dynomite::embed::{Server, ServerBuilder};

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let handle = Server::start_with(
        ServerBuilder::new("dyn_o_mite")
            .listen("127.0.0.1:0".parse()?)
            .dyn_listen("127.0.0.1:0".parse()?),
    ).await?;

    eprintln!("up on {:?}", handle.listen_addr());
    handle.shutdown().await?;
    Ok(())
}

Run it:

cargo run --example embedded_minimal
embedded dynomite up; client listen=Some(127.0.0.1:NN) dnode listen=Some(127.0.0.1:NN)
shutdown ok

What the defaults give you when you do not call any other setter:

• one pool named "dyn_o_mite" (any string works);
• one stub backend so validation passes (override with .servers(...) or .datastore(...));
• a single token (0) on a single rack in a single datacenter (the local node);
• gossip off, retry timeout, mbuf budget, and stats interval at the same defaults dynomited would apply via apply_defaults;
• the in-crate MemoryDatastore standing in for the wire-level Redis bridge.

The defaults exist so the cookbook's "smallest" example fits in five lines; production embeddings always customise the backend and the topology.

Custom Datastore: plug a non-Redis store

Dynomite's gossip / ring / quorum layer is independent of the backing store. The default in-crate RedisDatastore and MemcacheDatastore front the two protocols Dynomite was born to proxy, but any type that implements dynomite::embed::Datastore slots in. Common targets:

• RocksDB, sled, redb, in-process B-Trees - host the storage layer in the same process as Dynomite to remove the socket round-trip and avoid serialising structured values to RESP / Memcache wire format.
• HTTP-fronted KV stores - call out via reqwest or hyper.
• The Riak K/V protocol - the workspace ships crates/dyniak on top of the same trait.

The trait surface (verbatim from hooks.rs):

#![allow(unused)]
fn main() {
use dynomite::embed::hooks::{BoxFuture, Datastore, DatastoreError, Protocol};
use dynomite::msg::{Msg, MsgType};
use std::sync::Arc;
use parking_lot::Mutex;
#[derive(Default, Clone)]
struct InMemoryDatastore {
    map: Arc<Mutex<std::collections::BTreeMap<u64, MsgType>>>,
}

impl Datastore for InMemoryDatastore {
    fn protocol(&self) -> Protocol { Protocol::Custom }

    fn dispatch(&self, req: Msg) -> BoxFuture<'_, Result<Msg, DatastoreError>> {
        let map = self.map.clone();
        Box::pin(async move {
            // Application logic here. SET stores; GET / anything
            // else replies with the previously stored kind.
            let stored = {
                let mut g = map.lock();
                if matches!(req.ty(), MsgType::ReqRedisSet) {
                    g.insert(req.id(), MsgType::RspRedisStatus);
                }
                g.get(&req.id()).copied().unwrap_or(MsgType::RspRedisStatus)
            };
            let mut rsp = Msg::new(req.id(), stored, false);
            rsp.set_parent_id(req.id());
            Ok(rsp)
        })
    }
}
}

Plug it via ServerBuilder::datastore:

#![allow(unused)]
fn main() {
use dynomite::embed::{Server, ServerBuilder};
use dynomite::embed::hooks::{BoxFuture, Datastore, DatastoreError, Protocol};
use dynomite::msg::{Msg, MsgType};
#[derive(Default, Clone)]
struct InMemoryDatastore;
impl Datastore for InMemoryDatastore {
    fn protocol(&self) -> Protocol { Protocol::Custom }
    fn dispatch(&self, req: Msg) -> BoxFuture<'_, Result<Msg, DatastoreError>> {
        Box::pin(async move {
            let mut rsp = Msg::new(req.id(), MsgType::RspRedisStatus, false);
            rsp.set_parent_id(req.id());
            Ok(rsp)
        })
    }
}
tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(async {
let store = InMemoryDatastore::default();
let handle = Server::start_with(
    ServerBuilder::new("dyn_o_mite")
        .listen("127.0.0.1:0".parse().unwrap())
        .dyn_listen("127.0.0.1:0".parse().unwrap())
        .datastore(Box::new(store)),
).await.unwrap();
handle.shutdown().await.unwrap();
});
}

Drive traffic with ServerHandle::inject_request (in-process fast path) or via the bound listen address (cross-process). crates/dynomite/tests/embed_api.rs shows the request-shape round-trip end to end, and embedded_cluster3.rs shows the same custom datastore behind three nodes.

Custom Transport: swap TCP for QUIC or anything else

Dynomite's Transport trait is the per-connection abstraction: any AsyncRead + AsyncWrite + Send + Unpin byte stream tagged with a ConnRole is a valid transport. The two ready-made implementations are TcpTransport (always available) and the QUIC variant (gated behind the quic feature).

What ships today, and where the boundary is:

• The trait shape is stable. crates/dynomite/examples/embedded_custom_transport_sketch.rs shows how to wrap a tokio::io::DuplexStream, a tokio_rustls::TlsStream, or a Unix domain socket into a Transport impl. The wrapper carries a ConnRole tag and reports a synthetic peer_addr.
• Plugging a custom listener (a factory that yields these transports as connections arrive) into ServerBuilder is tracked as a follow-up: the builder does not yet expose a transport_listener setter. The embedded server serves the client plane over TCP on its bound listen: socket today (parse -> dispatcher -> the configured Datastore hook), and ServerHandle::inject_request drives in-process traffic; custom transports beyond TCP are available by driving Proxy / QuicProxy directly, as the dynomited binary does. Cross-process peer-plane traffic is served by dynomited.

In short: write your Transport impl today (the shape is permanent); wire a custom listener through Proxy / QuicProxy until a ServerBuilder transport_listener setter lands. The embedded sketch double-checks the API contract for you.

Subscribing to cluster events

The engine publishes cluster-wide events on two complementary buses:

1. ServerHandle::events() -> Arc<EventManager> - the structured ClusterEvent broadcast (PeerUp, PeerDown, GossipRoundComplete, AaeExchangeStarted, RingChanged, ...). Mirrors the C reference's diagnostic logging in typed, matchable form. Use this for application-facing observability.
2. ServerHandle::subscribe_events() -> EventStream - the lower-level ServerEvent broadcast (ConnectionAccepted, ConnectionClosed, ConfigReloaded, Lagged, ...). Use this for connection-level tooling.

The two streams are independent; subscribe to whichever (or both) suits your integration. Lagging consumers receive a ClusterEvent-side lag (handled by your match arm via the #[non_exhaustive] wildcard) or a ServerEvent::Lagged payload, respectively, so a slow consumer never silently drops events.

#![allow(unused)]
fn main() {
use dynomite::embed::{Server, ServerBuilder};
use dynomite::events::ClusterEvent;
tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(async {
let handle = Server::start_with(
    ServerBuilder::new("p")
        .listen("127.0.0.1:0".parse().unwrap())
        .dyn_listen("127.0.0.1:0".parse().unwrap()),
).await.unwrap();

let events = handle.events();
let mut sub = events.subscribe();
tokio::spawn(async move {
    while let Ok(evt) = sub.recv().await {
        match evt {
            ClusterEvent::PeerUp { peer_id, dc, .. } => {
                eprintln!("peer {peer_id} up in {dc}");
            }
            ClusterEvent::PeerDown { peer_id, .. } => {
                eprintln!("peer {peer_id} down");
            }
            _ => {}
        }
    }
});
handle.shutdown().await.unwrap();
});
}

ClusterEvent is #[non_exhaustive]; your match must include a wildcard arm so future variants stay non-breaking. The same rule applies to ServerEvent.

Reading metrics from inside the embedder

Dynomite exposes its metrics surface through three mechanisms, each suited to a different integration shape:

1. Pull, lock-free: ServerHandle::stats_handle() -> Arc<Stats> returns a clone-cheap handle to the live aggregator. Read it whenever you want; the snapshot it returns is a value type that is safe to hold across awaits. Use this for pull-model exporters (Prometheus scrape handlers, OpenTelemetry pull readers) that want to read current counters without going through a periodic flush.

2. Pull, snapshot-only: ServerHandle::stats() -> Snapshot is the simpler shape - it returns the latest snapshot the stats aggregator computed. Cheaper to call than stats_handle().snapshot() because the runtime caches the most recent snapshot under a lock; identical content, lower contention.

3. Push: implement MetricsSink and plug it via ServerBuilder::metrics_sink. The runtime calls MetricsSink::emit(&snapshot) on the cadence the trait reports through flush_interval. Use this to forward the snapshot to OTLP, statsd, a custom dashboard adapter, or any other push-based collector.

#![allow(unused)]
fn main() {
use std::sync::Arc;
use dynomite::embed::{Server, ServerBuilder};
use dynomite::stats::Stats;
tokio::runtime::Builder::new_current_thread().enable_all().build().unwrap().block_on(async {
let handle = Server::start_with(
    ServerBuilder::new("p")
        .listen("127.0.0.1:0".parse().unwrap())
        .dyn_listen("127.0.0.1:0".parse().unwrap()),
).await.unwrap();

// Pull model.
let stats: Arc<Stats> = handle.stats_handle();
let snap = stats.snapshot();
println!("pool: {}", snap.pool.name);

// Or use the cached snapshot accessor.
let snap = handle.stats();
println!("pool again: {}", snap.pool.name);
handle.shutdown().await.unwrap();
});
}

The describe_stats accessor on the handle returns the manifest of every metric the engine emits, which is what a typical exporter wires into its registry at startup.

Graceful shutdown patterns

Two sanctioned shapes integrate Dynomite with your application's own lifecycle:

Pattern A: app-driven shutdown

Your application owns the shutdown signal (a ctrl_c, a kill switch from your supervisor, an internal "drain" command). Park the embedder on ServerHandle::join and trigger the cancel from a side task that watches the signal. join returns when the engine's background tasks finish.

use dynomite::embed::{Server, ServerBuilder};

#[tokio::main(flavor = "multi_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let handle = Server::start_with(
        ServerBuilder::new("dyn_o_mite")
            .listen("127.0.0.1:0".parse()?)
            .dyn_listen("127.0.0.1:0".parse()?),
    ).await?;

    let shutdown = handle.clone();
    tokio::spawn(async move {
        let _ = tokio::signal::ctrl_c().await;
        let _ = shutdown.shutdown().await;
    });

    handle.join().await;
    Ok(())
}

Pattern B: timeout-bounded shutdown

In the daemon shape, dynomited puts a wall-clock budget on shutdown so a stuck task does not block the process. The same pattern works in an embedder:

#![allow(unused)]
fn main() {
use std::time::Duration;
use dynomite::embed::{Server, ServerBuilder};
tokio::runtime::Builder::new_multi_thread().enable_all().build().unwrap().block_on(async {
let handle = Server::start_with(
    ServerBuilder::new("p")
        .listen("127.0.0.1:0".parse().unwrap())
        .dyn_listen("127.0.0.1:0".parse().unwrap()),
).await.unwrap();

// Fire shutdown then wait at most 5 seconds for the join set.
let h = handle.clone();
tokio::spawn(async move { let _ = h.shutdown().await; });
let _ = tokio::time::timeout(Duration::from_secs(5), handle.join()).await;
});
}

Two important guarantees from the embed API:

• ServerHandle::shutdown is idempotent. Calling it twice is safe; the second call is a no-op and returns immediately.
• ServerHandle::join after a shutdown returns immediately because the task set is drained on the first shutdown.

Production readiness

Dynomite as a library has been exercised under multi-host chaos across all four supported backends:

• TCP + Redis;
• TCP + Memcache;
• TCP + Riak (via dyniak);
• peer-plane TLS variants of each.

The committed reports under dist/chaos-reports/v0.1.0/ record seven independent multi-host chaos passes against four EC2 hosts at a time, exercising:

• gossip convergence under partition / merge cycles;
• per-DC quorum read repair under packet loss / corruption / reorder;
• hinted handoff drain during a sustained peer outage;
• anti-entropy reconciliation across rolling restarts;
• peer-plane TLS handshake under clock skew.

Every committed pass shows zero invariant violations, the post-test sweep confirms the host is left in its pre-test state, and the report itself is the production sign-off for the matching backend / transport pair. Embedders that ship a distinct backend should run the same harness against their own deployment shape; the shape lives in crates/dynomite/tests/stage_16_chaos.rs and is documented in Operations / Chaos test.

The library surface itself is covered by crates/dynomite/tests/embed_api.rs (the API smoke tests this cookbook references) and the broader crates/dynomite/tests/stage_13_embed.rs integration suite. Both suites run in CI on every push.

Introduction to Dyniak

This part of the manual is a getting-started guide and reference for Dyniak, the Riak-compatible protocol layer. It is written to be read front to back the first time and dipped into by section afterwards. If you have run a Riak cluster before, most of what follows will feel familiar: buckets and keys, per-request quorums, siblings, secondary indexes, MapReduce, CRDTs, and active anti-entropy are all here. What is new is where they run -- on the Dynomite substrate described in The Dynomite Engine architecture -- and a small set of capabilities Riak never shipped, chief among them cross-node XA transactions.

What Dyniak is

Dyniak is one of three protocol layers Dynomite can present to clients. The other two -- Redis / Valkey RESP and the Memcache binary and ASCII protocols -- are covered under Protocols. Dyniak is the Riak-shaped one:

Wire compatible

Riak's PBC (Protocol Buffers Client) binary protocol on a TCP listener, plus an HTTP gateway with Riak's /buckets/<bucket>/keys/<key> route shape. The same Riak client libraries connect unmodified.

Object model

Buckets, keys, values with content types, per-object secondary indexes, links, and causal context. See Buckets, Keys, and Objects.

Convergent data types

The same six CRDTs Riak shipped -- Counter, Set, Register, Flag, Map, and HyperLogLog -- with sibling-aware merge. See Convergent Data Types.

Transactions

Cross-node multi-key atomic updates over two-phase commit, plus a non-blocking read-atomic path. This is beyond Riak. See Distributed Transactions.

Query

Secondary-index (2i) queries, MapReduce with optional WebAssembly phases, link walking, and durable full-text / vector / regex search. See Secondary Indexes and MapReduce and Full-Text, Vector, and Regex Search.

Repair

Merkle-tree active anti-entropy, read repair, hinted handoff, and an optional divergence-proportional Merkle-Search-Tree reconcile. See Anti-Entropy and Repair.

The Riak-compatibility promise, precisely

Dyniak aims to occupy the socket a Riak cluster used to occupy so that client applications keep working. That promise has an honest boundary, and it is worth stating up front so you are not surprised later.

What is compatible:

• The PBC operation set (Ping, ServerInfo, Get, Put, Del, GetBucket, SetBucket, ListBuckets, ListKeys, Index, MapRed) and the HTTP route shapes.
• Per-request quorum semantics: R, W, PR, PW, DW, RW.
• Sibling-aware conflict resolution: racing writes produce siblings; a read surfaces them and the client resolves.
• Bucket properties (n_val, allow_mult, last_write_wins, and the rest) declared, not auto-created, per Riak semantics.
• CRDT merge semantics for all six data types.

Where the byte shape differs:

What is deliberately out of scope:

• Strong-consistency mode (Riak's riak_ensemble). Dyniak is honest about being eventually consistent. If you need linearizable single-object reads across the cluster, use a different store.
• Cross-datacenter realtime replication (Riak's riak_repl). The substrate -- gossip, ring ownership, anti-entropy -- is in place; the realtime queue is a follow-up.

The Basho and Riak heritage

Dyniak exists because of two decades of work by the engineers at Basho and the broader Riak open-source community. Riak was the canonical real-world implementation of the Amazon Dynamo paper (DeCandia et al., SOSP 2007): a masterless, ring-distributed, eventually-consistent key-value store with configurable per-request quorums, vnode-based partitioning, hinted handoff, read repair, active anti-entropy, sloppy quorums, secondary indexes, CRDTs, MapReduce, and a production-tested operations story. Many patterns now standard in distributed databases -- consistent-hash rings with virtual nodes, quorum tuples exposed at the API level, sibling-aware writes -- were first widely deployed in Riak.

The riak_core, riak_kv, riak_pipe, riak_search, and riak_dt Erlang/OTP code is the reference implementation for an entire category of systems. Dyniak is downstream of that thinking. Its explicit goal is to make it possible to drop a Dyniak-fronted Dynomite node into a slot a Riak cluster used to occupy and have client applications keep working. The full acknowledgement lives in the crates/dyniak/README.md and the project NOTICE.

Where Dyniak sits

Dynomite is the cluster substrate: consistent hashing, gossip, virtual nodes, quorum, hinted handoff, read repair, and anti-entropy. Dyniak is a thin protocol layer in front of it that owns only the Riak-specific pieces -- the wire codec, the request dispatch, and the storage bridge -- and reuses everything else.

flowchart TB
  subgraph clients [Riak clients]
    PB[PBC client library]
    HT[HTTP / curl / L7 proxy]
  end
  PB -->|4-byte len, msg code, protobuf| L1(dyniak PBC listener)
  HT -->|GET / PUT / POST / DELETE| L2(dyniak HTTP gateway)
  L1 --> D{dyniak dispatch}
  L2 --> D
  D -->|ring lookup| R((Dynomite ring))
  R -->|owns the key| N[(local Noxu env)]
  R -->|replica elsewhere| P1(peer node)
  R -->|replica elsewhere| P2(peer node)
  P1 --> N1[(Noxu)]
  P2 --> N2[(Noxu)]

A Riak client speaks PBC or HTTP to any node. Dyniak decodes the request, hands it to the Dynomite ring, and the ring serves it from the local Noxu environment or forwards it to the owning peers over the DNODE peer plane. The client never learns the topology.

Two facts about that picture matter operationally:

1. The backend is Noxu. Dyniak is served against an in-process, transactional Noxu environment -- an embedded B+tree engine. A Dyniak pool opens the environment at its configured noxu_path: and serves the Riak surface directly against it. It does not run a RESP client proxy and does not dial an external Redis backend; all traffic enters through the PBC and HTTP listeners.

2. The feature is opt-in. Dyniak is compiled into dynomited only when built with --features riak. Operators who do not run Riak workloads pay nothing for the extra dependencies and listeners.

When to use Dyniak

Reach for Dyniak when:

• You operate a Riak cluster today and want to migrate off the end-of-life Erlang stack while keeping the client API contract.
• You want Riak's data model -- buckets, siblings, CRDTs, 2i, MapReduce -- on a Rust runtime with built-in OpenTelemetry traces and Prometheus metrics.
• You need multi-key atomic updates that Riak's per-key model could not give you, without adopting a heavyweight consensus system.

Reach for a different Dynomite protocol layer, or a different store, when:

• Your clients already speak Redis or Memcache; use the RESP or Memcache layer instead (see Protocols).
• You need strong single-object consistency across the cluster; Dyniak is eventually consistent by design.

The road through these chapters

The chapters build on each other:

1. Getting Started with Dyniak -- build, config, first object, small cluster.
2. Buckets, Keys, and Objects -- the data model and conflict resolution.
3. Convergent Data Types -- CRDTs and why they beat last-write-wins.
4. Distributed Transactions -- XA / 2PC and the read-atomic path.
5. Links and Link Walking -- object graphs.
6. Secondary Indexes and MapReduce -- query and aggregation.
7. Full-Text, Vector, and Regex Search -- the FT.* surface.
8. Anti-Entropy and Repair -- how divergence is reconciled.

For the exact wire surface see Dyniak wire protocols; for operator concerns see Riak mode ops and Dyniak features ops.

Getting Started with Dyniak

This chapter takes you from an empty checkout to a running Dyniak node you can write objects to, and then to a small cluster. It assumes you have read Introduction to Dyniak for the shape of the system. Every shell command assumes you have run nix develop first, as described in the manual's conventions.

Step 1: build dynomited with the riak feature

Dyniak is compiled into the server only when you ask for it:

cargo build -p dynomited --features riak

Without the feature, dynomited builds and runs identically to a Redis or Memcache deployment; the riak: configuration block is still parsed and validated but is a no-op at run time. With the feature, the server gains the PBC and HTTP listeners and the data_store: dyniak backend.

Step 2: write a minimal Dyniak config

A Dyniak pool needs three things beyond a normal pool: data_store: dyniak (or the integer 2), a noxu_path: for the on-disk environment, and a riak: block naming the listener addresses. Save this as dyniak.yml:

dyn_o_mite:
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  tokens: '0'
  data_store: dyniak
  noxu_path: /tmp/dyniak-noxu
  servers:
  - 127.0.0.1:6379:1        # placeholder, ignored under dyniak
  riak:
    pbc_listen: 127.0.0.1:8087
    http_listen: 127.0.0.1:8098

A few things to understand about this config:

• data_store: dyniak selects the Noxu-backed Riak backend. The integer form 2 means the same thing; it sits alongside the historic 0 (valkey / redis) and 1 (memcache).
• noxu_path: is where the pool opens its in-process, transactional Noxu environment. It is created if absent. This directory is the durable home of every object, index, and transaction log.
• servers: is preserved for schema compatibility but is not contacted. A Dyniak pool does not run a RESP client proxy and does not dial an external backend, so the listen: address is not bound and the placeholder 127.0.0.1:6379:1 is never dialled. All client traffic enters through the PBC and HTTP listeners.
• riak.pbc_listen and riak.http_listen name the two client listeners. Either can be omitted to disable it; both can run side-by-side and share a single backend.

Validate the config before you start it:

dynomited -t -c dyniak.yml

The -t flag parses and validates without binding any sockets. It is the same validator CI runs, so a config that passes -t starts cleanly.

Step 3: start the node

dynomited -c dyniak.yml

The node opens the Noxu environment at /tmp/dyniak-noxu, binds the PBC listener on 127.0.0.1:8087 and the HTTP gateway on 127.0.0.1:8098, and waits for clients. Confirm it is alive with the HTTP liveness probe:

curl -s http://127.0.0.1:8098/ping
# OK

Step 4: write and read an object over HTTP

The HTTP gateway is the human-debuggable path. It uses Riak's route shape: an object lives at /buckets/<bucket>/keys/<key>. Store one:

curl -s -X PUT http://127.0.0.1:8098/buckets/users/keys/alice \
  -H 'Content-Type: application/json' \
  -d '{"value": "Alice Liddell", "content_type": "text/plain"}'

A successful store replies 204 No Content. Read it back:

curl -s http://127.0.0.1:8098/buckets/users/keys/alice
# {"value":"Alice Liddell","content_type":"text/plain", ...}

The gateway negotiates the response encoding from the Accept header: application/json, application/cbor, or application/x-protobuf. A value stored under one encoding is fetchable under any other, because the object is persisted in a canonical, codec-independent envelope.

Ask for CBOR instead:

curl -s http://127.0.0.1:8098/buckets/users/keys/alice \
  -H 'Accept: application/cbor' --output alice.cbor

Delete it:

curl -s -X DELETE http://127.0.0.1:8098/buckets/users/keys/alice
# 204 No Content

As in Riak, deleting an absent key is not an error -- the delete path replies 204 No Content whether or not the key existed.

Step 5: the same operation over PBC

Production clients use PBC -- the binary protocol -- because it is lower overhead and it is what the Riak client libraries speak by default. You almost never hand-assemble PBC frames; you point a Riak client library at the pbc_listen port and it does the framing for you. The framing itself is simple: a 4-byte big-endian length, a 1-byte message code, then a prost-encoded protobuf body.

Here is the same put/get using the Python riak client:

import riak

client = riak.RiakClient(host='127.0.0.1', pb_port=8087)
bucket = client.bucket('users')

obj = bucket.new('alice', data='Alice Liddell')
obj.content_type = 'text/plain'
obj.store()

fetched = bucket.get('alice')
print(fetched.data)     # Alice Liddell

The client library issues an RpbPutReq and an RpbGetReq over the socket; Dyniak decodes each frame, routes the key through the Dynomite ring, reads or writes the Noxu environment, and encodes the framed reply. The wire protocol chapter lists the full PBC message surface.

Step 6: choose your quorum

Riak's per-request quorum knobs are honoured. Ask for a read that requires two replicas to agree:

fetched = bucket.get('alice', r=2)

Or over HTTP with a query parameter:

curl -s 'http://127.0.0.1:8098/buckets/users/keys/alice?r=2'

The knobs are r, w, pr, pw, dw, and rw, with the same meaning they carry in Riak: how many replicas (or primary replicas) must respond before the request is considered done. On a single-node cluster every quorum is trivially satisfied; the knobs earn their keep once you have peers, which is the next step. The replication and consistency model underneath these knobs is described in Consistency.

Step 7: grow to a small cluster

A single node is a store, not a cluster. To get replication, add peers. Each node runs its own dynomited with its own noxu_path: and its own token, and they discover one another over gossip on the dyn_listen peer plane. A minimal three-node config differs from the single-node one in the tokens: value (each node owns a different slice of the ring) and the peer wiring.

Node A (dyniak-a.yml):

dyn_o_mite:
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  tokens: '0'
  data_store: dyniak
  noxu_path: /tmp/dyniak-a
  servers:
  - 127.0.0.1:6379:1
  riak:
    pbc_listen: 127.0.0.1:8087
    http_listen: 127.0.0.1:8098

Nodes B and C are identical except for their listener ports, their noxu_path:, and their tokens: (for example 1431655765 and 2863311530 to split a three-way ring evenly). The peers find each other through the seed list and gossip; the mechanics of standing up the ring are the same as any Dynomite cluster and are covered in Your First Cluster.

Once the ring is formed, a write to any node with w=2 is replicated to the owning peers before the write is acknowledged, and a read with r=2 waits for two replicas to agree. The client still connects to any one node and never learns the topology.

Where to next

You now have a running Dyniak node (or a small cluster) and can read and write objects over both wire protocols. From here:

• Buckets, Keys, and Objects explains the data model, content types, siblings, and bucket properties in depth.
• Convergent Data Types shows how to store counters, sets, and maps that merge cleanly under concurrent writes.
• Distributed Transactions covers multi-key atomic updates.
• For operator concerns -- listener tuning, AAE cadence, 2i storage layout -- see Riak mode ops.

Buckets, Keys, and Objects

The Dyniak data model is Riak's data model: a flat namespace of buckets, each holding objects addressed by a key. An object is a value plus metadata -- a content type, secondary-index entries, links, and a causal-context blob. This chapter walks the model from the outside in, then covers the two things that trip up newcomers to a Dynamo-style store: siblings and conflict resolution.

If you have not yet stood up a node, do that first in Getting Started with Dyniak; the examples here assume a running gateway on 127.0.0.1:8098 (HTTP) and 127.0.0.1:8087 (PBC).

Buckets and keys

A bucket is a named container for objects. Buckets are cheap: there is no create step for a plain bucket -- you write an object into a bucket and the bucket exists. A key is an arbitrary byte string that names an object inside a bucket. The pair (bucket, key) is the object's full address.

flowchart LR
  subgraph b1 [bucket: users]
    k1[key: alice] --> v1[value + metadata]
    k2[key: bob] --> v2[value + metadata]
  end
  subgraph b2 [bucket: sessions]
    k3[key: s-9f3a] --> v3[value + metadata]
  end

A bucket is a flat namespace of keys; each key addresses one object, which is a value plus its metadata. Buckets do not nest.

Over HTTP the address is a URL path:

# address:  bucket "users", key "alice"
curl http://127.0.0.1:8098/buckets/users/keys/alice

Over PBC the address is the bucket and key fields of the request message. The routing layer hashes the pair (or just the bucket, if the bucket's chash_keyfun property says so -- see below) to choose the ring position that owns the object.

The object: value plus metadata

An object carries more than its value. The HTTP envelope makes each piece explicit:

{
  "value": "Alice Liddell",
  "content_type": "text/plain",
  "indexes": [
    {"name": "age_int", "value": "42"},
    {"name": "city_bin", "value": "seattle"}
  ],
  "links": [
    {"bucket": "users", "key": "bob", "tag": "friend"}
  ]
}

The pieces:

value

The opaque payload. Dyniak stores the bytes verbatim; it does not interpret them except where a query feature (2i, search, MapReduce) asks it to.

content_type

A MIME type describing the value. It is round-tripped, not enforced: Dyniak stores whatever you send and returns it. Set it so clients (and MapReduce phases) know how to read the value.

indexes

Secondary-index entries. Each is a (name, value) pair; the name suffix _int or _bin selects an integer or binary index. Covered in Secondary Indexes and MapReduce.

links

Typed pointers to other objects, each a (bucket, key, tag) triple. Covered in Links and Link Walking.

Content types over the wire

Over HTTP, the value's content type is negotiated two ways. The gateway's transport encoding -- how the whole envelope is serialized -- is chosen from the Accept header (application/json, application/cbor, or application/x-protobuf). The value's own content type is a field inside the envelope. Because the envelope is persisted in a canonical, codec-independent form, a value written as JSON is fetchable as CBOR and vice versa:

# write as JSON
curl -X PUT http://127.0.0.1:8098/buckets/users/keys/alice \
  -H 'Content-Type: application/json' \
  -d '{"value": "Alice", "content_type": "text/plain"}'

# read the same object as CBOR
curl http://127.0.0.1:8098/buckets/users/keys/alice \
  -H 'Accept: application/cbor' --output alice.cbor

Over PBC the value and its content type ride in the RpbContent message; indexes are RpbPair entries and links are RpbLink entries inside the same RpbContent.

Bucket properties

A bucket's behaviour is governed by its properties. Fetch them:

curl -s http://127.0.0.1:8098/buckets/users/props

{
  "props": {
    "name": "users",
    "n_val": 3,
    "allow_mult": false,
    "last_write_wins": false,
    "r": "quorum", "w": "quorum",
    "pr": 0, "pw": 0,
    "dw": "quorum", "rw": "quorum",
    "basic_quorum": false,
    "notfound_ok": true
  }
}

The properties that matter most day to day:

• n_val -- the replication factor: how many copies of each object the ring keeps. The default is 3.
• r / w -- the default read and write quorums for the bucket, in the absence of a per-request override. "quorum" means a majority of n_val.
• pr / pw -- primary-replica read and write quorums: how many of the responding replicas must be primary owners (not fallback nodes holding a hinted-handoff copy).
• dw -- durable-write quorum: how many replicas must have committed the write durably.
• allow_mult -- whether the bucket keeps siblings on a conflict (below).
• last_write_wins -- whether conflicts are resolved by timestamp rather than by keeping siblings.

Set properties with PUT:

curl -X PUT http://127.0.0.1:8098/buckets/users/props \
  -H 'Content-Type: application/json' \
  -d '{"props": {"n_val": 5, "allow_mult": true}}'

Causal context

Every object read returns a small context blob that encodes the object's causal history. The client's job is simple: read the blob, hold it, and echo it back on the next write of the same key. The server uses the echoed context to decide whether the new write supersedes what is stored (a normal update) or diverged from it (a conflict).

sequenceDiagram
  participant C as client
  participant S as dyniak node
  C->>S: GET users/alice
  S-->>C: value + context v1
  Note over C: client edits the value
  C->>S: PUT users/alice (value2, context v1)
  Note over S: context v1 matches stored history
  S-->>C: 204 (clean update)

The read-modify-write cycle. The client echoes the context it read; the server uses it to distinguish a clean update from a conflict. A client that skips the context risks creating an unnecessary sibling.

Dyniak encodes the context as an Interval Tree Clock (ITC) rather than Riak's dotted version vector. The two answer the same question -- did these two writes see each other, or did they diverge? -- but ITC scales with the currently-live actor population rather than with every actor that ever participated, which suits Dynomite's dynamic-membership cluster model. Retired nodes leave no residual cost in the clock.

Siblings and conflict resolution

Here is the heart of the Dynamo model. Because Dyniak is masterless and eventually consistent, two clients can write the same key at the same time without either seeing the other. What happens next depends on the bucket's properties.

With allow_mult: true (recommended)

The two concurrent writes are both kept as siblings. A later read of the key surfaces both values and the client resolves them:

flowchart TB
  W1[client 1 PUT value=A ctx v0] --> N1(node)
  W2[client 2 PUT value=B ctx v0] --> N1
  N1 --> S{concurrent?<br/>both saw v0}
  S -->|yes| SIB[store both:<br/>siblings A and B]
  SIB --> R[next GET returns A and B]
  R --> RES[client merges to C, writes back with the combined context]
  RES --> DONE[single value C]

Concurrent writes that both descend from the same context become siblings. The next reader sees both, resolves them into one value, and writes the resolution back. Siblings are a feature: they never silently drop a write.

Over HTTP a sibling read is signalled by a 300 Multiple Choices status; the client fetches each sibling and resolves. This is the safe default because it never loses a write.

With last_write_wins: true

The store keeps only the write with the higher timestamp and silently discards the other. This is simpler for clients -- no sibling handling -- but it can lose a concurrent write. Use it only when the value is disposable or the write rate makes conflicts vanishingly rare.

A worked read-modify-write

Putting the pieces together, the canonical safe update loop is:

import riak

client = riak.RiakClient(host='127.0.0.1', pb_port=8087)
bucket = client.bucket('users')
bucket.set_property('allow_mult', True)

# fetch (carries the causal context)
obj = bucket.get('alice')

if len(obj.siblings) > 1:
    # resolve the conflict: application-specific merge
    merged = resolve(obj.siblings)
    obj.data = merged
else:
    obj.data = update(obj.data)

# store echoes the context automatically through the client library
obj.store()

The client library carries the context for you; the only judgement call is resolve(), and even that disappears if you model the value as a CRDT. That is the subject of the next chapter.

Where to next

• Convergent Data Types -- make conflict resolution automatic.
• Links and Link Walking -- connect objects into a graph.
• Secondary Indexes and MapReduce -- query objects by index value.
• Dyniak wire protocols -- the exact PBC and HTTP surface.

Convergent Data Types

The previous chapter left you with a choice on every concurrent write: keep siblings and resolve them yourself, or let last-write-wins silently drop one. Convergent data types -- CRDTs -- give you a third option that is almost always the right one for structured values. A CRDT merges concurrent writes automatically and correctly, with no sibling handling and no lost write, because the merge is defined by the data type's algebra rather than by a timestamp race or a human callback.

This chapter explains why CRDTs work, walks the six types Dyniak ships, and shows the API for each.

Why CRDTs

A CRDT is a data type whose merge operation is associative, commutative, and idempotent. Those three laws are what make it safe in an eventually-consistent, masterless store:

Commutative

merge(a, b) == merge(b, a). The order in which two replicas exchange state does not matter.

Associative

merge(merge(a, b), c) == merge(a, merge(b, c)). The order in which three or more replicas' states combine does not matter.

Idempotent

merge(a, a) == a. Receiving the same state twice -- a retransmit, a re-read during anti-entropy -- changes nothing.

Together these give strong eventual consistency: any two replicas that have received the same set of updates, in any order, with any duplicates, converge to exactly the same value. No coordination, no locks, no consensus round. A replica can accept a write while partitioned from the rest of the cluster, and when the partition heals, the merge reconciles the divergence deterministically.

Every type below satisfies all three laws; property tests in the crate (crates/dyniak/tests/datatypes_properties.rs) exercise associativity, commutativity, and idempotence on randomly generated states.

Actors

CRDTs track who made each change so the merge can tell concurrent edits apart. Riak keyed this by an Erlang vnode_id tuple; Dyniak uses an ActorId, the (datacenter, peer) pair the Dynomite substrate already publishes through gossip. The pair is stable across gossip rounds and totally ordered, which is exactly what an OR-Set tag generator and a register tiebreaker need.

#![allow(unused)]
fn main() {
use dyniak::datatypes::ActorId;

let a = ActorId::new("dc1", "peer-a");
let b = ActorId::new("dc1", "peer-b");
assert!(a < b);   // lexicographic on (dc, peer)
}

The six types

Dyniak ships the same six data types Riak did. Four are primitive; the map composes them; the HyperLogLog is a cardinality estimator.

flowchart TB
  Counter[Counter<br/>PN-counter]
  Set[Set<br/>observed-remove]
  Register[Register<br/>last-write-wins]
  Flag[Flag<br/>enable-wins]
  Map[Map<br/>observed-remove, recursive]
  HLL[HyperLogLog<br/>cardinality estimate]
  Map --> Counter
  Map --> Set
  Map --> Register
  Map --> Flag
  Map --> Map

The six Dyniak CRDTs. A Map field may hold any primitive type or, recursively, another Map -- the same composition Riak's riak_dt_map offers.

Counter

A positive-negative counter (PnCounter). Internally it is a pair of per-actor grow-only counts; the value is sum(positive) - sum(negative), and merge is the element-wise maximum of each actor's count. That maximum is what makes concurrent increments on different replicas both survive.

#![allow(unused)]
fn main() {
use dyniak::datatypes::{ActorId, Crdt, PnCounter};

let a = ActorId::new("dc1", "a");
let b = ActorId::new("dc2", "b");

// Two replicas increment independently while partitioned.
let mut left = PnCounter::new();
left.increment(&a, 5);

let mut right = PnCounter::new();
right.increment(&b, 3);
right.decrement(&b, 1);

// Partition heals; the two states merge.
left.merge(&right);
assert_eq!(left.value(), 5 + 3 - 1);   // 7 -- neither increment lost
}

Over the wire this is a DtUpdateReq carrying a counter op. A common use is a page-view or like counter that many nodes bump concurrently.

Set

An observed-remove set (OrSet) of arbitrary byte-string elements. Each add mints a fresh per-actor tag; a remove tombstones only the tags it has observed. The rule is "add wins": a concurrent add and remove of the same element resolves to present, because the add carries a tag the remove never saw.

#![allow(unused)]
fn main() {
use dyniak::datatypes::{ActorId, Crdt, OrSet};

let a = ActorId::new("dc1", "a");
let b = ActorId::new("dc2", "b");

let mut left = OrSet::new();
left.add(&a, "red");
left.add(&a, "green");

// Concurrently, replica b adds "blue" and removes "green"
// (b had observed green before removing it).
let mut right = left.clone();
right.add(&b, "blue");
right.remove(b"green");

left.add(&a, "green");   // a re-adds green concurrently -> add wins
left.merge(&right);

assert!(left.contains(b"red"));
assert!(left.contains(b"blue"));
assert!(left.contains(b"green"));   // survived: the re-add's tag was unseen
}

Use a Set for tag lists, group memberships, or any collection where concurrent add and remove must both be respected.

Register

A last-write-wins register (LwwRegister). State is (value, timestamp, actor); merge picks the higher timestamp, breaking ties by the higher actor id. Unlike bucket-level last-write-wins, a register is a deliberate, typed choice: you are saying "for this field, newest wins" and you get a deterministic tiebreak instead of a silent sibling. Registers are most useful as fields inside a Map.

Flag

An enable-wins boolean (EwFlag) -- an OR-Set restricted to a singleton domain. Concurrent enable and disable resolves to enabled. Use it for a feature toggle or a "seen" bit where turning it on should not be lost to a concurrent turn-off.

Map

An observed-remove map (Map), ORSWOT-style. It is keyed by a FieldKey -- a (name, type) pair -- and each value is one of the four primitives or, recursively, another Map. Two fields with the same name but different types are distinct, which is how a map namespaces its fields. Field presence follows the same add-wins rule as the Set:

#![allow(unused)]
fn main() {
use dyniak::datatypes::map::FieldType;
}

The field types a map may carry are Counter, OrSet, LwwRegister, EwFlag, and NestedMap. A worked example: a user profile as a map with a visits counter, an interests set, and a name register, all merging independently.

HyperLogLog

A probabilistic cardinality estimator (HyperLogLog). It answers "how many distinct elements have been added?" with a small, fixed-size state and a bounded error, and it merges by taking the per-register maximum -- the same lattice join pattern as the counter. Use it to count unique visitors or unique keys touched without storing every element.

How convergence plays out on the ring

The laws are abstract; here is what they buy you operationally. Two replicas of the same CRDT key diverge under a partition, then reconcile:

sequenceDiagram
  participant R1 as replica 1
  participant R2 as replica 2
  Note over R1,R2: network partition
  R1->>R1: increment counter (+5)
  R2->>R2: increment counter (+3)
  Note over R1,R2: partition heals; anti-entropy runs
  R1->>R2: ship state {r1: +5}
  R2->>R1: ship state {r2: +3}
  Note over R1: merge -> {r1:+5, r2:+3} = 8
  Note over R2: merge -> {r1:+5, r2:+3} = 8

Both replicas accepted a write during the partition; the anti-entropy exchange ships state both ways; the merge is order-independent and idempotent, so both converge to 8. No write was lost and no coordination was needed.

The exchange that carries CRDT state between replicas is the same anti-entropy machinery described in Anti-Entropy and Repair; read repair on a quorum read also merges divergent replicas before answering.

Choosing a type

A running total

Counter. Increments and decrements from any replica all count.

A collection with add and remove

Set. Concurrent add wins over remove.

A single "newest wins" value

Register, usually as a Map field.

A boolean where "on" should stick

Flag (enable-wins).

A structured record

Map, composing the above; nest maps for sub-records.

An approximate distinct count

HyperLogLog.

An arbitrary opaque blob with a custom merge

Not a CRDT -- use a plain object with siblings and resolve in the application. See siblings.

Where to next

• Distributed Transactions -- when you need atomic updates across several keys at once, which no single CRDT gives you.
• Anti-Entropy and Repair -- how CRDT state actually flows between replicas.
• Buckets, Keys, and Objects -- the plain-object model CRDTs sit alongside.

Distributed Transactions (XA / 2PC)

CRDTs make a single key converge without coordination. But some updates must span several keys and be all-or-nothing: move a balance from one account object to another, or write an object and its index entry together so a reader never sees one without the other. Riak's per-key model had no answer for that -- a multi-key atomic write in Riak would have needed a consensus layer it did not ship. Dyniak adds one, built on the transactional Noxu engine underneath it.

This chapter covers the two transaction paths Dyniak offers: the heavyweight two-phase commit (XA) path for full atomicity across nodes, and the lighter read-atomic (RAMP) path for the common "see all of a batch or none of it" case without blocking.

The two layers

A multi-key transaction stacks in two layers, and which one runs depends on where the keys live:

flowchart TB
  B[TxnBatch: put a, put b, delete c] --> R{keys owned by<br/>one node or many?}
  R -->|one node| L1[Layer 1:<br/>single-environment<br/>one engine transaction]
  R -->|many nodes| L2[Layer 2:<br/>cross-node XA<br/>two-phase commit]
  L1 --> C1[commit or roll back atomically]
  L2 --> C2[prepare all branches, then commit all]

A batch whose keys all land on one node commits in a single Noxu engine transaction. A batch that spans nodes is coordinated with X/Open XA two-phase commit over the DNODE peer plane.

Layer 1: single-environment

Every op in the batch routes to one node's Noxu environment and commits inside one engine transaction. Simple, fast, fully atomic.

Layer 2: cross-node XA

The batch touches keys owned by different primary nodes. Each node prepares its branch; the coordinator commits every branch only once every prepare has voted to commit. This is X/Open two-phase commit.

The client does not choose a layer. It submits a batch; the coordinator partitions the ops by owning node and picks the cheapest correct path.

Submitting a transaction

Over HTTP, a transaction is a JSON batch to POST /transactions (cluster-wide) or POST /buckets/{bucket}/transactions (bucket-scoped, where every op must target the URL's bucket):

curl -s -X POST http://127.0.0.1:8098/transactions \
  -H 'Content-Type: application/json' \
  -d '{
    "operations": [
      {"op": "put", "bucket": "accounts", "key": "alice",
       "value": "balance=90",
       "indexes": [{"name": "owner_bin", "value": "alice"}]},
      {"op": "put", "bucket": "accounts", "key": "bob",
       "value": "balance=110"},
      {"op": "delete", "bucket": "pending", "key": "xfer-7"}
    ]
  }'

A committed batch replies 200 OK:

{"result": "committed", "operations": 3}

The ops are replayed in order inside the transaction. A put carries an optional list of 2i entries, fanned into the secondary-index layer as part of the same atomic unit, so the object and its index land together or not at all.

The abort path

A batch with "abort": true applies every op inside the transaction and then deliberately rolls back, leaving the keyspace untouched. It exists so clients and tests can exercise the rollback path deterministically:

curl -s -X POST http://127.0.0.1:8098/transactions \
  -H 'Content-Type: application/json' \
  -d '{"abort": true, "operations": [
        {"op": "put", "bucket": "b", "key": "k", "value": "v"}]}'

{"result": "aborted", "reason": "client requested abort"}

A batch that the engine rolls back because of a serialization conflict replies 409 Conflict; the client may retry. A datastore that is not transactional (any backend other than the Noxu-backed one) replies 501 Not Implemented.

The two-phase commit protocol

When a batch spans nodes, the coordinator runs X/Open XA. Each node is one resource manager backed by its own Noxu environment; the coordinator is the transaction manager.

sequenceDiagram
  participant C as coordinator
  participant E as branch east
  participant W as branch west
  Note over C: partition ops by owning node
  C->>E: xa_start, apply east ops, xa_end
  C->>W: xa_start, apply west ops, xa_end
  C->>E: xa_prepare
  E-->>C: vote OK
  C->>W: xa_prepare
  W-->>C: vote OK
  Note over C: every prepare voted OK -> commit
  C->>E: xa_commit
  C->>W: xa_commit
  E-->>C: done
  W-->>C: done

Two-phase commit across two branches. Work is applied and each branch is prepared; only after every branch votes to commit does the coordinator issue the commits. A single "no" vote (or a force-abort) rolls back every prepared branch.

Three details make this robust against the network failure modes a single-process commit never sees:

• Read-only optimization. A branch that performed no writes votes read-only on prepare and skips the second phase entirely -- no commit round for a branch that changed nothing.
• Presumed abort. If a prepare fails or times out, the coordinator presumes abort and rolls back every branch already prepared. No branch is left holding a prepared-but-undecided transaction.
• Commit-in-doubt recovery. A branch that voted to commit is recorded in a durable in-doubt log. If the coordinator crashes after the commit decision but before every branch confirmed, a cold restart re-reads the log and forward-recovers each in-doubt branch, so a branch that voted to commit is never left dangling.

The local-branch machinery (XaParticipant, one per node) is shared verbatim between the in-process coordinator and the cross-node coordinator; only the transport differs. The cross-node coordinator drives the identical prepare-then-commit phases over the DNODE peer plane described in Dyniak wire protocols.

The read-atomic path (RAMP)

Full 2PC is the right tool when you need serializable atomicity, but it is heavy: it blocks, it coordinates, it commits in two rounds. For the very common case of "read several keys and see a consistent snapshot, or write several keys so no reader ever sees a partial batch," Dyniak offers a lighter path: RAMP-Fast (Bailis et al., SIGMOD 2014).

RAMP guarantees read-atomic isolation: a transaction sees all of another transaction's writes or none of them -- never a fractured read where you observe transaction T's write to key a but miss its write to key b. Crucially, a reader never blocks on a writer.

sequenceDiagram
  participant Wr as writer
  participant Rd as reader
  Note over Wr: pick one timestamp ts for the whole batch
  Wr->>Wr: PREPARE: write each key at ts (invisible),<br/>metadata = sibling keys
  Wr->>Wr: COMMIT: advance each key's visible pointer to ts
  Rd->>Rd: round 1: read latest-visible + metadata for all keys
  alt no fracture detected
    Note over Rd: snapshot is already atomic
  else metadata names a newer sibling
    Rd->>Rd: round 2: fetch exactly the missing versions by ts
  end
  Note over Rd: fracture-free snapshot

RAMP-Fast. Writes are two-phase but non-blocking: prepare writes invisible versions, commit advances the visible pointers. Reads take one round plus a conditional second round only when the first round detects a fracture. In the contention-free case the second round is skipped.

A RAMP write picks one monotonic timestamp for the whole batch, writes every key as an invisible versioned record carrying the set of sibling keys, then advances each key's visible pointer. A RAMP read fetches the latest visible version plus metadata for every key; if any version's metadata names a sibling the reader saw at an older version, round 2 fetches exactly those missing versions -- which prepare guarantees are present -- and the repaired snapshot is fracture-free.

Over HTTP:

# atomic multi-key write
curl -s -X POST http://127.0.0.1:8098/ramp/transactions \
  -H 'Content-Type: application/json' \
  -d '{"writes": [{"bucket": "b", "key": "a", "value": "1"},
                  {"bucket": "b", "key": "b", "value": "2"}]}'
# {"result":"committed","ts":<timestamp>,"keys":2}

# atomic multi-key read (fracture-free snapshot)
curl -s -X POST http://127.0.0.1:8098/ramp/read \
  -H 'Content-Type: application/json' \
  -d '{"keys": [{"bucket": "b", "key": "a"}, {"bucket": "b", "key": "b"}]}'
# {"snapshot": {"a": "1", "b": "2"}, "rounds": 1}

The response reports rounds (1 or 2) so you can see when the read hit contention.

XA versus RAMP: which to use

Use XA / 2PC when

You need full atomicity with the possibility of rollback on conflict -- move money, enforce an invariant across keys, write an object and its index as one unit that can fail cleanly.

Use RAMP when

You need a consistent multi-key snapshot or an all-or-nothing multi-key write, but you do not need to abort on conflict and you want readers never to block. Lower latency, availability-native.

Use a single CRDT when

The update is confined to one key. No transaction needed -- the merge is automatic. See Convergent Data Types.

Where to next

• Convergent Data Types -- the coordination-free single-key alternative.
• Anti-Entropy and Repair -- how the substrate reconciles divergence outside the transaction path.
• Dyniak features ops -- operator view of cross-node transactions.
• Consistency -- the underlying replication and quorum model.

Links and Link Walking

Riak let an object carry typed links to other objects, turning a flat key/value store into a navigable graph: a user links to their friends, a blog post links to its comments, an order links to its line items. Dyniak keeps that model. This short chapter covers how links are stored, how they ride on each wire protocol, and how you traverse them -- which, in Dyniak as in Riak, is done through MapReduce rather than a dedicated route.

What a link is

A link is a typed pointer from one object to another: a (bucket, key, tag) triple. The tag names the kind of relationship -- friend, author, parent -- so one object can link to many others with different meanings.

flowchart LR
  A[users/alice] -->|tag: friend| B[users/bob]
  A -->|tag: friend| C[users/carol]
  A -->|tag: author| P[posts/hello-world]
  P -->|tag: comment| K1[comments/c-1]
  P -->|tag: comment| K2[comments/c-2]

An object graph built from links. Each edge is a (bucket, key, tag) triple stored on the source object; the tag names the relationship so a single object can carry many kinds of link.

Links live on the source object. They are part of the object's metadata, alongside its content type and secondary indexes (see Buckets, Keys, and Objects). Adding, changing, or removing a link is just storing the object with a different link list.

Storing links

Over HTTP, links ride in the Link header, using the same grammar Riak used. Each link-value names the target resource and carries a riaktag parameter:

curl -X PUT http://127.0.0.1:8098/buckets/users/keys/alice \
  -H 'Content-Type: application/json' \
  -H 'Link: </buckets/users/keys/bob>; riaktag="friend", </buckets/users/keys/carol>; riaktag="friend"' \
  -d '{"value": "Alice Liddell"}'

The gateway accepts both the modern /buckets/<bucket>/keys/<key> resource form and the legacy /riak/<bucket>/<key> form, matches riaktag or tag case-insensitively, and honours several Link: header lines or several comma-separated values in one line. A link-value that lacks a tag, or whose resource is not an object path, is skipped rather than rejected -- the same lenient parse Riak used.

You can also carry links inside the JSON envelope directly:

{
  "value": "Alice Liddell",
  "links": [
    {"bucket": "users", "key": "bob",   "tag": "friend"},
    {"bucket": "posts", "key": "hello", "tag": "author"}
  ]
}

On read, the gateway re-emits the object's links as Link headers, plus a synthesized bucket-up link so a client can navigate from an object back to its bucket:

Link: </buckets/users>; rel="up"
Link: </buckets/users/keys/bob>; riaktag="friend"

Over PBC, links are RpbLink entries inside the object's RpbContent message -- the same slot the value and content type occupy. A Riak client library exposes them through its usual link API:

import riak

client = riak.RiakClient(host='127.0.0.1', pb_port=8087)
users = client.bucket('users')

alice = users.new('alice', data='Alice Liddell')
bob = users.new('bob', data='Bob')
alice.add_link(bob, tag='friend')
alice.store()

Walking links

Here is the design choice worth calling out: Dyniak has no dedicated link-walk route. Links are traversed by a MapReduce Link phase. This matches Riak, where link-walking was ultimately expressed as a pipeline phase, and it means link traversal composes with the rest of the MapReduce toolbox -- you can walk a link and then map over the resolved objects in one job.

A Link phase follows the links on its input objects, optionally filtered by bucket and tag, and emits the resolved target objects as its output. Chain it into a pipeline submitted to POST /mapred (HTTP) or RpbMapRedReq (PBC):

curl -s -X POST http://127.0.0.1:8098/mapred \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": [["users", "alice"]],
    "query": [
      {"link": {"bucket": "users", "tag": "friend"}},
      {"map":  {"fn_name": "map_object_value"}}
    ]
  }'

Read that pipeline as: start from users/alice, follow every friend link into the users bucket, then extract each resolved friend's value. The Link phase does the graph traversal; the Map phase does whatever you want with the objects it lands on.

flowchart LR
  I[input: users/alice] --> LP{Link phase<br/>bucket=users<br/>tag=friend}
  LP --> B[users/bob]
  LP --> C[users/carol]
  B --> MP{Map phase<br/>map_object_value}
  C --> MP
  MP --> O[bob value, carol value]

A two-phase link walk. The Link phase resolves Alice's friend links to the target objects; the Map phase projects each target's value. Because it is a MapReduce pipeline, you can add more phases -- another link hop, a reduce, a sort -- after it.

Filtering a walk

Both filters on a Link phase are optional:

bucket

Only follow links whose target is in this bucket. Omit to follow links into any bucket.

tag

Only follow links with this tag. Omit to follow links of every tag.

To follow every link Alice has, regardless of kind, drop both filters:

{"link": {}}

To walk two hops -- Alice's friends' authored posts -- chain two Link phases:

{
  "inputs": [["users", "alice"]],
  "query": [
    {"link": {"tag": "friend"}},
    {"link": {"bucket": "posts", "tag": "author"}},
    {"map":  {"fn_name": "map_object_value"}}
  ]
}

When to use links

Links are a good fit when the relationships are part of the object and you want to navigate them on demand. They are not an index: to find "all objects that link to Alice" you would need a secondary index or a MapReduce over the source bucket, because links are stored on the source and point outward. For query-by-attribute, reach for secondary indexes; for graph navigation from a known starting object, links are the tool.

Where to next

• Secondary Indexes and MapReduce -- the query and pipeline machinery that link-walking is built on.
• Buckets, Keys, and Objects -- where links live in the object model.
• Dyniak wire protocols -- the exact link representation on each wire.

Secondary Indexes and MapReduce

Objects are addressed by key, but you often need to find them by attribute: every user aged 42, every order placed in a date range. That is what secondary indexes (2i) are for. And once you can select a set of objects, you often want to aggregate them -- count, sum, sort, project. That is what MapReduce is for. The two compose: a MapReduce job can seed itself from a 2i query. This chapter covers both.

Secondary indexes (2i)

A secondary index attaches searchable (name, value) tags to an object at write time. The index name's suffix picks its type:

_int

An integer index. Supports equality and range queries over integer values.

_bin

A binary index. Supports equality and range queries over byte-string values.

Attaching index entries

Over HTTP the entries ride in the object envelope or in X-Riak-Index-* headers:

curl -X PUT http://127.0.0.1:8098/buckets/users/keys/alice \
  -H 'Content-Type: application/json' \
  -d '{
    "value": "Alice",
    "indexes": [
      {"name": "age_int", "value": "42"},
      {"name": "city_bin", "value": "seattle"}
    ]
  }'

Over PBC the entries are RpbPair items in the RpbPutReq.indexes field, one pair per entry, where the pair key names the index (with its _int or _bin suffix) and the pair value carries the value bytes. A Riak client library exposes this as add_index:

o = bucket.new('alice', data='profile')
o.add_index('age_int', '42')
o.add_index('city_bin', 'seattle')
o.store()

Querying an index

Two query types are supported, matching Riak:

# equality: every key whose age_int is exactly 42
hits = bucket.get_index('age_int', '42').results

# range: every key whose age_int is in [10, 50] inclusive
hits = bucket.get_index('age_int', '10', '50').results

Over PBC these are RpbIndexReq with qtype: 0 (equality) or qtype: 1 (range, inclusive bounds). The current response returns a single frame with done = true; streaming one frame per chunk is a tracked follow-up.

MapReduce

MapReduce runs a pipeline of phases over a set of input objects. Each phase transforms a stream of values into another stream; the phases are chained, and the last phase's output is the job result. The shape is Riak's "pipe of phases."

The job envelope

A job has two parts: inputs (the seed values) and query (the phase list). Submit it to POST /mapred (HTTP) or RpbMapRedReq (PBC):

curl -s -X POST http://127.0.0.1:8098/mapred \
  -H 'Content-Type: application/json' \
  -d '{
    "inputs": [["orders", "o-1"], ["orders", "o-2"], ["orders", "o-3"]],
    "query": [
      {"map":    {"fn_name": "map_object_value"}},
      {"reduce": {"fn_name": "reduce_sum", "keep": true}}
    ]
  }'

Inputs can be an explicit list of (bucket, key) pairs (above), an inline list of values, or a bucket name (all keys in the bucket, enumerated by the executor).

Built-in phases

The built-in registry ships map and reduce functions named by Riak's convention (map_*, reduce_*):

map_object_value

Extract the object's value field.

map_object_value_list

Emit each element if the value is a JSON array, else the scalar.

map_extract_field

Project a named field out of each object.

map_identity

Pass inputs through unchanged.

reduce_count

Count the inputs.

reduce_sum

Sum numeric inputs.

reduce_sort

Sort the inputs.

reduce_set_union

Union the inputs into a distinct set.

reduce_identity

Pass inputs through unchanged.

Two more phase kinds round out the pipeline:

• Link -- follow object links, optionally filtered by bucket and tag. This is how link-walking is expressed; see Links and Link Walking.
• WasmModule -- run a registered WebAssembly module as a map or reduce phase. Covered below.

The data flow

flowchart LR
  I[inputs:<br/>bucket/key pairs<br/>or a bucket] --> P1{Map phase}
  P1 -->|values| P2{Reduce phase}
  P2 -->|aggregated| O[result envelope]
  subgraph pipe [pipeline: one task per phase, mpsc between phases]
    P1
    P2
  end

A MapReduce pipeline. The executor runs one task per phase, wired by FIFO channels; the previous phase's output is the next phase's input, and the final phase's output is collected into the response. Built-in phases are pure and deterministic, so the same job over the same inputs is byte-identical across runs.

Because the built-in phases are pure Rust and the channels preserve FIFO order, a job is deterministic: run it twice against the same inputs and you get the same bytes.

Custom phases in WebAssembly

When a job needs logic the built-ins do not cover, a WasmModule phase runs an operator-registered WebAssembly module as a map or reduce step. This requires two things:

1. The binary is built with the wasm feature.
2. The module is registered -- at startup via riak.wasm_modules: (a list of {id, path} entries pointing at .wasm or .wat files) or at runtime.

{
  "inputs": ["events"],
  "query": [
    {"wasm_module": {"id": "sessionize", "kind": "map"}},
    {"reduce": {"fn_name": "reduce_count", "keep": true}}
  ]
}

Combining 2i, links, and MapReduce

The three tools compose into one pipeline. A job can start from a 2i query (find objects by attribute), walk their links (navigate the graph), and reduce the result (aggregate) -- for example, "sum the order totals of every customer in Seattle":

{
  "inputs": {
    "bucket": "customers",
    "index": "city_bin",
    "key": "seattle"
  },
  "query": [
    {"link":   {"bucket": "orders", "tag": "placed"}},
    {"map":    {"fn_name": "map_extract_field", "arg": "total"}},
    {"reduce": {"fn_name": "reduce_sum", "keep": true}}
  ]
}

Read it as: select Seattle customers by 2i, follow each customer's placed links into the orders bucket, extract each order's total, and sum them. The 2i seeds the pipeline, the Link phase traverses, and the map/reduce phases aggregate.

Where to next

• Links and Link Walking -- the Link phase in detail.
• Full-Text, Vector, and Regex Search -- richer query than 2i's equality and range.
• Distributed Transactions -- writing an object and its index entries atomically.
• Dyniak wire protocols -- the exact MapReduce and index wire surface.

Full-Text, Vector, and Regex Search

Secondary indexes answer equality and range queries: age is 42, age is between 10 and 50. That is not enough when you want to find objects by what they contain -- documents mentioning a word, records near a vector in embedding space, values matching a pattern. Dyniak adds a durable search layer for exactly that: full-text substring search, k-nearest-neighbour vector search, and approximate regular-expression search, all built into the process rather than delegated to a separate search cluster.

The search surface is gated behind the search Cargo feature. This chapter shows both faces of it: the FT.* commands on the RESP plane (the same surface RediSearch exposes) and the HTTP index/search routes on the Dyniak gateway.

Two ways to reach the same engine

The text and vector engines are shared. You can drive them two ways:

RESP FT.* commands

The RediSearch-shaped command surface, spoken over the Valkey / RESP plane with valkey-cli or any Redis client. This is the path the search tutorial walks end to end.

Dyniak HTTP routes

Per-bucket index-management and search routes under /buckets/{b}/index/... and /buckets/{b}/search/..., so a Riak-shaped deployment can declare and query indexes over the objects it PUTs.

Both require the search feature and a wired-in vector registry. Without the feature -- or with the feature but no registry -- the search routes reply 501 Not Implemented, and the object, list, and transaction surfaces are unchanged.

The FT.* command surface

The FT.* commands are the fastest way to see the engine work. Build with --features riak (which pulls in search) and point valkey-cli at the node. Create an index over a hash keyspace with a text field and a vector field:

valkey-cli -p 18402 FT.CREATE myidx \
  ON HASH PREFIX 1 doc: \
  SCHEMA title TEXT vec VECTOR HNSW 6 TYPE FLOAT32 DIM 4 DISTANCE_METRIC L2

Store some rows:

valkey-cli -p 18402 HSET doc:1 title "hello world" vec "$(python3 -c '...')"
valkey-cli -p 18402 HSET doc:2 title "hello there" vec "$(python3 -c '...')"

Full-text search

Query the text field for a term:

valkey-cli -p 18402 FT.SEARCH myidx '@title:hello'

The text path is a trigram funnel: the query term is broken into overlapping three-character shingles, the funnel narrows the candidate set by trigram membership, and the survivors are confirmed by exact substring match. You can inspect the trigrams the planner extracted:

valkey-cli -p 18402 FT.EXPLAIN myidx '@title:hello'

Vector search

A KNN query finds the rows whose vectors are nearest the query vector. The query vector is passed as a binary float blob through a PARAMS clause:

python3 -c 'import struct,sys; sys.stdout.buffer.write(struct.pack("<4f", 0.1,0.2,0.3,0.4))' \
  | valkey-cli -p 18402 -x FT.SEARCH myidx '*=>[KNN 3 @vec $blob]' PARAMS 2 blob

The [KNN 3 @vec $blob] clause asks for the 3 nearest neighbours of $blob in the vec field. Under the hood this walks the HNSW graph.

Approximate regex search

FT.REGEX is a Dynomite extension beyond RediSearch. It matches a field against a regular expression with a tunable edit distance K:

# exact regex match (K=0)
valkey-cli -p 18402 FT.REGEX myidx title 'hello' K=0

# allow up to one edit (K=1) -- matches "hallo", "helo", "hellos"
valkey-cli -p 18402 FT.REGEX myidx title 'hello' K=1

# allow up to two edits (K=2)
valkey-cli -p 18402 FT.REGEX myidx title 'hello' K=2

K is the maximum edit distance; it must be a non-negative integer. K=-1, K=foo, and an empty K= are all syntax errors. The matcher is backed by TRE, so full regex syntax works with the approximate budget:

valkey-cli -p 18402 FT.REGEX myidx title 'h(e|x)l*o' K=0

Managing indexes

The operations surface mirrors RediSearch:

FT.LIST (alias FT._LIST)

List every registered index.

FT.INFO <name>

Schema metadata and index counters.

FT.ALTER <idx> ADD <field> <type>

Add a field to a live index.

FT.DROPINDEX <idx> [DD]

Remove the index; with DD, also delete the indexed rows.

FT.EXPLAIN <idx> <query>

Show the query plan (the trigrams, or the KNN shape).

The search tutorial walks all of these end to end with verbatim wire output; treat it as the hands-on companion to this reference.

Search over Dyniak objects (HTTP)

The RESP surface indexes a hash keyspace. The Dyniak HTTP gateway lets a Riak-shaped deployment declare indexes over the objects it PUTs and query them by bucket. For indexing purposes the object's value is interpreted as a JSON document.

Declare a text index on a field

curl -X PUT http://127.0.0.1:8098/buckets/articles/index/text/title

This declares a text index on the title field. On every subsequent object write whose JSON payload carries a string under title, that string is fed into the bucket's trigram-backed text index. Query it:

curl -s 'http://127.0.0.1:8098/buckets/articles/search/text/title?q=machine'

Approximate regex over the same field:

curl -s 'http://127.0.0.1:8098/buckets/articles/search/regex/title?pattern=mach.ne&k=1'

Create a vector index for a bucket

curl -X POST http://127.0.0.1:8098/buckets/articles/index/vector \
  -H 'Content-Type: application/json' \
  -d '{"dim": 384, "metric": "l2", "field": "embedding"}'

A bucket gets one vector index. The JSON body selects the dimension, distance metric, codec, and the document field carrying the vector (default _vector). On every object write whose JSON payload carries a numeric array under that field, the array is upserted into the HNSW engine keyed by the object key; the remaining top-level fields ride along as row metadata for post-filtering. Query it:

curl -s -X POST http://127.0.0.1:8098/buckets/articles/search/vector \
  -H 'Content-Type: application/json' \
  -d '{"vector": [0.1, 0.2, ...], "k": 5}'

List a bucket's declared indexes:

curl -s http://127.0.0.1:8098/buckets/articles/index

The durable index

The search index is not a rebuild-on-restart cache. It is durable: the index state persists across restarts alongside the object data, so a node that comes back after a crash does not have to re-scan the whole keyspace to answer a query. Writes update the index as part of the write path; the index and the objects it covers stay in step.

flowchart LR
  PUT[PUT object<br/>JSON payload] --> STORE[(Noxu: object durable)]
  STORE --> IDX{indexing<br/>best-effort}
  IDX -->|string field| TXT[trigram text index]
  IDX -->|numeric array| VEC[HNSW vector index]
  Q1[text / regex query] --> TXT
  Q2[KNN query] --> VEC
  TXT --> HITS[matching keys]
  VEC --> HITS

The write path stores the object durably first, then feeds declared fields into the text and vector indexes. Queries hit the durable indexes and return matching keys, which the caller can then fetch as full objects.

Choosing a query tool

Exact attribute match or range

Secondary index (2i). Cheaper than search; see Secondary Indexes.

Word or substring in text

Text search (FT.SEARCH text field, or the HTTP text route).

Fuzzy or pattern match

FT.REGEX with an edit-distance budget.

Nearest-neighbour in embedding space

Vector KNN search.

Where to next

• Tutorial: Vector, Text, and Regex Search -- the hands-on, copy-paste walkthrough with real output.
• Secondary Indexes and MapReduce -- the structured query and aggregation surface search complements.
• Dyniak features ops -- operator view of the search feature.

Anti-Entropy and Repair

An eventually-consistent, masterless store lets a write land on some replicas but not others -- a node was briefly down, a partition dropped a message, a hint was never delivered. Left alone, replicas of the same key drift apart. Anti-entropy is the machinery that finds that drift and reconciles it, in the background, without any client involvement. This chapter covers the three layers Dyniak uses to keep replicas converging: read repair on the hot path, hinted handoff for briefly-absent peers, and active anti-entropy (AAE) as the continuous background sweep. It ties into the substrate's failure-handling model in Anti-entropy.

The three layers of repair

flowchart TB
  W[write] --> HH{replica<br/>reachable?}
  HH -->|no| HINT[hinted handoff:<br/>store hint, replay when it returns]
  HH -->|yes| OK[replicated]
  R[quorum read] --> RR{replicas<br/>agree?}
  RR -->|no| REPAIR[read repair:<br/>merge, write back the winner]
  RR -->|yes| SERVE[serve]
  AAE[background AAE sweep] --> DIFF[compare merkle trees,<br/>repair divergent keys]

Three complementary repair layers. Hinted handoff patches writes to a peer that was briefly away; read repair fixes divergence it happens to observe on a quorum read; AAE is the continuous sweep that finds divergence nobody read. Together they drive the cluster toward convergence.

Hinted handoff

When a write's target replica is unreachable, a fallback node stores a hint and replays the write to the real owner when it returns. An explicit FSM drives the chunked, throttled transfer. Same shape as Riak's handoff.

Read repair

A quorum read that sees divergent replicas merges them (by causal context, or by CRDT merge for a data type) and writes the reconciled value back to the stale replicas before answering the client.

Active anti-entropy (AAE)

A continuous background process that compares replicas by exchanging merkle-tree summaries and repairs the divergent keys -- catching the drift that no read ever touched.

Read repair and hinted handoff are opportunistic: they only fix what a request happens to expose. AAE is the safety net that guarantees convergence for cold data.

Active anti-entropy: the Tictac tree

Dyniak's AAE is modelled on Riak's continuously-running merkle-tree synchroniser (the "Tictac" tree). Each node maintains a rolling merkle tree over its (bucket, key, causal-context) tuples. Two peers periodically exchange tree summaries; where the summaries differ, they drill down to the exact divergent keys and repair them.

The tree is two levels deep. The top level splits the keyspace into time buckets; each time bucket splits into segments. A key hashes to one (time-bucket, segment) leaf, and its content hash mixes into that leaf's rolling hash. Changing a key flips its leaf, which flips its segment root, which flips its time-bucket root -- so a divergence is visible at the top of the tree and located by descending.

flowchart TB
  ROOT[tree roots<br/>per time bucket] --> TB0[time bucket 0]
  ROOT --> TB1[time bucket 1]
  TB0 --> S0[segment 0]
  TB0 --> S1[segment 1]
  S0 --> K1[keys hashing here]
  S1 --> K2[keys hashing here]

The two-level Tictac tree. A key mixes into one (time-bucket, segment) leaf; a change ripples up to the roots. Peers compare roots first, then descend only into the diverging subtree.

The three-phase exchange

When two peers compare trees they run a three-phase protocol, each phase narrowing the scope, so the amount of data on the wire is proportional to the divergence and not to the dataset:

1. ROOT-SYNC exchanges the top-level per-time-bucket root vector. Identical roots mean identical data for that time bucket -- no further work.
2. TREE-SYNC recurses into one diverging time bucket and exchanges its per-segment hash vector, isolating the diverging segments.
3. KEY-SYNC enumerates the diverging keys in one (time-bucket, segment) pair and hands them to the repair scheduler.

The exchange rides the substrate's entropy channel -- length-prefixed framing over an AES-128-CBC reconciliation link -- so the tree summaries travel encrypted between peers.

Repair: choosing the winner

Once KEY-SYNC surfaces a divergence, the repair scheduler decides which side holds the correct value and enqueues a repair task on the per-peer outbound channel -- the same channel gossip and hinted handoff use. The winner is chosen by causal context: the merkle tree treats contexts as opaque bytes, and a pluggable comparator decides the order. The default project comparator is the Interval Tree Clock the rest of Dyniak uses for per-key causality (see Buckets, Keys, and Objects).

For CRDT keys the reconciliation is even cleaner: two divergent replicas of a CRDT do not need a winner at all -- they merge, and the merge is the correct converged value by construction (see Convergent Data Types). AAE ships the states both ways and each side merges.

Configuring AAE

AAE is off by default and enabled per pool. The cadence and tree shape are operator knobs:

dyn_o_mite:
  data_store: dyniak
  noxu_path: /var/lib/dynomite/noxu
  riak:
    pbc_listen: 127.0.0.1:8087
    http_listen: 127.0.0.1:8098
    aae_enabled: true
    aae_full_sweep_interval_seconds: 86400   # one full sweep per day
    aae_segment_interval_seconds: 60         # one exchange tick per minute

aae_enabled

Spawn the AAE scheduler. Default false.

aae_full_sweep_interval_seconds

The cadence over which one full sweep across every peer pair completes. Default 86400 (24 hours).

aae_segment_interval_seconds

The cadence of one (peer, time-bucket) exchange tick. Default 60. Must be less than or equal to the full-sweep interval.

The tree shape itself -- number of time buckets, segments per time bucket, time-window width, snapshot cadence -- has sensible defaults and is validated at startup (a zero segment count or an out-of-range time bucket count is rejected). The tree is persisted across restarts, so a node that comes back does not rebuild its tree from a full keyspace scan.

The divergence-proportional alternative: MST reconcile

The fixed-grid Tictac tree walks every segment root on each exchange, so its comparison cost grows with the dataset even when the divergence is tiny. Dyniak ships an optional alternative that scales with the divergence instead: a Merkle Search Tree (MST) reconcile.

Instead of a fixed segment grid, the MST reconcile builds a content- addressed tree over the actual key set and diffs it against a peer's MST. The diff yields exactly the keys present-or-differing on one side, at a cost proportional to the symmetric difference of the two key sets -- not their total size. On a large, mostly-in-sync cluster where a partition touched a handful of keys, the MST path transfers work for those keys and little else.

The MST reconcile also composes with delta shipping: a value change flips exactly one MST entry (the entry's value side is a content digest of the object), just as it flips exactly one Tictac leaf, so a change is located and shipped precisely.

How divergence is reconciled, end to end

Putting the layers together, here is the life of a divergence:

1. A write reaches replicas 1 and 2 but not replica 3 (it was briefly down). If a fallback node held a hint, hinted handoff replays the write to replica 3 when it returns -- and the divergence never outlives the outage.
2. If no hint covered it, a later quorum read that touches replica 3 sees it disagree with 1 and 2, merges, and writes the winner back (read repair).
3. If nobody reads the key, the next AAE sweep compares replica 3's merkle tree with a peer's, drills down to the divergent key, and repairs it in the background.

No single layer is sufficient alone; together they guarantee that a key written to a quorum is not lost and that replicas converge, whether or not anyone reads the data again.

Where to next

• Convergent Data Types -- why CRDT keys reconcile by merge rather than by winner selection.
• Buckets, Keys, and Objects -- the causal context AAE uses to pick a winner.
• Anti-entropy -- the substrate-level failure-handling and repair model.
• Riak mode ops -- the operator view of the AAE scheduler.

Dyniak (Riak PBC / HTTP)

The dyniak data store is the built-in, Riak-compatible backend. It is gated behind the riak Cargo feature and serves two client wire surfaces against the same in-process, transactional Noxu environment:

• Riak Protocol Buffers Client (PBC) -- the binary protocol, with [4-byte BE length][1-byte msg-code][prost body] framing.
• HTTP gateway -- the same operations over GET / PUT / POST / DELETE, negotiating application/x-protobuf, application/json, or application/cbor per request.

The implementation lives in the dyniak crate. For the operator walk-through (building, listeners, AAE, 2i storage layout, bucket properties) see Riak mode. This page is the protocol-surface summary.

PBC message surface

Ping, ServerInfo, Get, Put, Del, GetBucket, SetBucket, ListBuckets, ListKeys, Index (2i), and MapRed, plus the Dynomite cluster-admin extensions (DynListPeers, DynClusterJoin / Leave / Plan / Commit, DynAaeStatus). ListBuckets and ListKeys are chunked into a multi-frame stream.

HTTP routes

The search feature adds index-management and search routes under /buckets/{b}/index/... and /buckets/{b}/search/...; without the feature (or without a registry wired in) those routes reply 501 Not Implemented.

Object links

An object carries typed links: pointers to other objects, each a (bucket, key, tag) triple. Over HTTP they ride in the Link header; over PBC they are RpbLink entries inside RpbContent. Links are not walked by a dedicated route -- they are traversed by a MapReduce link phase (see below).

Secondary indexes (2i)

Objects can carry secondary-index entries (integer *_int and binary *_bin). The PBC RpbIndexReq handler answers equality and range queries. The storage layout and a client example are documented in Riak mode.

MapReduce

A MapReduce job is a pipeline of phases submitted over POST /mapred (HTTP) or RpbMapRedReq (PBC). Phase kinds:

• Map / Reduce -- named functions resolved through the phase registry.
• Link -- follows object links, optionally filtered by bucket and tag; this is how link-walking is expressed.
• WasmModule -- invokes a registered Wasm module as a map or reduce phase. Available only when the binary is built with the wasm feature and the module is registered (via riak.wasm_modules: or at runtime); without it, a WasmModule phase returns a WasmNotImplemented error.

FT.* search

When built with the search feature, the dyniak HTTP gateway exposes per-bucket text (substring + approximate-regex) and vector-KNN index management and search. The same FT.* surface is available on the RESP plane; see Valkey (RESP) and the search tutorial.

Transactions and causality

dyniak extends Riak's per-key model with atomic multi-key transactions and tracks per-key causality with an Interval Tree Clock. See Dyniak features for cross-node XA transactions and the custom Wasm keyfun, and Riak mode for the ITC context blob.

Recommendations

This page is populated by the stage that ports the corresponding subsystem (see PLAN.md). Until then it intentionally stays brief so the manual structure is visible.

Running dynomited

The configuration file

dynomited is configured by a YAML file, one top-level key naming the pool. The smallest working single-node config:

dyn_o_mite:
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  tokens: '101134286'
  servers:
  - 127.0.0.1:22122:1
  data_store: 0

Start it:

cargo run -p dynomited -- --conf-file my.yml
# or, from an installed binary:
dynomited --conf-file my.yml

Key fields

listen

The client-facing address (RESP / memcache clients connect here).

dyn_listen

The DNODE peer plane address (other nodes connect here). See DNODE.

servers

The backing store endpoint(s), as host:port:weight.

data_store

0 = valkey/redis, 1 = memcache, 2 = dyniak. See Configuration.

tokens

The token(s) this node owns on the ring.

datacenter / rack

This node's placement; drives replication and consistency.

dyn_seeds

Peers to gossip with, as host:port:rack:dc:token.

A two-node config adds placement and a seed pointing at the other node:

dyn_o_mite:
  datacenter: dc
  rack: rack
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  dyn_seeds:
  - 127.0.0.2:8101:rack:dc:1383429731
  servers:
  - 127.0.0.1:22122:1
  tokens: '12345678'
  data_store: 0
  stats_listen: 0.0.0.0:22222

See Your First Cluster for a full hands-on walk-through, and Configuration for every field.

Validate before you start

--test-conf parses and validates the file and exits without binding any socket. Use it in CI and before a rolling restart:

dynomited --conf-file my.yml --test-conf && echo ok

Common flags

The complete list, with every flag and its default, is in dynomited(8) and dynomited --help.

Daemonizing and PID files

dynomited --conf-file my.yml --daemonize --pid-file /run/dynomited.pid

--daemonize double-forks, becomes a session leader, and redirects stdio to /dev/null. Pair it with --pid-file so a supervisor can find and signal the process.

Signals

dynomited handles the standard termination signals for a clean shutdown -- it stops accepting new connections, drains in-flight requests, and closes the backend and peer connections before exiting. Send SIGTERM (or SIGINT in the foreground) to trigger it.

Where to go next

• Metrics and Distributed Tracing for observability once it is running.
• Admin CLI (dyn-admin) for inspecting and operating a live cluster.
• Recommendations for production tuning.

dyn-admin: cluster admin CLI

dyn-admin is the operator-facing CLI that ships alongside dynomited. It is the Dynomite Rust port's answer to Riak's riak-admin tool: a single binary an operator points at one running node and asks structured questions of. The subcommand surface is intentionally narrow in v0; mutating operations land in v1 once the gossip substrate exposes the necessary admin-only DNODE messages.

Connecting

Every subcommand reads from a running node over one of two ports:

Override the address with --node <host:port> (or --seed for cluster-list). status and ring-status accept --stats-node to reach a stats endpoint that lives on a different host or port, and --no-stats to skip the HTTP fetch entirely.

Subcommands

ping

Sends RpbPingReq and times the round-trip:

$ dyn-admin ping
PONG from 127.0.0.1:8087 (1.234 ms)
$ dyn-admin ping --json | jq .rtt_us
1234

Exit code 0 on success. Non-zero exit and an error printed on stderr otherwise.

status

Reads RpbGetServerInfoResp from the PBC port and combines it with a slice of the /stats JSON snapshot:

$ dyn-admin status
node: 127.0.0.1:8087
server_node: dyniak
server_version: dyniak 0.0.1
engine_source: node-a
engine_version: 0.0.1
datacenter: dc1
rack: r1
uptime_seconds: 7
pool: dyn_o_mite

When the stats endpoint is unreachable, the human formatter notes stats: unavailable (...) and the JSON formatter sets stats: null plus stats_error: "...".

ring-status

Best-effort ring/topology view. The substrate does not yet expose a multi-peer ring map over PBC, so the v0 output is a single-row table for the contacted node:

$ dyn-admin ring-status
Ring status (queried 127.0.0.1:8087)

node                          dc          rack        state   version               token
----------------------------  ----------  ----------  ------  --------------------  --------
node-a                        dc1         r1          up      dyniak 0.0.1        <unset>

token and richer per-vnode state become populated once the substrate gossips a peer table over PBC. Until then the column is present so the output shape is forward-compatible.

stats

Pretty-prints the high-value fields from the /stats JSON snapshot. --json echoes the raw snapshot byte-for-byte so downstream jq filters keep working:

$ dyn-admin stats
engine_source: node-a
engine_version: 0.0.1
datacenter: dc1
rack: r1
uptime_seconds: 7
pool: dyn_o_mite
latency_p99_us: 1234
latency_max_us: 2345
alloc_msgs: 10
free_msgs: 5

$ dyn-admin stats --json | jq '.dyn_o_mite.client_eof'
0

metrics

Forwards the /metrics Prometheus text endpoint verbatim:

$ dyn-admin metrics | head -3
# HELP dynomite_uptime_seconds Uptime in seconds.
# TYPE dynomite_uptime_seconds gauge
dynomite_uptime_seconds 7

A trailing newline is appended when the upstream body lacks one, so the output is always safe to pipe into line-oriented tools.

cluster-list

Reports what the seed knows about itself, plus a note recording the deferred multi-peer discovery. Once peer-list messages land, the loop here grows naturally:

$ dyn-admin cluster-list --seed 127.0.0.1:8087
Cluster (seed 127.0.0.1:8087)

addr                      node                          state   version
------------------------  ----------------------------  ------  --------------------
127.0.0.1:8087            dyniak                      up      dyniak 0.0.1

note: multi-peer discovery deferred: substrate does not yet expose a peer-list message
      over PBC; reporting only the contacted seed

bucket-props

Inspects or updates a bucket's RpbBucketProps over PBC. Two actions:

• get <bucket> issues an RpbGetBucketReq and pretty-prints the reply.
• set <bucket> [flags] reads the current properties first, applies the named overrides on top, and submits an RpbSetBucketReq. Fields the operator does not name are sent back unchanged so the update is always a partial overlay.

$ dyn-admin bucket-props get users
Bucket properties for users via 127.0.0.1:8087
  n_val: 3
  keyfun: std
  replication_strategy: successors

$ dyn-admin bucket-props set users --n-val 5 --keyfun bucketonly
Updated bucket properties for users via 127.0.0.1:8087
  n_val: 5
  keyfun: bucketonly
  replication_strategy: successors

$ dyn-admin bucket-props get users --json | jq '.props.n_val'
5

Flags accepted by set:

The symbolic quorum names map to Riak's published magic uint32 values so a Riak client and dyn-admin agree on the semantics. set rejects an empty flag list with a hard error rather than a no-op round-trip.

See riak.md for the bucket-property semantics and the locations in the registry where the values are stored.

aae-status

Returns a snapshot of the AAE worker's state: per-peer last-exchange wall-clock time, per-peer divergent-key counts since the most recent full sweep, the configured snapshot path, last save / last load wall-clock times, and the local tree shape (time-buckets * segments) plus a memory estimate:

$ dyn-admin aae-status
AAE status (node 127.0.0.1:8087)

 peer  dc          rack             last_exchange    divergent    repaired
-----  ----------  ----------  ----------------  -----------  ----------
    0  dc1         rA               1700000000           12           9
    1  dc1         rB                        0            0           0

snapshot_path: /var/lib/dynomite/aae/tree.snapshot
snapshot_last_save_unix: 1700000300
snapshot_last_load_unix: 0
snapshot_save_total: 5
snapshot_load_total: 0
snapshot_corruption_total: 0
tree: 24 time-buckets * 1024 segments, window 3600s, ~4096 bytes
2 peer(s)

The JSON form returns the same fields as a stable object suitable for jq filters and dashboard pipelines:

$ dyn-admin aae-status --json | jq .snapshot_save_total
5

When the embedding has not wired an AaeStatusProvider (the default dynomited build does not), the response is an empty snapshot; the CLI prints 0 peer(s) and zero values for every counter rather than failing.

Output formats

• Default: human-readable. One key: value pair per line for the scalar subcommands; bordered tables for ring-status and cluster-list.
• --json: pretty-printed JSON. The schema is stable across patch releases of the v0 line; breaking changes ship under a major bump and are recorded in docs/journal/.

Exit codes

• 0: success.
• 1: any subcommand-level failure (connection refused, timeout, server error, malformed response). The reason is written to stderr as dyn-admin: <error>.
• 2: bootstrap failure (tokio runtime). Rare.

Deferred subcommands

The following riak-admin mutating commands are listed in docs/riak-compat-plan.md Section 5 but absent in v0:

• cluster-join
• cluster-leave
• cluster-plan
• cluster-commit

They are not stubbed; clap rejects them as unknown. The journal entry docs/journal/2026-05-24-dyn-admin-v0.md captures the sequencing: each one needs an admin-only DNODE message that the substrate has not yet exposed, plus a confirmation prompt and a dry-run cluster-plan step before any state mutation.

Stats and metrics

Dynomite exposes its runtime counters, gauges, and histogram rollups on the stats_listen HTTP endpoint. The same in-memory snapshot is served in two formats so operators can pick the one that fits their stack:

• GET / (and the aliases /info and /stats) returns the legacy Netflix Dynomite JSON layout. This is the original wire format and remains byte-for-byte stable; existing scrapers, dashboards, and scripts that target the legacy schema continue to work unchanged.
• GET /metrics returns Prometheus 0.0.4 text exposition. Every metric family is annotated with a # HELP description and a # TYPE declaration. This is the recommended path for modern observability stacks (Prometheus, VictoriaMetrics, Grafana Agent, Mimir, Thanos, OpenTelemetry collectors with the Prometheus receiver).

Both endpoints read the same cached Snapshot value the aggregator publishes once per second, so they always agree.

Metric reference

The table below covers every metric family the Prometheus endpoint emits.

Failure-cause counters

The metrics above describe what the engine is doing when traffic is flowing normally. The counters in this section disambiguate the different error causes the dispatcher and gossip planes can produce. All of them initialise to zero and only become meaningful once the operator wires the dispatcher and gossip handler with a shared FailureMetrics accumulator (the dynomited binary does this automatically; embedders do it via ClusterDispatcher::with_failure_metrics(...) and GossipHandler::with_failure_metrics(...)).

The counters answer questions like "is the cluster losing requests because peers are flapping in and out of Down, or because perper-peer outbound channels are saturated?" Pre-existing aggregate error counters (dynomite_pool_client_err_total, dynomite_pool_client_dropped_requests_total) report the total but do not separate the cause; the families below do.

The _milli suffix on gossip_phi_score_milli is deliberate. Prometheus integer gauges are 64-bit signed; the phi value is a floating-point number that we widen by 1000 to preserve thousandths precision while staying within the integer-gauge wire format. A suspicion threshold of phi > 8.0 therefore corresponds to gossip_phi_score_milli > 8000 in PromQL.

Active Anti-Entropy (AAE) counters

The AAE worker (Tictac merkle-tree exchange + repair sink) ships its own family of counters and gauges. They start at zero and only become meaningful once the embedding wires the AAE handle into the scheduler and repair scheduler:

• dyniak::aae::Scheduler::install_metrics(handle) plus Scheduler::observe_exchange_attempt / observe_exchange_success / observe_divergent_keys from the per-tick hot path.
• dyniak::aae::RepairScheduler::with_metrics(handle, dc, rack) for the repair-dispatched count.
• dyniak::aae::metrics::save_snapshot_with_metrics(...) / load_snapshot_with_metrics(...) for the snapshot counters.

The families and labels:

Sample PromQL queries

Exchange success rate per peer (operators want this near 100%):

sum by (peer_idx) (rate(aae_exchange_success_total[5m]))
  /
sum by (peer_idx) (rate(aae_exchange_attempts_total[5m]))

Divergence rate per DC (a sustained non-zero rate is the signal that AAE is doing useful work):

sum by (dc) (rate(aae_exchange_divergent_keys_total[5m]))

Seconds since last full sweep on every peer (alert when this exceeds the configured full_sweep_interval_seconds):

time() - aae_full_sweep_last_completed_seconds_gauge

Snapshot health indicator (any non-zero corruption rate that is NOT immediately after a deploy is a paging condition):

rate(aae_snapshot_corruption_total[15m])

Sample PromQL queries

Total NoTargets rate per consistency level (this is the metric to watch during chaos runs to confirm peer-state oscillation as the root cause):

sum by (consistency_level) (
  rate(dispatch_no_targets_total[1m])
)

Per-peer flap count over the last hour (a peer that is flapping will show a high transition count):

sum by (peer_idx) (
  increase(peer_state_transitions_total[1h])
)

Live phi score per peer, in raw phi units:

gossip_phi_score_milli / 1000

Dispatch error breakdown (one line per cause; useful as a stacked graph in Grafana to see which cause dominates the error budget):

sum (rate(dispatch_no_targets_total[1m])) +
sum (rate(dispatch_peer_send_full_total[1m])) +
sum (rate(dispatch_peer_send_closed_total[1m])) +
sum (rate(dispatch_backend_send_full_total[1m])) +
sum (rate(dispatch_backend_send_closed_total[1m])) +
sum (rate(dispatch_response_timeout_total[1m]))

The histogram quantile rollups are emitted as gauges, not as Prometheus histograms, because the engine stores Cassandra-style estimated histograms whose internal buckets are not the standard Prometheus le ladder. Exposing the pre-computed mean, 0.95, 0.99, 0.999, and max rollups keeps the wire payload small while preserving the same percentiles the JSON endpoint already publishes.

Sample scrape configuration

A minimal prometheus.yml snippet that scrapes a three-node cluster on port 22222:

scrape_configs:
  - job_name: dynomite
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets:
          - dynomite-0.example.internal:22222
          - dynomite-1.example.internal:22222
          - dynomite-2.example.internal:22222
        labels:
          cluster: prod-east

Sample Grafana panel

A single-stat panel that charts cluster-wide request volume from the counter dynomite_pool_client_read_requests_total:

{
  "type": "timeseries",
  "title": "Pool requests/sec",
  "targets": [
    {
      "expr": "sum by (pool) (rate(dynomite_pool_client_read_requests_total[1m]))",
      "legendFormat": "{{pool}}",
      "refId": "A"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "reqps"
    }
  }
}

Drop the panel into a Grafana dashboard JSON under panels[] and adjust the datasource to point at your Prometheus instance.

Distributed tracing and OTLP logs

dynomited ships with first-class OpenTelemetry support: when the operator sets one or both of the OTLP endpoint fields, every client request fans out into a span tree and every tracing event is mirrored as an OTLP log record. Any OTLP-aware collector (Jaeger, Tempo, Honeycomb, the OpenTelemetry Collector, Loki via the OTLP receiver, ...) can consume the resulting streams. Both exporters are off by default - the binary pays no OTel-SDK cost when neither field is set.

Span shape

A successful round-trip emits the following tree, anchored at the per-connection accept span:

client.accept                                 (per accepted client TCP connection)
  client.parse        msg_id, msg_type        (per parsed inbound request)
    dispatch.plan     req_id, plan, targets   (synchronous routing decision)
    backend.send      req_id, bytes           (local datastore write; LocalDatastore plan)
      backend.parse   req_id, bytes           (response parsed off the backend wire)
    peer.send         req_id, bytes           (cross-rack / cross-DC fan-out; Replicas plan)
      peer.parse      req_id, bytes           (DNODE response parsed off the peer wire)
    client.send       req_id, bytes           (response writeback to the client)

Plus the supervisor-level long-lived spans created at startup:

server.run            pool, listen, peers
  proxy.run           local
  dnode_proxy.run     local
  backend_supervisor  backend, ds
    run_one_backend_conn
  peer_supervisor.spawn  peer_idx, peer
    peer_supervisor   peer
  stats_server.run    local

The cross-task work (backend.send, backend.parse, peer.send, peer.parse, client.send) nests under the originating client.parse span because the dispatcher captures tracing::Span::current() on the way out of the dispatch call and the receiver tasks re-enter that span before doing their work. The default Span::none() is zero-cost, so this propagation is free when the OTLP exporter is off.

Configuration

Add an observability: block to the pool body:

my_pool:
  listen: 0.0.0.0:8102
  dyn_listen: 0.0.0.0:8101
  tokens: '101134286'
  servers:
  - 127.0.0.1:22122:1
  data_store: 0
  observability:
    # Enables distributed tracing. OTLP gRPC URL of the trace
    # collector. Unset (or empty string) disables the trace
    # exporter entirely.
    otlp_traces_endpoint: "http://localhost:4317"
    # Enables the OTLP log appender. OTLP gRPC URL of the log
    # collector (often the same collector as the trace one).
    # Unset (or empty string) disables the log exporter
    # entirely. The fmt layer (stderr or `--output` file) keeps
    # writing in parallel; this knob only adds the OTLP
    # appender.
    otlp_logs_endpoint: "http://localhost:4317"
    # Optional. Overrides the service.name resource attribute
    # attached to every span and log record. Defaults to
    # "dynomited".
    service_name: "dynomited-prod-us-east-1"
    # Optional. Trace sampling ratio in [0.0, 1.0]. Values <1.0
    # apply a TraceIdRatioBased sampler. Defaults to 1.0
    # (record every trace). Does not affect the log appender:
    # every event that passes the global EnvFilter is exported.
    traces_sampling: 0.05

When both otlp_traces_endpoint and otlp_logs_endpoint are unset, the binary uses the same plain fmt subscriber it always has and pays no OTel-SDK cost. Either knob alone is fine; the binary installs only the pillar(s) you turn on.

Sample collector setup

A minimal OpenTelemetry Collector configuration that accepts spans and log records from dynomited over OTLP/gRPC and forwards them to a local Jaeger and a local file-backed log store:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  file/logs:
    path: /var/log/dynomite/otlp.log

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/jaeger]
    logs:
      receivers: [otlp]
      exporters: [file/logs]

Run that side-by-side with dynomited and point the pool's otlp_traces_endpoint and / or otlp_logs_endpoint at http://collector:4317. Spans show up in Jaeger and log records land in /var/log/dynomite/otlp.log within a few seconds.

Operator-visible knobs

Trade-offs

• Single global subscriber. tracing only allows one global subscriber per process, so the binary composes the fmt layer (with the configured --log-format / log_format: shape and the SIGHUP-reopen handle), the EnvFilter, the OTel trace layer (when traces are on), and the OTLP log appender (when the log endpoint is on) into one Registry that is installed exactly once. The fmt layer's SIGHUP-reopen state is populated whether OTLP is on or off; SIGHUP-driven log rotation works in every mode.
• Performance. The OTel SDK's batch processors run on the existing tokio runtime; a 1.0 sampling ratio over a multi-thousand-QPS workload routinely doubles per-request allocation count vs the no-exporter baseline. Production deployments usually run at traces_sampling: 0.01 or lower. The log appender does not have a sampling knob; gate volume via RUST_LOG instead.
• Backpressure. If the collector is unreachable the batch processors drop spans / log records after their internal queues fill (default 2048 each). They do not slow the request path.
• Local logs and OTLP logs are independent. The fmt layer always writes to the configured stderr / --output file. The OTLP log appender mirrors the same events to the collector. Operators see the local stream even if the collector is down.

Implementation notes

• Trace context propagation. The dispatcher hands an OutboundRequest to either the backend supervisor or the per-peer DNODE driver via an mpsc channel. Both envelopes (OutboundRequest and OutboundEnvelope) carry a tracing::Span field that captures tracing::Span::current() at send time. The receiving task re-enters the captured span before doing its work so cross-task spans nest correctly under the originating client request.
• Sync-only spans across awaits. Where a span guard would otherwise cross an .await boundary (EnteredSpan is !Send), the code uses Span::in_scope for synchronous work and Instrument to attach a span to a future. This keeps the spawned futures Send and tokio-spawn-compatible.
• Log-record body and attributes. The OTLP log appender is the unmodified opentelemetry-appender-tracing bridge (0.27 release train). The event message lands in the OTLP log record body; structured fields land as record attributes. The instrumentation scope is fixed at opentelemetry-appender-tracing. Trace context is automatically attached when an event fires inside a span that the OTel trace layer also sees, so log records can be correlated to spans in the collector.

Distribution modes

This page is the operator's reference for the two first-class distribution algorithms the engine supports: vnode (the historical default) and random_slicing (new). The engine-level reference, including the C-to-Rust mapping, lives in docs/design/random-slicing-integration.md; the configuration syntax is in Configuration; this page covers operator workflows.

When to pick which

• vnode is the historical algorithm: each peer publishes a list of tokens and the dispatcher walks a per-rack continuum to find the owning peer. Pick vnode when you have an existing operator-managed token plan you want to preserve byte-identically, or when you depend on the exact vnode-to-peer mapping for an external system (e.g. a backup pipeline that walks the per-peer token list).
• random_slicing is the recommended mode for new deployments. Coverage is gap-free by construction: the chaos pass-3 failure mode (a 3-of-4 host topology silently leaving a quarter of the ring unowned) is structurally impossible. The operator-facing knob shrinks from "list of magic 32-bit integers per peer" to "one float per peer" (or simply nothing for a uniform partition).

In --features riak builds, random_slicing is the default when a Riak listener is configured.

Migration playbook

1. Run shadow mode

Set distribution_shadow: in the YAML (or pass --distribution-shadow=random_slicing on the command line). Every routing decision then computes both the live vnode plan and the shadow random_slicing plan; the dispatcher routes by distribution: (the live mode) and bumps distribution_shadow_disagreement_total whenever the two disagree.

dynomited --conf-file /etc/dynomite.yml \
          --distribution-shadow=random_slicing

Run shadow mode for a working day. Watch the counter:

dyn-admin distribution-dump --node 127.0.0.1:22222
# or, for the raw counter,
curl -s 127.0.0.1:22222/metrics | grep distribution_shadow

The counter is a u64 that grows monotonically; a stable value means no recent disagreements (which on a fresh cluster will never happen because the algorithms produce independent partitions).

2. Cut over

Edit the pool YAML:

dyn_o_mite:
  ...
  distribution: random_slicing       # was 'vnode'
  distribution_shadow: vnode         # keep the old mode as
                                     # the shadow for safety

Issue kill -HUP $(pgrep dynomited) on every node. The SIGHUP-reload pipeline rebuilds the rack ring atomically; no restart is required. The first request after the reload that lands on a peer that does not yet hold the key locally returns a miss (memcache) or kicks off a read-repair (Redis dyn-mode); the cluster converges through the usual entropy / repair machinery.

3. Drop the shadow

Once the cluster is happy and the disagreement counter has stopped growing on every node, remove distribution_shadow: from the YAML and SIGHUP again. The shadow path is now fully dormant.

Rollback

If shadow mode disagreed but the operator cut over anyway and is now unhappy, revert the YAML and SIGHUP. The vnode ring rebuilds deterministically from the unchanged tokens: lists. Keys written under random_slicing are returned by their original vnode owner once that owner runs read-repair against the new primary.

Peer-state interaction

In v1, the random-slicing slice table is built once per rebuild_ring and includes every peer in the rack regardless of state. Down peers are filtered out by the dispatcher's existing is_routable() filter on top of the slice lookup; a Down peer is invisible to a per-key route just like it is under vnode. Removing or replacing a peer requires a configuration reload (the gossip path that already does this for vnode is unchanged).

This v1 limitation is documented in docs/parity.md under the random-slicing Deviation entry.

Riak mode

dynomited ships an optional Riak-compatible protocol surface through the dyniak crate. The surface is gated behind the riak Cargo feature so operators who do not run Riak workloads pay nothing for the extra dependencies and listeners.

Building

cargo build -p dynomited --features riak

Without the feature, dynomited builds and runs identically to a Redis / Memcache deployment. The YAML riak: block is parsed and validated either way; under the default build it is a no-op at run time.

Listeners

Two independent listeners are available:

• PBC (Protocol Buffers Client) -- Riak's binary wire format. Hand-rolled prost-derived messages plus the standard [4-byte BE length][1-byte msg-code][prost body] framing. The surface covers RpbPingReq/Resp, RpbGetServerInfoReq/Resp, RpbGetReq/Resp, RpbPutReq/Resp, RpbDelReq/Resp, RpbGetBucketReq/Resp, RpbSetBucketReq/Resp, RpbListBucketsReq/Resp, RpbListKeysReq/Resp (both chunked into a multi-frame stream), RpbIndexReq/Resp (secondary indexes), and RpbMapRedReq/Resp (MapReduce), plus the Dynomite cluster-admin extensions (DynListPeers, DynClusterJoin / Leave / Plan / Commit, DynAaeStatus) and error responses. Whether a given operation succeeds also depends on the backing datastore's capabilities: 2i and MapReduce require an object-capable store such as the Noxu-backed dyniak datastore; against a plain RESP / memcache backend those calls return a not implemented for this datastore error.
• HTTP gateway -- the same operations exposed over application/x-protobuf, application/json, or application/cbor via the dyn-encoding registry. The GET /ping endpoint is the simplest liveness probe.

Either listener can be enabled on its own; both can run side-by-side. They share a single Datastore so request accounting accumulates in one place.

YAML configuration

my_pool:
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  tokens: '101134286'
  servers:
  - 127.0.0.1:6379:1
  data_store: 0
  riak:
    pbc_listen: 127.0.0.1:8087
    http_listen: 127.0.0.1:8098
    aae_enabled: true
    aae_full_sweep_interval_seconds: 86400
    aae_segment_interval_seconds: 60

aae_segment_interval_seconds must be <= aae_full_sweep_interval_seconds. The validator surfaces a BadServer error when either constraint is violated; dynomited -t -c <file> reports the same diagnostic.

CLI overrides

Three flags override the YAML at startup; each requires the binary to be compiled with --features riak:

--riak-pbc-listen HOST:PORT     override riak.pbc_listen
--riak-http-listen HOST:PORT    override riak.http_listen
--riak-aae-enabled              force-enable the AAE scheduler

The flags compose with the rest of the CLI: an existing YAML without a riak: block can have one materialised purely from the command line.

PBC vs HTTP gateway

Both listeners execute against the same underlying Datastore. Choice between them is operational rather than correctness-driven:

• PBC is the lower-overhead binary protocol and matches the Erlang Riak client libraries' default. Use this for production workloads where every microsecond counts.
• HTTP is human-debuggable, traverses any L7 proxy, and supports content-type negotiation (protobuf / JSON / CBOR) for clients that prefer text formats. Use this for ops tooling, smoke tests, and any environment where HTTP middleware is already in place.

Active anti-entropy (AAE)

When aae_enabled: true, a background scheduler ticks at aae_segment_interval_seconds and walks the configured peer rotation. A future slice will wire the per-peer Tictac tree exchange and surface RepairTasks to the per-peer outbound channels (the same mpsc::Sender<OutboundRequest> map used by gossip and hinted handoff). For now the task is observable via tracing::debug! events under the dynomite::riak::aae target so operators can confirm that the cadence is firing as configured.

The repair sink wiring is materialised today as a PeerChannelRepairSink adapter living next to the scheduler; once the exchange protocol lands, only the body of the scheduler tick has to change.

Noxu as the backing datastore

The Riak protocol surface speaks to a Datastore implementation; operators select which one via the pool's data_store: knob. With --features riak, dynomited accepts a third value alongside the historical valkey (0, also accepted as the back-compat alias redis) and memcache (1):

dyn_o_mite:
  listen: 127.0.0.1:8102
  dyn_listen: 127.0.0.1:8101
  tokens: '0'
  data_store: dyniak        # or '2', mirroring the integer form
  noxu_path: /var/lib/dynomite/noxu
  servers:
  - 127.0.0.1:6379:1        # placeholder, ignored under dyniak
  riak:
    pbc_listen: 127.0.0.1:8087
    http_listen: 127.0.0.1:8098

When data_store: dyniak is set:

• The pool opens an in-process Noxu environment in transactional mode at noxu_path: and serves the dyniak Riak PBC / HTTP surface directly against it.
• The pool does NOT run a RESP client proxy and does NOT dial an external backend: there is no RESP backend supervisor and the listen: address is not bound. All traffic enters through the Riak PBC / HTTP listeners.
• A 2i index entry written via RpbPutReq is visible to subsequent RpbIndexReq queries against the same environment.
• The servers: list is preserved for schema compatibility but is not contacted; the placeholder 127.0.0.1:6379:1 is conventional.

If dynomited is built without --features riak, selecting data_store: dyniak is rejected at configuration validation time with dyniak data_store requires dynomited built with --features riak.

Secondary indexes (2i)

The Noxu-backed Datastore implements the Riak 2i extension trait methods used by the PBC RpbIndexReq handler. Two query types are supported:

• Equality (qtype: 0): scan keys whose index value matches exactly.
• Range (qtype: 1): scan keys whose index value falls inside [range_min, range_max] (inclusive bounds).

Index entries are attached to an object at put time. The RpbPutReq.indexes field carries one RpbPair per entry where pair.key names the index (suffixed with _int for integer indexes, _bin for binary indexes) and pair.value carries the value bytes.

Example (Python, using riak PBC client):

client = riak.RiakClient(host='127.0.0.1', pb_port=8087)
b = client.bucket('users')
o = b.new('alice', data='profile')
o.add_index('age_int', '42')
o.add_index('city_bin', 'seattle')
o.store()

# Equality query:
hits = b.get_index('age_int', '42').results
# Range query:
hits = b.get_index('age_int', '10', '50').results

Storage layout (deviation from upstream Riak's 2i_partition_table schema): index entries are stored as plain records inside the same Noxu environment as the primary KV data, under three reserved prefixes:

• Primary: K\\0{bucket}\\0{key} -> value
• Forward 2i: I\\0{bucket}\\0{name}\\0<u32-be vlen>{value}{key} -> empty
• Reverse 2i: R\\0{bucket}\\0{key} -> length-prefixed (name, value) list, used to clean stale forward entries on delete / overwrite.

The fixed-width length prefix on the value keeps prefix scans unambiguous when value bytes contain the structural separator; see docs/journal/2026-05-24-noxu-firstclass-and-2i.md for the schema rationale.

Streaming response and follow-ups

The current RpbIndexResp handler emits a single response frame with done = Some(true). Streaming (one frame per chunk plus a body-less terminator) is scoped as a follow-up. The $key reserved internal index for primary-key range queries is also deferred.

Default distribution

--features riak builds default the pool's distribution: to random_slicing whenever a Riak listener is configured (riak.pbc_listen or riak.http_listen). The choice mirrors classic Riak behaviour: Riak-shaped deployments inherit a gap-free partition table by default, so a 3-of-4 host topology cannot silently leave a quarter of the ring unowned.

Operators who want the legacy vnode behaviour can still set distribution: vnode explicitly in the YAML; the override is respected. See Distribution modes for the full reference and migration playbook.

Causality tracking

The Riak surface tracks per-key causality with an Interval Tree Clock (ITC). ITC is the default for every Riak listener; operators do not normally need to think about it.

What it does, in one paragraph: each per-key context blob the server returns to a client is a small encoded clock that records the causal history of the key as a pair of small trees (an id tree describing event-issuing authority shared between live actors and an event tree describing observed history). Clients echo the blob back on the next update so the server can make a correct merge / sibling decision. Compared with classic vector clocks, ITC scales with the population of currently-live actors instead of the population of every actor that has ever participated -- retired actors leave no residual cost in the clock. The on-the-wire shape of the context blob is opaque to clients (you round-trip the bytes verbatim), so a client that treats the context as opaque continues to work without modification. Clients that crack the blob and parse it as a Riak DVV need to switch decoders; the byte shape is documented as a deviation under docs/parity.md D4 and the "Causality clock divergence" ambiguity entry.

Operator-visible behaviour is unchanged from a typical client's perspective: the same R / W quorum semantics, the same sibling presentation, the same return_body shape on DtUpdateResp.

References:

• Almeida, Baquero, Fonte, "Interval Tree Clocks: A Logical Clock for Dynamic Systems" (2008).
• The implementation lives in crates/dyniak/src/datatypes/itc.rs; the deviation is recorded in docs/parity.md D4 and the migration notes in docs/journal/2026-06-01-dvv-to-itc.md.

Bucket properties

Two operator-confirmed bucket-property knobs let Riak deployments match the upstream behaviour byte-for-byte without forcing every deployment onto the same defaults. See the dyn-admin bucket-props reference for the operator-facing CLI that fetches and edits these knobs over PBC.

chash_keyfun: bucket-only hashing

By default Dynomite hashes <bucket>/<key> to choose a partition; this is Riak's chash_std_keyfun. Some deployments want every key in a bucket to land on the same partition (so the bucket is effectively a single shard); for that case the per- bucket-property chash_keyfun selector accepts BUCKETONLY, which hashes only <bucket>. The wire-level enum is:

The selector is stored in RpbBucketProps.chash_keyfun (Dynomite extension at tag 30). Set it through the standard RpbSetBucketReq admin path; the in-memory enum is dyniak::datatypes::keyfun::KeyFun. The shaping happens before the cluster's hash function: the distribution layer (vnode or random-slicing) keeps consuming the already-hashed bytes verbatim.

replication_strategy: walk-N-successors

Dynomite's classic replication fans a write across datacenters and racks per the configured consistency level; Riak instead replicates a key to the primary partition plus the next n_val - 1 peers reached by walking forward on the ring, deduplicating peers with multiple ring slots. Both models are now available behind a per-bucket-property selector:

The default is mode-aware:

• Non-Riak pools always run TOPOLOGY; the knob is not exposed to operators.
• Riak-mode pools default to SUCCESSORS for newly-created bucket-types. Operators override per-bucket-type by RpbSetBucketReq.

Edge cases honoured by the planner:

• Fewer peers than n_val: the plan returns whatever peers are available and the operator sees a tracing::warn! at config- validation time.
• Peers in Down state: NOT skipped during planning (they are returned as targets and the runtime is_routable() filter handles the actual exclusion). This matches the topology mode's behaviour.

The selector is stored in RpbBucketProps.replication_strategy (Dynomite extension at tag 31). The in-memory enum is dyniak::replication::ReplicationStrategy; the planning function is dyniak::replication::plan_replicas.

Dyniak features

The built-in dyniak store (the riak Cargo feature) ships a few capabilities beyond a stock Riak deployment. This page documents the ones that need operator attention. The protocol surface is summarised in Dyniak (Riak PBC / HTTP); the listener and storage operations are in Riak mode.

Cross-node XA transactions

dyniak extends Riak's per-key, eventually-consistent model with atomic multi-key transactions: a client groups several put and delete operations into one batch over POST /transactions (cluster-wide) or POST /buckets/{bucket}/transactions (bucket-scoped), and the backend applies all of them or none of them.

Two layers stack:

• Single-environment: every op in the batch routes to one node's storage engine and commits in a single engine transaction.
• Cross-node: when the batch touches keys owned by different primary nodes, the operations are coordinated with X/Open XA two-phase commit. Each node prepares its branch; the coordinator commits all branches only once every prepare has voted to commit. A branch that performed no writes votes read-only and skips the second phase. Cross-node coordination travels over the DNODE peer plane.

The cross-node coordinator handles the network failure modes that a single-process commit never sees: presumed-abort on prepare, commit-in-doubt forward recovery, and a durable in-doubt log that a cold restart re-drives so a branch that voted to commit is never left dangling.

A transaction whose force_abort flag is set rolls back every prepared branch regardless of votes.

Custom Wasm keyfun (chash_keyfun: CUSTOM)

By default a key is routed by hashing <bucket>/<key> (STD), and a bucket property can switch a bucket to hash <bucket> only (BUCKETONLY); see bucket properties. A third option, CUSTOM, routes through an operator-supplied Wasm module instead of a fixed rule.

Scoped limitation: CUSTOM is only usable once a Wasm keyfun module is registered. A bucket whose chash_keyfun is CUSTOM but which has no module registered cannot be routed, and the engine returns a typed error rather than guessing. The module must export a keyfun_route entry point. Registration reuses the same Wasm module store, compilation cache, and resource limits as the MapReduce executor.

Link-walking

Objects carry typed links -- (bucket, key, tag) pointers to other objects. There is no dedicated link-walk route; links are traversed by a MapReduce Link phase, optionally filtered by bucket and tag. Chain a Link phase into a MapReduce pipeline (POST /mapred or RpbMapRedReq) to follow links and feed the resolved objects into the next phase.

Wasm MapReduce phases

A MapReduce pipeline can include a WasmModule phase that runs a registered Wasm module as a map or reduce step. This requires the binary to be built with the wasm feature, and the module to be registered -- either at startup via riak.wasm_modules: (a list of {id, path} entries pointing at .wasm or .wat files) or at runtime. Without the wasm feature the phase is parsed and validated but a submission returns a WasmNotImplemented error.

Benchmarks

The workspace ships two bench harnesses:

• Micro (crates/dynomite/benches/): per-component criterion benches. Runs unattended on any developer machine.
• Macro (crates/dynomite/benches/macro_throughput.rs): live three-node cluster driven through a tc/netem impairment matrix. Gated behind the bench-macro feature because it needs CAP_NET_ADMIN to install netem qdiscs on the loopback interface.

Micro benches

The micro suite covers seven components:

Running

cargo bench --bench parsers -p dynomite
cargo bench --bench mbuf    -p dynomite
cargo bench --bench hashkit -p dynomite
cargo bench --bench tokens  -p dynomite
cargo bench --bench dnode   -p dynomite
cargo bench --bench crypto  -p dynomite
cargo bench --bench quorum  -p dynomite

-- --test smoke-runs each case once; CI uses this mode.

Baselines and the regression budget

Every micro bench has a baseline manifest at crates/dynomite/benches/baseline/<bench>.json. The manifest records the criterion baseline name (stage-15), the capture timestamp, the git sha of the captured run, and the per-case regression budget (10% by default).

Capture a baseline on a quiescent host:

cargo bench --bench parsers -p dynomite -- --save-baseline stage-15

Compare a new run against the recorded baseline:

cargo bench --bench parsers -p dynomite -- --baseline stage-15

The CI gate consumes the criterion change/ reports under target/criterion/<bench>/ and exits non-zero on any case whose median time regressed by more than the manifest's regression_budget_pct.

Macro benches

The macro harness exercises a live three-node cluster on localhost, drives valkey-benchmark for 30 seconds per condition, and writes the per-condition latency snapshot to target/bench/macro-<git-sha>.json. Conditions:

• baseline (no impairment)
• delay 5ms
• loss 1%, loss 5%, loss 10%
• corrupt 0.1%
• reorder 25% 50%

Setup

The harness installs tc qdisc ... netem on the loopback interface; this needs CAP_NET_ADMIN. Two options:

1. Run as root: sudo cargo bench --features bench-macro --bench macro_throughput. Not recommended outside CI.

2. Grant CAP_NET_ADMIN to a worker namespace:

   sudo unshare -n bash -c 'ip link set lo up && exec sudo -u $USER cargo bench --features bench-macro --bench macro_throughput'
   

   The flake's shellHook documents the same recipe in detail.

The harness clears any qdisc it installed at the end of every condition. If the harness aborts mid-run, clear leftover state manually:

sudo tc qdisc del dev lo root netem

Output

target/bench/macro-<git-sha>.json

Each entry is one condition with ops_per_sec, latency_p50_us, latency_p99_us, latency_p999_us, latency_p9999_us, and wall_seconds. The Stage 15 commit ships the orchestration scaffold; the actual workload generator (a valkey-benchmark spawn) is the operator's responsibility because CI does not have permission to install netem qdiscs.

CI integration

scripts/check.sh does not run cargo bench. The bench gate is opt-in:

scripts/check.sh                   # default; no benches
cargo bench --workspace            # opt-in micro suite
cargo bench --features bench-macro # opt-in macro suite

CI for tagged releases additionally runs the micro suite with --baseline stage-15 and fails on per-case regression beyond the manifest's budget.

Conformance suite

The conformance suite verifies that dynomited (the Rust port) behaves equivalently to the upstream C dynomite daemon on a representative workload. It is the entry gate for "drop-in replacement" claims: every supported transport, every consistency level, and every Redis command class the C harness exercised has a corresponding Rust scenario.

Layout

The suite is one Cargo integration-test binary plus a companion differential rig:

crates/dynomited/tests/
  conformance.rs                  - test crate entry
  conformance/
    mod.rs                        - cluster spawner, redis
                                    backend spawner, RESP client
    single_node.rs                - 1-node Redis workload
    three_node_single_dc.rs       - 3-node single-DC workload
    multi_dc.rs                   - 2 DCs * 2 racks * 2 nodes
    quic_transport.rs             - QUIC transport (gated)
    python_harness.rs             - Rust adaptation of the
                                    functional test scenarios
  differential.rs                 - C-vs-Rust corpus driver
  fixtures/conformance/commands.txt - 100+ RESP/Memcached lines

Each scenario starts with a runtime check for valkey-server on PATH. When Redis is missing the test prints a skip notice and returns successfully; the suite never fails just because Redis is not installed.

Running locally

The Nix flake provides valkey-server, cargo-nextest, and the rest of the toolchain. From the workspace root:

nix develop
cargo nextest run --profile conformance \
    -p dynomited \
    --features integration \
    --test conformance --test differential

Add --features integration,quic to also exercise the QUIC transport scenarios. Without --features quic the QUIC file is not compiled.

JUnit output

The conformance profile in .config/nextest.toml writes a JUnit XML report to target/nextest/conformance/junit.xml. scripts/check.sh mirrors that file to target/junit/conformance.xml so CI workflows can upload it verbatim. Both paths are disposable (target/ is gitignored); the canonical run is invocation-by-invocation.

Differential rig

tests/differential.rs reads tests/fixtures/conformance/commands.txt, decodes each line into a wire frame, and drives it through the Rust cluster. Set CONFORMANCE_C_BINARY=/path/to/dynomite to also drive the C reference; without that env var the rig records the Rust replies under target/conformance/divergence/<id>.rust and skips the byte-equivalence assertion (the C reference build is not yet wired into the workspace).

Cleanup discipline

Every spawned valkey-server and dynomited child runs in its own process group (std::os::unix::process::CommandExt::process_group(0)). The Cluster Drop impl sends SIGTERM to each process group, waits a short grace window, then upgrades to SIGKILL. The unit test helpers::tests::drop_kills_child_process_group proves the guarantee end-to-end: spawning a long-sleeping child and dropping the wrapper terminates the entire group, even on a panic-driven unwind.

Adding scenarios

1. Drop a new *.rs file under tests/conformance/.
2. Add mod <name>; (or #[path = ...] mod <name>;) to tests/conformance.rs.
3. Use the helpers::Cluster::launch builder to spin up the topology you need; assert against RespClient replies.
4. Update docs/parity.md with any new C-side behaviour the scenario covers.

What the suite does NOT cover

• The 1-hour chaos test (PLAN.md Stage 16).
• The >= 95% coverage gate (PLAN.md Stage 15). Stage 14 pushes the listener / conn-FSM / dispatcher modules above 90%; the remaining gap is documented as a Stage 15 prerequisite in docs/parity.md Deviations.
• Live byte-level differential against the C reference for workloads beyond the 100-command corpus. That gate lights up once a static-lib build of dynomite is wired into target/cref/ (a Stage 16 packaging task).

Coverage gate

The workspace ships a tiered, blocking coverage gate. Core components must reach 95% line and function coverage; supporting and tool crates must reach 75%. The gate is enforced by scripts/coverage_gate.sh; CI runs it as part of scripts/check.sh (without || true, so it can fail the build).

The tiers are:

• Core (95%): the engine proto / cluster / io / hashkit / crypto / msg / core / net layers and the dyniak datastore / proto / datatypes / mapreduce layers -- the code a customer's data integrity depends on.
• Supporting (75%): the remaining library crates (dynomite-search, gen-fsm, dyn-sup, dyn-encoding, dynomite-text, dynomite-vec, dyn-hashtree, throttle-core).
• Tool (75%): dyniak-bench, dyn-hash-tool, dyn-admin, and the test-harness crates loom-tests / model-tests.

Running locally

scripts/coverage_gate.sh             # enforce the tiered policy
scripts/coverage_gate.sh --report    # report-only; do not fail

The script writes:

• target/coverage/summary.json - the raw cargo-llvm-cov summary (--json --summary-only).
• target/coverage/report.txt - human-readable percentages, documented deviations (warnings only), and undocumented per-file deviations (errors).

How the gate decides

1. The script invokes:

   cargo llvm-cov --workspace --features riak --summary-only --json \
       --output-path target/coverage/summary.json
   

2. It walks every per-file source entry under crates/ (skipping tests/, benches/, and the fuzz crate), assigns each file its tier threshold, and compares the file's line and function coverage to that threshold.

3. A file below its tier is classified as either:

   • Documented deviation: listed in docs/coverage-deviations.md. The gate logs a warning but does not fail.
   • Undocumented deviation: any file under its tier that is not in the deviations list. The gate fails.

4. The workspace-wide line, region, and function percentages are printed for trend tracking but are not themselves gated; the per-file tier policy is the enforcement axis.

Tracking deviations

docs/coverage-deviations.md lists each module allowed below its tier along with its line / region / function percentages and a concrete reason. Every entry is reachable only by an out-of-process suite (the conformance harness or the chaos rig), is a re-export facade, is process bootstrap, is rendering output, or has only unreachable defensive arms left -- none is an untested unit of pure logic. Regenerate the table with:

scripts/coverage_gate.sh --report
python3 scripts/regen_coverage_deviations.py

When you reduce a deviation:

1. Add tests that lift the module to its tier threshold.
2. Run scripts/coverage_gate.sh --report and regenerate the deviations table; the file drops out automatically.

Soak coverage

The soak job runs property tests at 1M cases each (make soak) and re-runs scripts/coverage_gate.sh with the same tiered thresholds. The expectation is that soak coverage matches or exceeds the per-PR coverage; any regression is a soak finding that blocks the next release tag.

Chaos test

The chaos test exercises a multi-DC dynomite cluster under continuous failure injection for one hour and asserts that no client request observes a permanent error and that every documented invariant holds throughout the run. It is the final pre-tag gate.

Location

• Test: crates/dynomite/tests/stage_16_chaos.rs
• Failure injectors: scripts/netem/
  • partition_dc.sh
  • slow_peer.sh
  • flap.sh
  • gc_pause.sh
  • clock_skew.sh
• Run artefact: target/chaos/<run-id>/report.md

Modes

The test honours two environment variables:

The test is gated behind --features chaos so a stock cargo test does not pull it in. Two practical run shapes:

# 60-second smoke (CI-friendly when --features chaos is set)
CHAOS_DURATION_SECS=60 \
  cargo nextest run --workspace --features chaos --test stage_16_chaos

# 1-hour production run (manual, requires CAP_NET_ADMIN)
sudo -E env "PATH=$PATH" \
  cargo nextest run --release --workspace --features chaos --test stage_16_chaos \
    --no-capture

Prerequisites

The test self-checks each prerequisite and emits a SKIP notice (test passes, body is a no-op) when one is missing:

• valkey-server on PATH. The harness spawns one Redis backend per dynomite node.
• tc (iproute2) and CAP_NET_ADMIN (or root). Required by every scripts/netem/* injector.
• faketime (libfaketime). Required by the clock-skew injector at the 30-minute mark.

When run without these, the test prints a structured SKIP report and exits 0; the smoke variant is therefore safe to run on any developer laptop.

Topology timeline

t=0            bootstrap: 3-node single-DC cluster.
t=0..600s      grow to 9 nodes across 3 DCs (3+3+3 racks).
t=600..3000s   steady state.
t=3000..3600s  shrink to 3 nodes single-DC.

Plus mid-run nodes joining and leaving on a Poisson cadence (mean inter-arrival 90 s) so every transition in dyn_state_t (INIT, STANDBY, WRITES_ONLY, RESUMING, NORMAL, JOINING, DOWN, RESET, UNKNOWN) is observed at least once.

Workload mix

• 50 / 50 read / write steady state.
• 60-second 90%-write spike windows every 10 minutes.
• Three concurrent client populations: interactive (P99 budget 5 ms), batch (P99 5 s), background (no latency budget).

Invariants asserted

Any violation aborts the run and writes a non-zero exit status; otherwise the report ends with STATUS: PASS.

Cleanup discipline

The test installs an on_drop sweep that:

1. Sends SIGTERM then SIGKILL to every spawned dynomited instance.
2. Sends SIGTERM to every spawned valkey-server.
3. Removes every tc qdisc injected on the loopback device.
4. Unsets FAKETIME for every spawned process.
5. Captures a final tc qdisc show dev lo and asserts it matches the pre-test baseline.

If any sweep step fails the test's exit code reflects the sweep failure, not the workload outcome, so the harness can distinguish between "workload failed" and "host left dirty".

Report artefact

The run produces target/chaos/<run-id>/report.md containing:

• Start / end times and total wall clock.
• Topology timeline event log.
• Per-injector firing counts.
• dyn_state_t per-state hit counter.
• Per-window throughput and tail-latency histograms (interactive / batch / background).
• Any invariant violations (empty on a clean run).

The report is the artefact the v0.1.0 release tag references. The lead curates the production-mode run output into dist/chaos-reports/v0.1.0/report.md before the signed tag.

Release process

This page documents the procedure the project lead executes to cut a tagged release. Workers do not run any of these steps; they are the manual final-step closure that follows Stage 16 review approval.

Pre-flight checklist

Run these before touching git history. Every item must be green.

nix develop
git status                                  # clean tree
scripts/check.sh                            # full local CI gate
scripts/check_clean.sh                      # cleanup-sweep
cargo public-api --diff-git-checkouts main  # no SemVer drift
mdbook build docs/book                      # docs render clean

Then the chaos test (the gate the C engine never had):

sudo -E env "PATH=$PATH" \
    cargo nextest run \
        --release --workspace \
        --features chaos \
        --test stage_16_chaos \
        --no-capture

The 1-hour run must report STATUS: PASS with zero invariant violations. Curate its target/chaos/<run-id>/report.md into dist/chaos-reports/v0.1.0/report.md and commit:

mkdir -p dist/chaos-reports/v0.1.0
cp target/chaos/<run-id>/report.md dist/chaos-reports/v0.1.0/
git add dist/chaos-reports/v0.1.0/report.md
git commit -s -m "release(chaos): v0.1.0 chaos run report"

Author history rewrite

AGENTS.md Sections 14 / 14a require every commit on main to be authored by Greg Burd <greg@burd.me>. The rewrite is the last commit-mutating operation before the tag.

# 1. Snapshot the pre-rewrite SHAs for archaeology.
git log --format='%H %an <%ae>' main \
    > docs/journal/pre-rewrite-shas.md
git add docs/journal/pre-rewrite-shas.md
git commit -s -m "docs(journal): archive pre-rewrite SHAs"

# 2. Rewrite author + committer across the entire branch.
#    git-filter-repo is the supported tool; --force is needed
#    because we are operating in-place.
git filter-repo --force --commit-callback '
    commit.author_name = b"Greg Burd"
    commit.author_email = b"greg@burd.me"
    commit.committer_name = b"Greg Burd"
    commit.committer_email = b"greg@burd.me"
'

# 3. Verify a single line of authorship history.
git log --format='%an <%ae>' | sort -u
# expected: Greg Burd <greg@burd.me>

After the rewrite, no further commits land on main until the tag is pushed.

Signed tag

git tag -a -s v0.1.0 -m "dynomite Rust port v0.1.0

First production tag of the Rust dynomite engine.
See CHANGELOG.md for the per-stage summary."

git tag --verify v0.1.0

Push to GitHub and Forgejo

The project ships dual-platform CI (AGENTS.md Section 14b); the tag must reach both forges before the release is considered shipped.

git push origin main
git push origin v0.1.0

# Codeberg / Forgejo mirror: the remote is named `forgejo`
# in the lead's git config.
git push forgejo main
git push forgejo v0.1.0

Verify both runners pass on the tag SHA:

• https://github.com/gburd/dynomite/actions
• https://codeberg.org/gregburd/dynomite/actions

Container image

After the tag is pushed, build and tag the Docker image:

docker build -f dist/docker/Dockerfile \
    -t dynomite:0.1.0 \
    -t dynomite:latest \
    .

Push the tagged images to the registry of choice; the project does not prescribe one.

Post-release

• Open [Unreleased] section in CHANGELOG.md for the next iteration.
• Bump workspace.package.version in Cargo.toml to 0.2.0-dev (or the next planned minor).
• Open the Stage-after-16 follow-up tickets enumerated in docs/coverage-deviations.md and docs/journal/blocked.md.

Reading the Examples

The programs under crates/*/examples/ are complete, compilable, runnable code -- not the illustrative snippets scattered through the rest of this manual. Each one is a focused study of one facet of the engine, and each has a page here that explains what it demonstrates, the design decisions it embodies, the trade-offs behind those decisions, and when you would reach for the pattern it shows.

cargo run -p <crate> --example <name>

The learning path

The examples are ordered here from smallest to most involved. If you are new to embedding Dynomite, read them in this order; each builds on the call chain the previous one introduced.

flowchart TD
  A[embedded_minimal<br/>the 5-call chain] --> B[embedded_single_node<br/>a real backend + config]
  B --> C[embedded_cluster3<br/>three nodes, a custom Datastore]
  C --> D[random_slicing<br/>distribution internals]
  A --> E[custom_transport_sketch<br/>the shape of a transport plug-in]
  F[demo_vector_text<br/>search over the engine] --> G[vec quickstart<br/>the vector store alone]

The example programs, arranged from the smallest runnable engine to the distribution internals and the search stack.

Index

embedded_minimal

Source: crates/dynomite/examples/embedded_minimal.rs -- run with cargo run -p dynomite --example embedded_minimal

What it demonstrates

The entire lifecycle of an embedded engine in the fewest possible calls: name a pool, bind two listeners, start, and shut down.

use dynomite::embed::{Server, ServerBuilder};

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let handle = Server::start_with(
        ServerBuilder::new("dyn_o_mite")
            .listen("127.0.0.1:0".parse()?)
            .dyn_listen("127.0.0.1:0".parse()?),
    )
    .await?;

    eprintln!(
        "embedded dynomite up; client listen={:?} dnode listen={:?}",
        handle.listen_addr(),
        handle.dyn_listen_addr()
    );

    handle.shutdown().await?;
    Ok(())
}

Two listeners are the irreducible minimum: listen is the client-facing port (where a Redis or Memcache client connects), and dyn_listen is the DNODE peer plane (where other nodes connect). Binding to port 0 lets the OS pick a free port, which you then read back with listen_addr() and dyn_listen_addr() -- handy in tests.

Server::start_with is a convenience that builds and starts in one call. The longer form -- ServerBuilder::...build()? then server.start().await? -- is what embedded_single_node uses, and it is what you want when you need the Server value before it starts (for example, to subscribe to its event stream).

Design decisions and trade-offs

In-memory default backend

With no datastore hook and no servers, the engine uses the in-crate MemoryDatastore. That makes the example self-contained -- no Valkey to start -- at the cost of realism. The moment you want to front a real store you add one call; see the next example.

Gossip off by omission

With no peers configured there is nothing to gossip with, so the example is effectively a single node. This keeps the output deterministic and the startup instant.

current_thread runtime

A single-threaded tokio runtime is enough for one node with no concurrent peer traffic and keeps the example's resource use trivial. Production embeddings use the multi-thread flavor.

When to use this pattern

As a smoke test that the engine links and starts in your process, and as the skeleton you paste and then grow. It is not a useful deployment on its own -- it stores nothing durable and talks to no peers.

Where to go next

• embedded_single_node adds a real Valkey backend and spells out the configuration.
• Your First Embedded Engine walks this chain call by call.

embedded_single_node

Source: crates/dynomite/examples/embedded_single_node.rs -- run with cargo run -p dynomite --example embedded_single_node (requires a Valkey/Redis on 127.0.0.1:6379)

What it demonstrates

The full ServerBuilder chain for a realistic single node: a named pool, both listeners, a Valkey backend, datacenter and rack identity, a token, a request timeout, and gossip explicitly disabled.

#![allow(unused)]
fn main() {
use std::time::Duration;
use dynomite::conf::{ConfServer, DataStore};
use dynomite::embed::{Server, ServerBuilder};

let server: Server = ServerBuilder::new("dyn_o_mite")
    .listen("127.0.0.1:18102".parse()?)
    .dyn_listen("127.0.0.1:18101".parse()?)
    .data_store(DataStore::Valkey)
    .servers(vec![ConfServer::parse("127.0.0.1:6379:1 backend")?])
    .datacenter("dc-local")
    .rack("rack-local")
    .tokens_str("0")
    .timeout(Duration::from_secs(5))
    .enable_gossip(false)
    .build()?;

let handle = server.start().await?;
}

Note the two-step build/start here, in contrast to embedded_minimal's start_with. The Server value exists before it starts, which is the seam you use to wire up observability.

The example then reads a stats snapshot and shuts down, rather than blocking on Ctrl-C, so it terminates cleanly under cargo run:

#![allow(unused)]
fn main() {
let snap = handle.stats();
eprintln!("snapshot pool={} uptime={}s", snap.pool.name, snap.uptime);
handle.shutdown().await?;
}

The configuration, knob by knob

data_store(DataStore::Valkey)

Selects the RESP/Valkey backend protocol. See Redis / Valkey.

servers(["127.0.0.1:6379:1 backend"])

The backing store endpoint and weight; parsed by ConfServer::parse. The trailing name is a label.

datacenter / rack

This node's placement in the topology. Even a single node has a DC and rack because the consistency and replication logic is defined in those terms; see Replication and Consistency.

tokens_str("0")

The token this node owns on the ring. A lone node owns the whole ring, so any single token works; see The Ring.

enable_gossip(false)

There are no peers, so gossip has nothing to do. Turning it off keeps startup instant and the log quiet.

Design decisions and trade-offs

Fronting an existing Valkey (rather than the in-memory default) is the point of this example: it shows that the engine's job is orchestration, not storage. The backing store stays a plain Valkey that knows nothing about the ring.

When to use this pattern

As the starting point for any single-node embedding that fronts a real store, and as the template you extend by adding peers (see embedded_cluster3) and hooks (see Hooks and Traits).

Where to go next

• embedded_cluster3 turns this into three cooperating nodes with a custom Datastore.
• Configuration is the full reference for every knob shown here and many more.

embedded_cluster3

Source: crates/dynomite/examples/embedded_cluster3.rs -- run with cargo run -p dynomite --example embedded_cluster3

What it demonstrates

Three things at once: building more than one node in a process, supplying your own Datastore hook, and driving requests without a socket via inject_request.

The custom datastore is a shared in-memory map that all three nodes write through:

#![allow(unused)]
fn main() {
#[derive(Default, Clone)]
struct SharedKv {
    inner: Arc<Mutex<std::collections::HashMap<u64, MsgType>>>,
}

impl Datastore for SharedKv {
    fn protocol(&self) -> Protocol { Protocol::Custom }

    fn dispatch(&self, req: Msg) -> BoxFuture<'_, Result<Msg, DatastoreError>> {
        let inner = self.inner.clone();
        Box::pin(async move {
            let mut g = inner.lock();
            if matches!(req.ty(), MsgType::ReqRedisSet) {
                g.insert(req.id(), MsgType::RspRedisStatus);
            }
            let stored = g.get(&req.id()).copied();
            drop(g);
            let mut rsp = Msg::new(req.id(), stored.unwrap_or(MsgType::RspRedisStatus), false);
            rsp.set_parent_id(req.id());
            Ok(rsp)
        })
    }
}
}

Each node is built with a distinct rack and token so the three of them tile the ring, then a write is injected at node 0 and reads at nodes 1 and 2:

#![allow(unused)]
fn main() {
let mut w = Msg::new(7, MsgType::ReqRedisSet, true);
w.set_parent_id(0);
let _ = n0.inject_request(w).await?;

for (label, h) in [("n1", &n1), ("n2", &n2)] {
    let req = Msg::new(7, MsgType::ReqRedisGet, true);
    let rsp = h.inject_request(req).await?;
    eprintln!("{label}: rsp ty={:?} parent={}", rsp.ty(), rsp.parent_id());
}
}

Design decisions and trade-offs

Shared in-memory Datastore

One SharedKv behind all three nodes lets the example show routing and response flow without three real backends. It is a test double, not a replication story -- the nodes are not actually replicating to independent stores.

inject_request over sockets

Requests go straight into the engine, skipping the client listener. That makes the example deterministic and fast and is exactly how you would unit-test a hook. Real clients still connect over the socket.

Gossip disabled, seeds empty

The topology is wired statically by giving each node its rack and token directly. This keeps the example reproducible; a live cluster would enable gossip and share seeds.

multi_thread runtime

Three nodes with concurrent injected traffic want real parallelism, unlike the single-node examples.

When to use this pattern

When you are implementing a Datastore, MetricsSink, or other hook and want to test it under multi-node routing in a single fast, deterministic process.

Where to go next

• Hooks and Traits for the full trait surface the SharedKv here implements.
• Your First Cluster for a real multi-process cluster.

random_slicing

Source: crates/dynomite/examples/random_slicing.rs -- run with cargo run -p dynomite --example random_slicing

What it demonstrates

How the dispatcher turns a key into an owning peer, and how the random-slicing distribution mode spreads keys across peers. It builds the ring by hand -- four peers, one rack, one datacenter -- and then asks the dispatcher to plan 10,000 requests, counting where each lands.

#![allow(unused)]
fn main() {
let cfg = PoolConfig {
    dc: "dc1".into(),
    rack: "r1".into(),
    hash: HashType::Murmur3X64_64,
    distribution: Distribution::RandomSlicing,
    ..PoolConfig::default()
};
// ... build four Peers, mark them Normal, rebuild the ring ...
let disp = ClusterDispatcher::new(pool);

for i in 0..10_000 {
    let req = Msg::new(i as u64, MsgType::ReqRedisGet, true);
    let key = format!("key-{i:08x}");
    match disp.plan(&req, key.as_bytes()) {
        DispatchPlan::Replicas { targets, .. } => counts[targets[0].peer_idx as usize] += 1,
        DispatchPlan::LocalDatastore => counts[0] += 1,
        DispatchPlan::NoTargets | DispatchPlan::Drop => {}
    }
}
}

The output is a histogram of ownership -- the same information the operator command dyn-admin distribution-dump reports for a live cluster.

Design decisions and trade-offs

Hand-built pool, no server

This example constructs ServerPool and ClusterDispatcher directly rather than through ServerBuilder. It is studying the routing layer in isolation, so it skips listeners, gossip, and backends entirely.

DispatchPlan exposed

Matching on DispatchPlan::{Replicas, LocalDatastore, NoTargets, Drop} shows the four outcomes the planner can produce for a key. Only the first replica is counted here, which measures primary ownership.

10,000 synthetic keys

Enough to see the distribution shape without being slow. A perfectly even split would be 2,500 per peer; the spread from that is the point of comparing distribution modes.

When to use this pattern

When you want to reason about or test routing and ownership without standing up a cluster: comparing distribution modes, checking that a token assignment balances, or unit-testing changes to the dispatcher.

Where to go next

• The Ring and the Token Space explains the token math this example exercises.
• Distribution Modes compares random slicing against the alternatives.

embedded_custom_transport_sketch

Source: crates/dynomite/examples/embedded_custom_transport_sketch.rs -- run with cargo run -p dynomite --example embedded_custom_transport_sketch

What it demonstrates

Two things, and it is careful about what it does not claim:

1. How to build an AsyncRead + AsyncWrite + Send + Unpin type -- here, over a tokio::io::DuplexStream -- which is the same pattern you use for a TLS stream, a Unix-domain socket, or a QUIC stream.
2. How that type implements Transport by carrying a ConnRole tag and reporting a peer_addr.

#![allow(unused)]
fn main() {
use dynomite::embed::{ConnRole, Transport};

pub struct PipeTransport {
    inner: tokio::io::DuplexStream,
    role: ConnRole,
    peer_addr: SocketAddr,
}
// impl AsyncRead / AsyncWrite by delegating to `inner`;
// impl Transport by returning `role` and `peer_addr`.
}

Why it is only a sketch

This honesty is itself a documentation decision: the example teaches the trait contract without pretending a capability exists. It compiles and runs (it exercises the PipeTransport end to end over the duplex stream), so the trait implementation is real -- only the listener integration is pending.

Design decisions and trade-offs

DuplexStream as the byte source

An in-memory pipe needs no OS resources and makes the example hermetic. The same AsyncRead + AsyncWrite bound is what a real socket, TLS session, or QUIC stream satisfies, so the pattern transfers unchanged.

ConnRole tag on the transport

A transport must say whether it carries client or peer traffic; that tag is how the engine routes a connection to the right handler.

When to use this pattern

When you need Dynomite to accept connections over something other than TCP or QUIC -- an in-process pipe for testing, a Unix socket, or a bespoke secure channel -- and you want to see the trait you will implement.

Where to go next

• Transports documents the built-in TCP and QUIC transports that implement this same trait.
• Hooks and Traits covers the other extension points.

demo_vector_text

Source: crates/dynomite-search/examples/demo_vector_text.rs -- run with cargo run -p dynomite --example demo_vector_text

What it demonstrates

The library side of the RediSearch-compatible FT.* surface: creating an index, inserting documents, and running vector-similarity, text, and combined searches. It renders each request and its RESP reply so you can see the wire shapes even though the calls go through the registry rather than a socket.

#![allow(unused)]
fn main() {
use dynomite_search::ft::{self, FtOutcome, InfoValue};
use dynomite_search::registry::VectorRegistry;
use dyntext::index::TextIndex;

// FT.CREATE myidx ON HASH PREFIX 1 docs: SCHEMA ...
//   a 4-dimensional cosine-similarity HNSW vector field, plus text fields
// FT.ADD / HSET the documents
// FT.SEARCH with a KNN vector query, a text query, and a combination
}

Vectors are converted to little-endian bytes -- the exact format Redis Stack clients send for VECTOR fields -- so the example doubles as documentation of that encoding:

#![allow(unused)]
fn main() {
fn f32_to_le_bytes(values: &[f32]) -> Vec<u8> {
    let mut out = Vec::with_capacity(values.len() * 4);
    for v in values { out.extend_from_slice(&v.to_le_bytes()); }
    out
}
}

Design decisions and trade-offs

Library, not wire

The example calls the FT.* registry directly. This is the same API dynomited's dispatcher uses once a request is parsed, so it exercises the real search engine while staying independent of the network stack.

HNSW for vectors

The vector field uses an HNSW index with cosine similarity. HNSW trades a little index-build cost and memory for fast approximate nearest-neighbor search at query time.

Trigram text index

Text search is trigram-based, which supports substring and fuzzy matching without a full inverted-index-per-term structure.

When to use this pattern

When you want to understand or test the FT.* semantics -- index creation, the vector byte encoding, query construction -- without a running server. It is the fastest way to see what a given FT.SEARCH returns.

Where to go next

• Tutorial: Vector, Text, and Regex Search does the same thing over valkey-cli against a running dynomited.
• Full-Text, Vector, and Regex Search is the reference for the FT.* surface.
• quickstart drops to the vector store alone.

quickstart (dynvec)

Source: crates/dynomite-vec/examples/quickstart.rs -- run with cargo run -p dynvec --example quickstart --features http

What it demonstrates

dynvec, the vector engine that backs Dynomite's VECTOR search fields, used directly and standalone. It creates an in-memory store with one table, seeds a handful of labeled vectors, and serves an HTTP API you can query.

#![allow(unused)]
fn main() {
use dynvec::api::serve;
use dynvec::distance::Distance;
use dynvec::encoding::Codec;
use dynvec::index::HnswParams;
use dynvec::storage::{IndexAlgorithm, TableSchema, VectorStore};

let store = Arc::new(VectorStore::in_memory());
store.create_table(TableSchema {
    name: "demo".to_string(),
    dim: 3,
    codec: Codec::Int8Quantized,
    distance: Distance::Cosine,
    hnsw: HnswParams::default(),
    algorithm: IndexAlgorithm::Hnsw,
})?;
}

It listens on 127.0.0.1:21900 by default (override with DYNVEC_LISTEN).

Design decisions and trade-offs

In-memory store

VectorStore::in_memory() keeps the example dependency-free. The same API backs a persistent store when you want durability.

Int8 quantized codec

The table uses Codec::Int8Quantized, which shrinks each vector roughly four-fold versus raw f32 at a small recall cost. Choosing the codec at table-creation time is a deliberate space/accuracy trade-off exposed to the caller.

Cosine distance + HNSW

Cosine similarity over an HNSW index -- the standard combination for semantic-similarity search. The Distance and IndexAlgorithm are per-table so different tables can make different choices.

HTTP surface

The http feature exposes a REST API, which is why the run command passes --features http. Without it, the store is a library only.

When to use this pattern

When you want the vector engine without the rest of Dynomite -- to prototype an embedding search, to benchmark a codec/distance combination, or to understand what the VECTOR fields in demo_vector_text sit on top of.

Where to go next

• demo_vector_text shows the same vector engine reached through the FT.* search surface.
• Full-Text, Vector, and Regex Search is the reference for search across Dynomite.

Index

This page is populated by the stage that ports the corresponding subsystem (see PLAN.md). Until then it intentionally stays brief so the manual structure is visible.

Design Decisions (Roads Not Taken)

Dynomite is opinionated, and its opinions are inherited from the Amazon Dynamo paper and refined by two decades of production experience with Netflix Dynomite and Basho Riak. This page is the honest ledger of the trade-offs. None of these choices is universally correct; each is correct for the problem Dynomite solves -- adding availability and cross- datacenter replication to storage engines that lack it.

Distribution as a layer, not a rewrite

Chosen: wrap an unmodified single-node store (Valkey, Memcached, Noxu) with a separate distribution layer.

Not chosen: fork the storage engine and build replication into it.

A store that knows nothing about the ring stays simple, stays fast, and stays independently upgradable. The distribution logic lives in one place and works the same regardless of backend. The cost is a network hop for non-local keys and the fact that the layer cannot exploit engine internals (it cannot, say, replicate at the storage-page level). For the goal -- portable HA across heterogeneous backends -- the separation wins.

Tunable quorum, not consensus

Chosen: the Dynamo model -- eventual consistency with per-request tunable quorums (DC_ONE, DC_QUORUM, and friends), plus read repair and anti-entropy to converge divergent replicas.

Not chosen: a consensus protocol (Raft, Paxos, or multi-Paxos) giving linearizable single-key writes.

Consensus buys linearizability at the price of availability under partition (a minority partition cannot make progress) and of a latency floor set by the slowest quorum member on every write. Dynomite's users chose availability. See Replication and Consistency.

Cross-node transactions: 2PC, not Paxos-commit

Chosen (Dyniak): two-phase commit (XA) over Noxu's transactional engine for cross-node multi-key atomic updates, with a RAMP-style path for read-atomic multi-key reads.

Not chosen: a Paxos-based atomic commit or a full distributed SQL transaction manager.

2PC is simple, well understood, and sufficient for the bounded, short-lived multi-key updates Dyniak targets. Its classic weakness -- a coordinator crash can block participants -- is bounded here by timeouts and the fact that participants are Dynomite peers already under failure detection. A Paxos-commit would remove the blocking window at the cost of substantially more machinery and latency; for the workload it was not worth it. See Distributed Transactions.

Conflict resolution: CRDTs and vector clocks, not last-write-wins-only

Chosen (Dyniak): convergent replicated data types (counters, sets, maps, registers) for conflict-free merges, plus causal context for sibling-aware writes.

Not chosen: last-write-wins on a wall-clock timestamp as the only resolution strategy.

Last-write-wins is available and is the right default for opaque values, but it silently discards concurrent updates. CRDTs let a counter that was incremented on two partitioned nodes converge to the correct sum instead of losing one increment. The cost is that CRDT values carry more metadata and the client must use the typed API. See Convergent Data Types.

Gossip membership, not a central coordinator

Chosen: decentralized gossip for membership, topology discovery, and failure detection.

Not chosen: a central coordinator or an external service (ZooKeeper, etcd) holding the cluster's membership.

A coordinator is a single point of failure and an operational dependency. Gossip has no such center: any node can join by contacting a seed, and state propagates epidemically. The trade-off is that membership is eventually consistent -- a just-joined node is not instantly visible everywhere -- and gossip uses steady background bandwidth. See Membership and Gossip.

phi-accrual failure detection, not a fixed timeout

Chosen: the phi-accrual detector, which outputs a continuous suspicion level adapting to observed network variance.

Not chosen: a fixed heartbeat timeout ("declare dead after N missed beats").

A fixed timeout forces a bad choice: short enough to detect failures quickly means false positives on a jittery link; long enough to avoid false positives means slow detection. phi-accrual adapts its threshold to the link's actual behavior. See Failure Handling.

Consistent hashing, with pluggable distribution

Chosen: a token ring as the default, with alternative distribution modes (including random slicing) selectable per pool.

Not chosen: hard-coding one hashing scheme.

Consistent hashing minimizes the keys that move when the ring changes, which is the property that matters most for a cache or store front. But it can leave ownership uneven with few nodes; random slicing rebalances more evenly at the cost of strict monotonicity. Rather than pick one, Dynomite makes the strategy a configuration choice and ships the random_slicing example so you can measure the difference. See Distribution Modes.

Property, DST, and Elle testing, not unit tests alone

Chosen: for anything on the distributed path, a deterministic simulation model (with a negative control that has teeth) plus an Elle-style consistency check on recorded histories, in addition to unit, property, and fuzz tests.

Not chosen: treating a green unit suite as sufficient for distributed correctness.

Several real distributed defects in this port passed the entire unit suite and were only caught at scale or by a model. The project codifies model-first reproduction of any distributed failure as a merge gate. This is a testing philosophy, but it is also a design constraint: a change that cannot be expressed in a deterministic model is, by that standard, not ready. See Chaos Test and the conformance and coverage chapters.

Rust, forbid(unsafe_code), not a C fork

Chosen: a from-scratch Rust reimplementation with unsafe forbidden by default.

Not chosen: maintaining a fork of the original C.

Memory safety without a garbage collector, a strong type system for the protocol and state machines, and a modern async runtime were judged worth the cost of a full rewrite and the discipline of maintaining parity with the C reference. Parity is tracked symbol by symbol in docs/parity.md.

Glossary

Terms used throughout this manual. Where a term has a chapter of its own, the definition links to it.

Active anti-entropy (AAE). A background process that compares replicas -- using Merkle trees so the comparison is cheap -- and repairs any divergence it finds. See Anti-Entropy and Repair.

Backend. The single-node storage engine a Dynomite node fronts: Valkey, Memcached, or the embedded Noxu store. The backend knows nothing about the ring.

Consistency level. The per-request quorum policy: DC_ONE, DC_QUORUM, DC_SAFE_QUORUM, DC_EACH_SAFE_QUORUM. It decides how many replicas must respond before a request succeeds and how reads pick a replica. See Replication and Consistency.

Coordinator. For a given request, the node that received it from the client and is responsible for routing it to replicas and coalescing the responses. Any node can coordinate any request.

CRDT (Convergent Replicated Data Type). A data type -- counter, set, map, register -- whose replicas merge deterministically without conflict. Dyniak feature. See Convergent Data Types.

Datacenter (DC). The top level of the topology. Replication and the consistency levels are defined across and within datacenters.

DNODE. The peer plane: the wire protocol Dynomite nodes use to talk to each other, distinct from the client-facing protocol. See DNODE.

Dyniak. The Riak-compatible layer built on the engine and backed by Noxu. See Introduction to Dyniak.

Gossip. The decentralized protocol nodes use to discover each other, share topology, and detect failures. See Membership and Gossip.

Hinted handoff. When a write's target peer is temporarily unavailable, a coordinator stores a durable "hint" and replays it when the peer returns, so the write is not lost. See Failure Handling.

Node. One dynomited process (or one embedded Server), owning one or more tokens and fronting one backend.

Noxu. The transactional Rust storage engine that backs Dyniak; a re-implementation of Berkeley DB Java Edition. Provides the XA support Dyniak's transactions use.

phi-accrual. The adaptive failure detector Dynomite uses: it outputs a continuous suspicion value rather than a binary alive/dead, adapting to observed network variance. See Failure Handling.

Quorum. A majority of the relevant replicas. For a local-DC quorum of n replicas the count is n/2 + 1.

Rack. A subdivision of a datacenter. In Dynomite's replication model a rack holds a full copy of the ring, so the number of racks in a DC is the replication factor within that DC.

Read repair. When a read observes divergent replicas, the coordinator writes the reconciled value back to the stale ones. See Failure Handling.

Replica. A node (in Dynomite's model, a rack) that holds a copy of a given key.

Ring / token ring. The circular token space over which keys are partitioned by consistent hashing. See The Ring and the Token Space.

Token. A point on the ring. A key hashes to a token; the node owning the next token clockwise owns the key.

Vnode (virtual node). A token owned by a physical node; a physical node may own several, spreading its ownership around the ring.

XA / 2PC. Two-phase commit, the protocol Dyniak uses for cross-node multi-key atomic transactions. See Distributed Transactions.

Manual Pages

Dynomite ships Unix manual pages for its executables. They are the authoritative reference for command-line flags and are installed alongside the binaries by the packaging in dist/.

dynomited(8)

The server daemon. Source: crates/dynomited/man/dynomited.8.

View it locally from a checkout:

man crates/dynomited/man/dynomited.8
# or, after install:
man 8 dynomited

It documents every flag (--conf-file, --test-conf, and the rest), the data_store backends, and the signal handling. The page is generated from the binary's own argument parser -- run cargo run -p dynomited --bin gen-man to regenerate it after a flag change, which keeps the man page and the --help output from drifting.

dyn-admin

The cluster administration CLI. It is self-documenting via --help on each subcommand:

dyn-admin --help
dyn-admin ring --help
dyn-admin distribution-dump --help

See Admin CLI (dyn-admin) for a task-oriented walk-through of the subcommands.

dyn-hash-tool

A small utility for computing the hashes and tokens Dynomite uses, useful when reasoning about key placement. See its --help:

dyn-hash-tool --help

Regenerating man pages

The dynomited.8 page is generated, not hand-maintained. The generator lives at crates/dynomited/src/bin/gen-man.rs and writes to crates/dynomited/man/dynomited.8:

cargo run -p dynomited --bin gen-man

Run it in the same commit as any change to dynomited's flags, so the shipped page always matches the binary.

Contributing and the Parity Discipline

This manual is for users of Dynomite. Contributors -- human or automated -- have two additional documents that govern how the project is built:

• AGENTS.md -- the operating manual: coding standards, the testing strategy, the git workflow, and the review discipline.
• PLAN.md -- the staged roadmap.

This page summarizes the two ideas a user-turned-contributor most needs to understand.

Parity with the C original

Dynomite (Rust) aims to be functionally identical to Netflix Dynomite (C). That aim is not aspirational prose -- it is tracked, symbol by symbol, in docs/parity.md, which maps each C function to its Rust home and records every deliberate divergence as a deviation with a rationale.

If you change behavior, you update docs/parity.md. If you find the C reference ambiguous, you record the ambiguity, choose the interpretation most consistent with the surrounding code, and pin it with a regression test.

The testing discipline for distributed changes

Anything that touches the distributed path -- routing, replication, gossip, quorum, anti-entropy, causality, cross-node transactions, CRDT convergence, or the peer plane -- must pass two gates beyond the ordinary unit, property, and fuzz tests, and both are merge blockers:

1. A deterministic simulation model (no real sockets, no wall clock, seeded RNG) that asserts the safety and liveness invariants the change must never violate, and that includes a negative control -- a deliberately broken variant the checker is shown to catch, so the model is proven to have teeth.

2. An Elle-style consistency check that drives the real built binary with a history-recording workload and finds zero anomalies of the classes it covers.

The everyday gate

Every commit must pass scripts/check.sh, which is what CI runs on both GitHub Actions and Codeberg Forgejo Actions. It covers formatting, clippy (pedantic, warnings denied), the full test matrix, doctests, the documentation link checker, dependency audits, the mdBook build, and the project hygiene checks (no todo!(), ASCII only, no port-acknowledgement comments). Run it before you propose a change.

Getting the environment

Everything runs inside the pinned Nix dev shell:

nix develop
scripts/check.sh

The flake pins the Rust toolchain and every auxiliary tool -- including the mdBook, mermaid, admonish, linkcheck, and lychee binaries this manual is built with -- so a fresh checkout reproduces CI exactly.