Failure Handling

This chapter walks the phi-accrual failure detector and the peer state machine, auto-eject and auto-rejoin, durable hinted handoff, read repair, the anti-entropy overview, and the guarantees that hold across a partition.

Detecting failure: phi-accrual

A naive detector flips a peer between "alive" and "dead" on a missed-heartbeat count. That is brittle: a network hiccup trips it, and a genuinely slow link never does. Dynomite uses a phi-accrual detector (PhiAccrual), the same family Cassandra, Akka, and Riak deploy, which produces a continuous suspicion level phi(t) instead of a boolean.

Phi is the negative log-probability that a heartbeat would not have arrived yet given the historical inter-arrival distribution. Modelling arrivals as exponential gives the closed form the detector computes:

    phi(t) = elapsed_ms / (mean_interval_ms * ln(10))

phi = 1.0

~10% chance the heartbeat is merely late.

phi = 2.0

~1% chance.

phi = 8.0

~10^-8 -- "almost certainly dead". This is the default threshold (DEFAULT_THRESHOLD), matching Cassandra's phi_convict_threshold.

The detector keeps a sliding window of the last ~100 inter-arrival times per peer, so it adapts to each link's real cadence. A jittery link that normally varies wildly will not be convicted by a one-second gap that would convict a metronome-steady link -- higher observed variance widens the tolerated gap. Two guards keep the math honest: the mean interval is clamped at a floor (default 1s) so burst arrivals cannot make phi spike, and phi is 0.0 when no heartbeat has ever been recorded (no data is not the same as dead) or when fewer than two heartbeats give no inter-arrival sample.

The peer state machine

Each peer carries a PeerState. Only Normal and Joining are routable; Down and Leaving are not. A remote peer starts Down and is promoted only after its first below-threshold heartbeat; the local node starts Joining.

stateDiagram-v2
  [*] --> Down: remote peer created
  [*] --> Joining: local node created
  Joining --> Normal: bootstrap complete,<br/>heartbeats flowing
  Down --> Normal: first heartbeat,<br/>phi <= threshold
  Normal --> Down: phi > threshold<br/>(silence) or announced departure
  Down --> Normal: heartbeats resume,<br/>phi <= threshold
  Normal --> Leaving: graceful departure
  Leaving --> [*]

Peer lifecycle. The routable states are Normal and Joining; the failure detector drives the Normal/Down toggle, and a graceful departure moves a peer to Leaving so it stops receiving traffic before it goes.

The gossip handler is the single owner of these transitions once gossip is wired. On each inbound heartbeat it records the arrival and, if phi is below threshold, promotes the peer to Normal at once. On each periodic tick it evaluates every non-local peer and toggles Normal / Down from the current phi. A peer that flaps -- goes silent, is convicted, then resumes -- produces exactly one Normal -> Down and one Down -> Normal transition per cycle, each surfaced as a metric and a structured event.

sequenceDiagram
  participant P as peer
  participant FD as phi detector
  participant GH as gossip handler
  loop steady state
    P->>FD: heartbeat (1 Hz)
    Note over FD: phi stays < 1.0
  end
  Note over P: peer goes silent
  GH->>FD: tick: phi(now)?
  FD-->>GH: phi = 12 (> 8)
  GH->>GH: mark peer Down, emit PeerDown
  Note over P: peer recovers
  P->>FD: heartbeat resumes
  GH->>GH: phi <= 8, promote Normal, emit PeerUp

Silence raises phi past the threshold on the next tick and convicts the peer; a resumed heartbeat promotes it back. The detector's window means recovery is judged on the same adaptive baseline as conviction.

Auto-eject and auto-rejoin

Eviction and readmission are gossip-driven, not operator-driven.

• Auto-eject. When a peer crosses the phi threshold (or announces its own departure), the handler marks it Down. The next continuum rebuild drops it from the routable set, and the dispatcher stops planning requests to it. No manual intervention, no config edit.
• Auto-rejoin. When a Down peer starts gossiping again and its phi falls below threshold, the handler promotes it back to Normal on the next heartbeat, and it re-enters the routable set on the next rebuild. Its failure detector is reset on re-add so old jitter does not bias the fresh baseline.

The whole loop is closed by gossip and the detector; the operator's role is to fix the underlying fault, not to click a node back in.

Hinted handoff

A write whose target replica is Down would normally just miss that replica. Hinted handoff makes the write durable anyway: the coordinator records a hint -- the on-the-wire request bytes, the intended peer index, and an expiry deadline -- and a background drainer ships it to the peer once the peer returns to Normal.

sequenceDiagram
  participant Cl as client
  participant Co as coordinator
  participant R1 as replica r1 (up)
  participant R2 as replica r2 (DOWN)
  participant HS as hint store
  Cl->>Co: SET k v (DC_QUORUM)
  Co->>R1: forward SET
  Co->>HS: record hint for r2
  Note over Co: synthetic +OK on r2's behalf<br/>counts toward quorum
  R1-->>Co: +OK
  Co-->>Cl: +OK
  Note over R2: r2 recovers -> Normal
  HS->>R2: drainer ships hinted SET
  R2-->>HS: +OK, hint cleared

Hinted handoff keeps a write durable across a replica outage. The hint counts toward the consistency threshold at write time and is replayed to the replica when it returns, so the down replica catches up without a full repair.

Handoff is only active when the hint store is wired and the pool sets enable_hinted_handoff. When active, Down write targets are kept in the routable set so the dispatcher can hint them; a synthetic +OK is fed to the coalescer on the hinted target's behalf so the surviving replicas plus the hint can meet the consistency threshold. Without handoff, a Down target is simply skipped.

Durability of hints

The hint store (HintStore) has two modes:

RAM-only (HintStore::new)

Hints live only in per-peer in-memory queues and are lost if the coordinator restarts.

Durable (HintStore::open, with hint_dir)

One append-only segment file per peer under <dir>/peer-<idx>.hints. Each record is framed with a length, an IEEE CRC-32 of the body, a wall-clock deadline, and the payload. Hints survive a coordinator restart -- replay re-anchors each deadline to the current clock and drops any already-expired hint.

The durable format is torn-tail safe: a crash mid-append leaves at most a trailing partial record, and replay detects it two ways -- a short read before the body completes, or a body whose CRC does not match -- and stops cleanly at the first damaged record, keeping every intact record before it. A torn tail never panics and never surfaces an error from open.

Hints are bounded by max_bytes and expire after hint_ttl_seconds (default one day). Over-capacity or zero-TTL enqueues are rejected so the store cannot grow without bound; when the store is full the coordinator falls back to its no-quorum error path rather than silently dropping the write.

Read repair

Read repair heals divergence that a quorum read observes: the majority value is returned to the client and the stale replicas are written back through the same channels. It is covered in full in Replication and Consistency; the important boundary is that read repair only heals replicas a real read touched. Keys that are written but rarely read, and replicas that were down during the read, are left for anti-entropy.

Anti-entropy: Merkle-tree repair

Anti-entropy is the background reconciliation that does not depend on a client reading a key. Replicas periodically compare compact Merkle-tree digests of their key ranges; where the trees differ, only the divergent sub-ranges are exchanged and reconciled, so a full replica comparison costs a tree walk rather than a full data transfer.

flowchart TD
  R1["replica r1<br/>Merkle root"] --> CMP{"roots equal?"}
  R2["replica r2<br/>Merkle root"] --> CMP
  CMP -->|yes| DONE["ranges agree,<br/>nothing to do"]
  CMP -->|no| DESC["descend into<br/>differing subtrees"]
  DESC --> EX["exchange only the<br/>divergent key ranges"]
  EX --> REC["reconcile, write back"]

Merkle-tree anti-entropy narrows a whole-range comparison to just the sub-ranges that actually differ. Equal roots mean the replicas agree and no data moves.

This is the safety net beneath read repair and hinted handoff: it catches divergence that neither of the other two mechanisms reached -- writes that were never read back, hints that expired before the peer recovered, or data that drifted during a long partition. The full anti-entropy design, including the transactional Dyniak layer's reconciliation, is documented in Dyniak AAE.

What happens during a partition

Put the three mechanisms together and the partition story is:

sequenceDiagram
  participant A as DC-A side
  participant B as DC-B side
  Note over A,B: link between DCs drops
  Note over A: A's detector convicts B's peers (phi > 8)
  Note over B: B's detector convicts A's peers
  Note over A,B: each side keeps serving with its<br/>own reachable replicas
  Note over A: writes to B's replicas -> hinted (if enabled)
  Note over A,B: link heals
  A->>B: gossip reconverges membership
  A->>B: hints drained to recovered replicas
  A->>B: read repair + anti-entropy reconcile divergence

A cross-DC partition. Each side stays available on its own replicas, records hints for the unreachable side, and reconciles via gossip, hint drain, read repair, and anti-entropy once the link heals.

Availability

Both sides of a partition keep serving reads and writes against the replicas they can still reach. There is no leader to lose, so neither side stalls.

Durability within quorum

A write that meets its consistency level on the reachable side is not lost: it is on that side's replicas, and -- if hinted handoff is enabled -- queued for the unreachable replicas. When the partition heals, gossip readmits the peers, hints drain, and anti-entropy plus read repair close any remaining gap.

Consistency

Divergent writes on the two sides are reconciled after the heal, not prevented during the split. Under the strict consistency levels a request that cannot meet its level returns a no-quorum error rather than a divergent answer (see Replication and Consistency).

Where to go next

• Membership and Gossip -- how the failure detector is fed and how ejection / readmission propagate.
• Replication and Consistency -- read repair, the consistency levels, and what a no-quorum error means.
• The Ring and the Token Space -- how a Down peer drops out of routing on the next continuum rebuild.
• Dyniak AAE -- the full Merkle-tree anti-entropy and the transactional reconciliation path.
• Configuration -- the enable_hinted_handoff, hint_dir, hint_ttl_seconds, and failure-detector knobs.