Stats and metrics

Dynomite exposes its runtime counters, gauges, and histogram rollups on the stats_listen HTTP endpoint. The same in-memory snapshot is served in two formats so operators can pick the one that fits their stack:

• GET / (and the aliases /info and /stats) returns the legacy Netflix Dynomite JSON layout. This is the original wire format and remains byte-for-byte stable; existing scrapers, dashboards, and scripts that target the legacy schema continue to work unchanged.
• GET /metrics returns Prometheus 0.0.4 text exposition. Every metric family is annotated with a # HELP description and a # TYPE declaration. This is the recommended path for modern observability stacks (Prometheus, VictoriaMetrics, Grafana Agent, Mimir, Thanos, OpenTelemetry collectors with the Prometheus receiver).

Both endpoints read the same cached Snapshot value the aggregator publishes once per second, so they always agree.

Metric reference

The table below covers every metric family the Prometheus endpoint emits.

Failure-cause counters

The metrics above describe what the engine is doing when traffic is flowing normally. The counters in this section disambiguate the different error causes the dispatcher and gossip planes can produce. All of them initialise to zero and only become meaningful once the operator wires the dispatcher and gossip handler with a shared FailureMetrics accumulator (the dynomited binary does this automatically; embedders do it via ClusterDispatcher::with_failure_metrics(...) and GossipHandler::with_failure_metrics(...)).

The counters answer questions like "is the cluster losing requests because peers are flapping in and out of Down, or because perper-peer outbound channels are saturated?" Pre-existing aggregate error counters (dynomite_pool_client_err_total, dynomite_pool_client_dropped_requests_total) report the total but do not separate the cause; the families below do.

The _milli suffix on gossip_phi_score_milli is deliberate. Prometheus integer gauges are 64-bit signed; the phi value is a floating-point number that we widen by 1000 to preserve thousandths precision while staying within the integer-gauge wire format. A suspicion threshold of phi > 8.0 therefore corresponds to gossip_phi_score_milli > 8000 in PromQL.

Active Anti-Entropy (AAE) counters

The AAE worker (Tictac merkle-tree exchange + repair sink) ships its own family of counters and gauges. They start at zero and only become meaningful once the embedding wires the AAE handle into the scheduler and repair scheduler:

• dyniak::aae::Scheduler::install_metrics(handle) plus Scheduler::observe_exchange_attempt / observe_exchange_success / observe_divergent_keys from the per-tick hot path.
• dyniak::aae::RepairScheduler::with_metrics(handle, dc, rack) for the repair-dispatched count.
• dyniak::aae::metrics::save_snapshot_with_metrics(...) / load_snapshot_with_metrics(...) for the snapshot counters.

The families and labels:

Sample PromQL queries

Exchange success rate per peer (operators want this near 100%):

sum by (peer_idx) (rate(aae_exchange_success_total[5m]))
  /
sum by (peer_idx) (rate(aae_exchange_attempts_total[5m]))

Divergence rate per DC (a sustained non-zero rate is the signal that AAE is doing useful work):

sum by (dc) (rate(aae_exchange_divergent_keys_total[5m]))

Seconds since last full sweep on every peer (alert when this exceeds the configured full_sweep_interval_seconds):

time() - aae_full_sweep_last_completed_seconds_gauge

Snapshot health indicator (any non-zero corruption rate that is NOT immediately after a deploy is a paging condition):

rate(aae_snapshot_corruption_total[15m])

Sample PromQL queries

Total NoTargets rate per consistency level (this is the metric to watch during chaos runs to confirm peer-state oscillation as the root cause):

sum by (consistency_level) (
  rate(dispatch_no_targets_total[1m])
)

Per-peer flap count over the last hour (a peer that is flapping will show a high transition count):

sum by (peer_idx) (
  increase(peer_state_transitions_total[1h])
)

Live phi score per peer, in raw phi units:

gossip_phi_score_milli / 1000

Dispatch error breakdown (one line per cause; useful as a stacked graph in Grafana to see which cause dominates the error budget):

sum (rate(dispatch_no_targets_total[1m])) +
sum (rate(dispatch_peer_send_full_total[1m])) +
sum (rate(dispatch_peer_send_closed_total[1m])) +
sum (rate(dispatch_backend_send_full_total[1m])) +
sum (rate(dispatch_backend_send_closed_total[1m])) +
sum (rate(dispatch_response_timeout_total[1m]))

The histogram quantile rollups are emitted as gauges, not as Prometheus histograms, because the engine stores Cassandra-style estimated histograms whose internal buckets are not the standard Prometheus le ladder. Exposing the pre-computed mean, 0.95, 0.99, 0.999, and max rollups keeps the wire payload small while preserving the same percentiles the JSON endpoint already publishes.

Sample scrape configuration

A minimal prometheus.yml snippet that scrapes a three-node cluster on port 22222:

scrape_configs:
  - job_name: dynomite
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets:
          - dynomite-0.example.internal:22222
          - dynomite-1.example.internal:22222
          - dynomite-2.example.internal:22222
        labels:
          cluster: prod-east

Sample Grafana panel

A single-stat panel that charts cluster-wide request volume from the counter dynomite_pool_client_read_requests_total:

{
  "type": "timeseries",
  "title": "Pool requests/sec",
  "targets": [
    {
      "expr": "sum by (pool) (rate(dynomite_pool_client_read_requests_total[1m]))",
      "legendFormat": "{{pool}}",
      "refId": "A"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "reqps"
    }
  }
}

Drop the panel into a Grafana dashboard JSON under panels[] and adjust the datasource to point at your Prometheus instance.