Monitoring

This page is for whoever runs a Postgres database with pg_improv in it, and for anyone running improv-server. It covers what to watch, what the signals mean, and when to page someone. It describes pg_improv 0.11.0, the version on main. Operations runbook has what to do when an alert fires.

Ready-made files:

• docs/ops/postgres_exporter-queries.yaml: custom queries for postgres_exporter, exposing every model's status, the slot and the worker as metrics.
• docs/ops/grafana-improv.json: a Grafana dashboard over those metrics.

Both were checked against a PG 18.6 cluster running pg_improv 0.11.0 built from main, with postgres_exporter 0.17.1.

What to watch

flowchart LR
  W[writers] -->|commit| WAL[(WAL)]
  WAL -->|decoded each poll| S[slot improv_db]
  S --> K["pg_improv sync worker<br/>(one per server)"]
  K -->|result rows| R[(improv_‹model›.*)]
  K -->|counters, last_error| ST[improv.sync_state]
  ST --> V["improv.status(model)"]
  S --> RS[pg_replication_slots]
  K --> PA[pg_stat_activity]
  K --> LOG[server log]

Only logical-mode models have a worker, a slot and lag. trigger and commit models fail inside the writer's transaction, so the writer sees the error. A manual model changes only when someone calls improv.recalc().

improv.status(model)

One row per model, SECURITY INVOKER, readable by anyone with USAGE on schema improv and SELECT on its tables (row-level security hides only a model's input cells from roles that may not act as its owner).

SELECT m.name, s.*
  FROM improv.model m, improv.status(m.name) s;

lag_bytes is measured against the shared slot, so it does not tell you which model is slow. Compare last_applied_at and the counters across models for that.

improv.status does not report the slot's wal_status. A lost slot (see below) leaves last_error NULL on 0.11.0 and earlier, so watch pg_replication_slots as well. D11 in the road to 1.0 makes the worker report and repair it.

The memory columns are estimates of the worker's heap. They come from the sizes of the structures the worker holds, within about 10 % of the live heap measured on the shapes in the tests (maps, joins, SUM, count, sum, count_distinct, min). They do not count the shared buffers the worker reads through. To find the models that use the most memory:

SELECT m.name, pg_size_pretty(s.fold_bytes + s.engine_bytes) AS live
  FROM improv.model m, improv.status(m.name) s
 WHERE s.fold_bytes IS NOT NULL
 ORDER BY s.fold_bytes + s.engine_bytes DESC;

Capping a model's memory

improv.max_model_memory (MB, default 0 = no cap, reloadable) caps each model's live state. The worker checks it whenever it measures the model (after each rebuild, and at most once a second as it applies), and checks the loaded inputs before it builds an engine. A model over the cap is refused: the worker drops its live state, logs pg_improv sync: model <m>: the live state needs about N MB (fold state F MB, engine E MB), over improv.max_model_memory = C MB, and sets that text as the model's last_error. It retries with backoff (each retry is a rebuild), and other models are not held back. The model's result tables keep their last values. Set the cap below what the server can spare for the worker divided by the number of large models, so one model that grows fails alone rather than getting the backend OOM-killed and taking every model down with it.

pg_replication_slots

SELECT slot_name, active, active_pid, wal_status, safe_wal_size,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))         AS retained,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unconfirmed
  FROM pg_replication_slots
 WHERE slot_name = 'improv_db';

• retained (pg_current_wal_lsn() − restart_lsn) is what the slot keeps on disk in pg_wal. This is the number that fills disks.
• unconfirmed is lag_bytes from improv.status.
• active is true only while the worker is decoding (during a poll), so it flickers; active_pid is the worker's pid then.
• wal_status is reserved or extended while the slot is healthy. unreserved means it is past max_wal_size and about to exceed max_slot_wal_keep_size. lost means Postgres removed WAL the slot needed: the slot is dead, results no longer follow writes, and on 0.11.0 and earlier the worker does not recover by itself (runbook).
• safe_wal_size is how much more WAL may be written before the slot is lost (NULL when max_slot_wal_keep_size is unlimited, which is the Postgres default; set it).

With no logical-mode model left, the worker drops the slot. A database whose models are all trigger, commit or manual has no improv_db row.

The worker in pg_stat_activity

SELECT pid, datname, backend_start, state, wait_event_type, wait_event
  FROM pg_stat_activity
 WHERE backend_type = 'pg_improv sync';

• There is one worker per server, serving the database named by improv.database (default postgres). It exists only when pg_improv is in shared_preload_libraries.
• No row means the worker is not running. Without the preload there is never one; with it, the postmaster restarts a dead worker 5 s after it exits.
• backend_start keeps resetting when the worker is crash-looping.
• Between polls it waits on the latch: wait_event_type = 'Extension', wait_event = 'Extension'.
• datname is the database it serves. Logical-mode models in any other database are never applied.

The worker's log lines

The worker logs at LOG level into the server log. Every line starts with pg_improv sync. Here is each line in crates/pg_improv/src/worker.rs:

Rebuild reasons (in crates/pg_model/src/live.rs and worker.rs):

Log volume on 0.11.0 and earlier. Every poll decodes the slot, and Postgres itself logs two LOG lines per decoding call, each with a DETAIL and a CONTEXT line: starting logical decoding for slot "improv_db" and logical decoding found consistent point at …. It does this even when nothing has changed. Measured on PG 18.6 (both PG 17 and 18 log these at LOG): about 3.6 KB/s of server log at the default improv.poll_ms = 200 (about 310 MB a day), and 13.5 KB/s at poll_ms = 50. Rotate the server log (log_rotation_size, log_rotation_age, or logrotate), and do not lower poll_ms below what you need. log_min_messages cannot hide these lines without hiding the worker's own LOG lines as well. D8 in the road to 1.0 skips the decode when there is no new WAL and quiets it, fixing this in 0.11.

Changing improv.poll_ms / improv.max_changes. Both take effect on pg_reload_conf() in 0.11 (D9). On 0.10.0 and earlier, and on main before D9 lands, the running worker keeps its old values until it restarts: SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE backend_type = 'pg_improv sync' (it is back 5 s later, and loses nothing).

Alerts

Thresholds are starting points for a database like the capacity planning example (one poll is 200 ms). Tune them to your write rate.

As PromQL over the queries file (metric names are <query>_<column>):

groups:
- name: pg_improv
  rules:
  - alert: ImprovWorkerDown
    expr: pg_improv_worker_running == 0
    for: 1m
    labels: {severity: page}
  - alert: ImprovSlotLost        # wal_status unreserved or lost
    expr: pg_improv_slot_at_risk == 1
    labels: {severity: page}
  - alert: ImprovRetainedWalHigh
    expr: pg_improv_slot_retained_bytes > 2 * 1024^3   # half of a 4 GB max_slot_wal_keep_size
    for: 10m
    labels: {severity: warn}
  - alert: ImprovLagGrowing
    expr: pg_improv_status_lag_bytes{sync_mode="logical"} > 64 * 1024^2
          and deriv(pg_improv_status_lag_bytes{sync_mode="logical"}[15m]) > 0
    for: 15m
    labels: {severity: warn}
  - alert: ImprovModelFailing
    expr: pg_improv_status_last_error == 1
    for: 5m
    labels: {severity: warn}
  - alert: ImprovRebuildsRising
    expr: increase(pg_improv_status_rebuilds[15m]) > 3
    labels: {severity: warn}

Run postgres_exporter against the database named by improv.database, with --extend.query-path=postgres_exporter-queries.yaml, and as a role that has USAGE on schema improv, SELECT on its tables, and pg_monitor.

The HTTP server

improv-server (The HTTP Server) serves one model store.

• Health: GET /health returns 200 ok without a token. It says the process is up and accepting requests. It does not open the store, so pair it with a periodic authenticated GET /measures if you need to know the model loads.
• Scheduler: GET /scheduler/status (token required) returns {"enabled", "tick_secs", "last_tick_refreshed", "last_error"} when IMPROV_SCHEDULER=1. Alert on a non-null last_error.
• Logs: plain lines on stderr, with no timestamps, so capture stderr with your supervisor (systemd's journal adds the time):

• Signals in responses: 409 means another writer saved since the server loaded (the next request reloads). 408 and 413 mean the request limits were hit. A rising 409 rate means two writers are fighting over one model.