BM25-style Ranked Search via rum

PostgreSQL's stock GIN-indexed tsvector @@ tsquery is great for filtering, but to rank matches by relevance the planner has to do a post-fetch scan and call ts_rank_cd per row. This is fine for tens of matches, painful for millions.

rum is a PostgreSQL-licensed (permissive!) GIN-derived index access method from PostgresPro that stores positional information alongside lexemes. Combined with the <=> distance operator and rum_ts_score() ranking function, it returns top-K relevance-ordered documents directly from the index. It is the closest permissive alternative to BM25 indexing in PostgreSQL — pg_mentat's choice over the AGPL-licensed pg_search (ParadeDB).

rum is an optional dependency. The (rum-fulltext ...) where-fn produces SQL that uses standard @@ for filtering and rum_ts_score(...) for ranking — both work without the rum extension installed (against a sequential scan), but the partial RUM index is where the speed-up lives.

When to use this vs other fulltext options

If you don't have rum installed and the dataset is < 100k rows, stay on (fulltext ...). Above that scale, install rum and switch to (rum-fulltext ...).

Quick start

# Build rum from source (one-time, ~30 sec).
git clone https://github.com/postgrespro/rum
cd rum
make USE_PGXS=1 PG_CONFIG=/path/to/your/pg_config
make USE_PGXS=1 PG_CONFIG=/path/to/your/pg_config install

CREATE EXTENSION pg_mentat;
CREATE EXTENSION rum;

-- Define a fulltext-tagged attribute.
SELECT mentat.t('[
  {:db/ident :issue/body
   :db/valueType :db.type/string
   :db/cardinality :db.cardinality/one
   :db/fulltext true}
]');

-- Create a partial RUM index on this attribute (idempotent).
SELECT mentat.create_rum_fulltext_index(':issue/body');
-- => 'datoms_text_rum_<entid>_english'

-- Top-K ranked search.
SELECT mentat.q('[
  :find ?body ?score
  :where [(rum-fulltext $ :issue/body "database crash") [[?e ?body ?score]]]
  :order (desc ?score)
]');

The rum-fulltext where-fn

[(rum-fulltext $ <:attr> <"search-text">) [[?e ?val ?score]]]

Binding shape [[?e ?val ?score]] (relation):

• ?e — entid of each matching datom.
• ?val — the stored text value.
• ?score — rum_ts_score(...) in [0.0, 1.0]-ish range. Higher = more relevant.

Plain text uses plainto_tsquery; double-quoted text uses phraseto_tsquery for proximity matching.

Index helpers

-- Idempotent. Returns the deterministic index name.
SELECT mentat.create_rum_fulltext_index(':issue/body');
-- => 'datoms_text_rum_<entid>_english'

-- Same attribute, different language config.
SELECT mentat.create_rum_fulltext_index(':issue/body', 'spanish');
-- => 'datoms_text_rum_<entid>_spanish'

-- Drop. Returns true if the index existed.
SELECT mentat.drop_rum_fulltext_index(':issue/body');
SELECT mentat.drop_rum_fulltext_index(':issue/body', 'spanish');

The index DDL is partial:

CREATE INDEX datoms_text_rum_<entid>_<lang>
    ON mentat.datoms_text_new
    USING rum (to_tsvector('<lang>', v) rum_tsvector_ops)
    WHERE a = <entid> AND added = true;

The WHERE a = <entid> AND added = true clause keeps the index small even in workspaces with many string attributes, and excludes retracted datoms automatically.

How rum's ranking differs from ts_rank_cd

ts_rank_cd (cover density) only knows lexeme presence and document length. rum_ts_score also knows lexeme positions, so it ranks:

• Phrase matches higher than disjoint matches of the same lexemes.
• Closer co-occurrences higher than distant ones.
• Documents where query lexemes appear early higher than late.

This is closer in spirit to BM25's term-frequency × inverse-document- frequency × proximity scoring than ts_rank_cd is, though it is not algorithmically identical to BM25.

Errors

Worked example: bug-tracker search

(d/q '[:find ?title ?score
       :where
         [(rum-fulltext $ :issue/title "memory leak") [[?e ?title ?score]]]
         [?e :issue/status :status/open]
         [?e :issue/priority :priority/high]
       :order (desc ?score)
       :limit 20]
     db)

Compiles to a query plan that:

1. Hits the partial RUM index on :issue/title for ranked top matches.
2. Joins the result back to the :issue/status and :issue/priority attribute datoms via the entid.
3. Returns the top 20 by descending rum_ts_score.

If the RUM index is sized appropriately and the status/priority datoms are also indexed, the entire pipeline is index-driven — no sequential scan over the issue body text.

License caveat

ParadeDB's pg_search is AGPL-3.0, which makes it unsuitable for many commercial deployments without source-disclosure. rum is PostgreSQL license — same terms as PostgreSQL itself. If you ship a SaaS or proprietary product, rum is the only credible permissive choice for index-backed BM25-style ranking in PostgreSQL today.

What this does NOT give you

• True BM25. rum's score is positional + length-aware but is not the algorithm Lucene/Tantivy implement. Close, not identical.
• Faceted search, aggregations, learned ranking. That's Elasticsearch/Tantivy territory.
• Cross-attribute ranking in one call. Each rum-fulltext binds one attribute. Combine with OR for multi-attribute search; the resulting plan does index lookups per attribute then unions.
• Fuzzy / typo-tolerant matching. Use pg_trgm's (similar-to) or pg_tre's (fuzzy-match) for that.