Create an index over an ftsdoc expression:

CREATE INDEX docs_bm25 ON docs USING fts (to_ftsdoc('english', body));

The expression form is the recommended “external content” model: the text lives in the table, and the index derives the analyzed ftsdoc from it, so no document copy is stored in the index.

The index answers a boolean match as a bitmap scan:

SELECT count(*) FROM docs
  WHERE to_ftsdoc('english', body) @@@ to_ftsquery('english', 'postgres & index');

and a ranked top-k as an ordering index scan with no Sort node:

SELECT id FROM docs
  WHERE to_ftsdoc('english', body) @@@ to_ftsquery('english', 'postgres')
  ORDER BY to_ftsdoc('english', body) <=> to_ftsquery('english', 'postgres')
  LIMIT 10;

The index is a set of immutable segments plus a small pending write buffer. An INSERT appends to the pending buffer and is immediately searchable without a REINDEX. A flush — performed automatically by VACUUM, or on demand by fts_merge — folds pending documents into a segment; a size-tiered merge coalesces segments and physically drops tombstoned (deleted) documents. All page writes go through GenericXLog, so the index is crash-safe and replicated on a physical standby. Deletes are handled MVCC-correctly: VACUUM records a per-segment tombstone, and scans and counts exclude tombstoned documents.

CREATE EXTENSION pg_fts;

CREATE TABLE docs (id serial PRIMARY KEY, body text);
INSERT INTO docs (body) VALUES
  ('the quick brown fox'),
  ('a quick red fox jumps'),
  ('lazy brown dogs sleep');

CREATE INDEX docs_bm25 ON docs USING fts (to_ftsdoc('english', body));

-- boolean match
SELECT id FROM docs
  WHERE to_ftsdoc('english', body) @@@ to_ftsquery('english', 'quick & fox');

-- ranked top-2 by relevance
SELECT id FROM docs
  WHERE to_ftsdoc('english', body) @@@ to_ftsquery('english', 'fox')
  ORDER BY to_ftsdoc('english', body) <=> to_ftsquery('english', 'fox')
  LIMIT 2;

-- fast count
SELECT fts_count('docs_bm25', to_ftsquery('english', 'brown'));

By default the bm25 index stores no token positions, so a phrase ("a b") or NEAR query generates the term-conjunction candidate set and then rechecks adjacency against the heap document, re-deriving the ftsdoc for each candidate. For a common two-word phrase whose conjunction set is large, that recheck dominates.

Building the index WITH (positions = on) stores per-token positions in the posting lists, so phrase and NEAR are answered directly from the index -- with no heap access and no recheck -- bringing phrase count and match to posting-scan speed:

CREATE INDEX docs_bm25 ON docs USING fts (to_ftsdoc('english', body))
  WITH (positions = on);

The trade-off is index size: positions roughly double the posting bytes on high-term-frequency corpora (little effect when terms occur once per document). Positions are decoded lazily, so plain ranked / boolean / count queries are unaffected whether positions are on or off. Phrase and NEAR are always correct either way; positions = on only makes them fast (index-only). This is an on-disk format change (BM25 v2 → v3); an index built by an older release must be REINDEXed.

Building an index over a large corpus of long, high-vocabulary documents (for example full email bodies or source code — many distinct, low-frequency terms per document) has three cost drivers: build memory, build time (dominated by per-document text analysis), and merge time.

Build time and throughput. The largest cost on a long-document corpus is usually the per-document text analysis itself (tokenizing and, for a language configuration such as english, stemming every token). This work is inherent to the text-search configuration, is proportional to the total token count, and for very long documents dominates everything the index does. It is also embarrassingly parallel: set max_parallel_maintenance_workers (and enough max_parallel_workers / max_worker_processes) so the analysis runs across cores — on a many-core host this is the single biggest reduction in wall-clock build time. Keep the memory formula above in mind when choosing the worker count (each participant holds its own budget). A serial build (max_parallel_maintenance_workers = 0) analyzes documents one at a time and, on a multi-gigabyte corpus, can legitimately run for a long time before the first segment is flushed — the buffer fills only after a whole budget's worth of (large) documents. During that window the segment count does not change and nothing is written yet; the build emits a LOG-level progress line as documents are analyzed and at each segment flush (set log_min_messages to log or lower to see them), so a long build can be distinguished from a stuck one.

Memory. Peak build memory is bounded by roughly shared_buffers + (max_parallel_maintenance_workers + 1) x 2 x maintenance_work_mem. Size maintenance_work_mem and max_parallel_maintenance_workers so that figure fits your host (and any cgroup MemoryMax). A larger maintenance_work_mem also flushes fewer, larger segments during the scan, which reduces the amount of merging afterwards — a good trade when you have RAM to spare.

Fewer segments on a very large build. A build flushes a new segment each time a participant's in-memory buffer reaches its budget; by default that budget grows only up to 2 x maintenance_work_mem, so a very large corpus can produce many segments. If you have spare RAM and want fewer, larger segments (less post-scan merging, and staying comfortably under the internal 128-segment limit), raise pg_fts.build_mem_ceiling_mb (the per-participant flush-budget ceiling, in MB; 0 keeps the default). Peak build memory is then about shared_buffers + (max_parallel_maintenance_workers + 1) x pg_fts.build_mem_ceiling_mb — size it against your host's free RAM.

Merge / finalization. After the scan the build compacts the many per-worker segments. Rather than always collapsing to a single segment (a single-backend pass over the whole index that can run for a long time on a very large corpus), a build whose total index size exceeds pg_fts.build_collapse_max_mb (default 4096 MB) stops at a bounded, size-tiered set of segments. The index is valid and fully queryable; ranked scans traverse a bounded handful of segments (a small, fixed cost). Run fts_merge(index) in a maintenance window to collapse to one segment when you want the smallest index and the lowest ranked-scan latency. Set pg_fts.build_collapse_max_mb to 0 to always collapse (regardless of size), or raise it if you have the time and memory to collapse a large index during the build itself.

Time, and planning a window. The scan phase parallelizes across max_parallel_maintenance_workers; the final single-segment collapse (whether at the end of a build under the cap, or an explicit fts_merge) is a single-backend pass that reads and rewrites the whole index once, so its cost is roughly linear in the index size. As an order-of-magnitude planning figure, that collapse writes on the order of tens of MB of index per second per core (measured ~40-60 MB/s on a commodity NVMe box; your hardware and vocabulary will vary), so a multi-GB index takes minutes, not hours — but budget for it in a maintenance window and watch the progress (below). If a build must be predictable on a modest box, a serial or low-parallelism build (max_parallel_maintenance_workers = 0 or a small value) trades throughput for a smaller, more predictable peak memory and segment count; combined with leaving the index tiered (the default above the cap) it completes in a bounded, observable time rather than one open-ended collapse.

Monitoring a build. The merge phase is not covered by pg_stat_progress_create_index (its blocks_done freezes once the scan finishes). Poll fts_index_nsegments(index) and fts_index_stats(index) (both work on an in-progress indisvalid = f index) to watch the segment count fall and the doc/term counts grow as merges complete, and set client_min_messages = debug1 (or log_min_messages = debug1) to see per-merge progress lines (“merging N of M segments ... wrote merged segment (T terms, D docs) in S s”).

Partitioning. For a very large corpus you can partition the table and build a per-partition fts index; each partition's build and merge are correspondingly smaller. A query with a partitionwise plan fans out across the per-partition indexes and each is scored against its own partition's corpus statistics. BM25 scores are therefore comparable within a partition; if you need a single global ranking across partitions, prefer one whole-corpus index and the tiered-build behavior above.

Gregory Burd.