f01eb5a06c
* MUL-6928 perf(issues): index scalar property contains behind a bigram prefilter
Scalar `contains` property filtering runs `(properties ->> def) ILIKE
'%needle%'`, which no index can serve, so it degrades to a workspace-wide
sequential scan. At 1M issues that costs ~8x the throughput of an equality
filter, and the pgxpool connections it holds queue every other tenant's cheap
query behind it.
Put a redundant, indexable prefilter in front of the per-key check:
`LOWER(properties::text) LIKE LOWER('%needle%')`, served by a new pg_bigm GIN
index on the same expression. The per-key ILIKE still decides the result; the
prefilter only narrows the candidate set. On a 500k-row pg_trgm proxy a
selective needle goes from a 183ms Parallel Seq Scan to a 1.4ms Bitmap Index
Scan.
The prefilter is skipped, falling back to today's behaviour, for needles it
cannot screen soundly or usefully: jsonb escapes `"`, `\` and control
characters on serialization, so those never appear literally in
properties::text, and a single character carries no bigram to narrow with.
Migration 446 cannot use the DO ... EXCEPTION guard migrations 032 / 036 use
for pg_bigm — a concurrent build cannot run inside one — so the runner gates it
on the operator class instead: databases without pg_bigm record the version
with its SQL skipped and keep the unaccelerated (still correct) filter.
cmd/migrate/README.md documents building it out of band afterwards.
Co-authored-by: multica-agent <[email protected]>
* MUL-6928 fix(issues): prefilter short contains needles too
The single-character skip rule was derived from the pg_trgm proxy used to
develop this change, where a 1-character needle can only drive a sequential or
full-index scan. pg_bigm is different — indexing 1- and 2-character keywords is
the capability it exists for over pg_trgm — so the rule denied the target
extension exactly the searches it was chosen to serve, and a single CJK
character is already a word.
Measured on PostgreSQL 16.15 with pg_bigm 1.2 over 1M issues: a rare
1-character CJK needle goes from a 134ms Parallel Seq Scan to a 1.2ms Bitmap
Index Scan (8.3 -> 10527 tps at 4 clients). Only the JSON-escaping guard
remains, which is about soundness rather than benefit.
Co-authored-by: multica-agent <[email protected]>
* MUL-6928 fix(issues): analyze the properties expression index after building it
Building an expression index does not collect statistics for its expression.
Until they exist the planner falls back to a pattern-length heuristic that
estimates any single-character LIKE at a few percent of the table, so it cannot
tell a needle matching 50 rows from one matching 50,000 — and migration 446's
index ends up built, valid, ready, and never chosen. Autovacuum does not rescue
a large, quiet issue table promptly: its default threshold is 10% of rows
changed.
Measured on PostgreSQL 16.15 with pg_bigm 1.2 over 1M issues, with
max_parallel_workers_per_gather=4: a rare single-character CJK needle plans as a
Parallel Seq Scan at ~101ms with 446 alone — indistinguishable from the
pre-change baseline — and as a 0.64ms Bitmap Index Scan once the statistics
exist. Whether the wrong estimate actually flips the plan depends on how cheap a
parallel scan looks, so without this the shipped behaviour would vary by
instance sizing.
447 is deliberately not gated on pg_bigm the way 446 is: ANALYZE is valid, cheap
(557ms on 1M rows) and side-effect-free everywhere, and refreshing ordinary
column statistics is useful even where the expression index was skipped.
Co-authored-by: multica-agent <[email protected]>
---------
Co-authored-by: J <[email protected]>
Co-authored-by: multica-agent <[email protected]>
3.5 KiB
Migration runner operations
Recover the comment content search index
Migration 371 keeps exactly one comment-content search index per environment:
idx_comment_content_bigm when pg_bigm is usable, otherwise the portable
idx_comment_content_trgm fallback. A conditionally skipped migration is still
recorded in schema_migrations, so rerunning migrate up does not recreate the
fallback if the selected bigram index is later dropped or becomes invalid.
First check whether either index is live, ready, and valid:
SELECT indexrelid::regclass AS index_name, indisvalid, indisready, indislive
FROM pg_index
WHERE indexrelid IN (
to_regclass('idx_comment_content_bigm'),
to_regclass('idx_comment_content_trgm')
);
If neither index is usable, restore the portable fallback before serving search traffic. Run each statement separately and outside a transaction so the concurrent index build is valid:
CREATE EXTENSION IF NOT EXISTS pg_trgm;
DROP INDEX CONCURRENTLY IF EXISTS idx_comment_content_trgm;
CREATE INDEX CONCURRENTLY idx_comment_content_trgm
ON comment USING gin (LOWER(content) gin_trgm_ops);
Verify that idx_comment_content_trgm reports all three flags as true before
resuming traffic. If idx_comment_content_bigm is repaired later, keep the
fallback until the bigram index also reports all three flags as true and
has the exact migration 036 shape: a non-unique, non-partial GIN index on
LOWER(content) using the pg_bigm-owned gin_bigm_ops operator class. Only
then can the fallback be dropped with DROP INDEX CONCURRENTLY during a
maintenance window.
Build the issue properties bigram index after installing pg_bigm
Migration 446 builds idx_issue_properties_bigm, the index behind the prefilter
that scalar contains property filtering puts in front of its per-key ILIKE.
The runner only executes it where the gin_bigm_ops operator class is
installed; everywhere else the version is recorded with its SQL skipped, and the
filter keeps working without index acceleration.
That record is permanent, so a database that gains pg_bigm later never builds
the index on its own. Check first:
SELECT indexrelid::regclass AS index_name, indisvalid, indisready, indislive
FROM pg_index
WHERE indexrelid = to_regclass('idx_issue_properties_bigm');
If the index is missing, create it out of band. Run each statement separately and outside a transaction so the concurrent build is valid:
CREATE EXTENSION IF NOT EXISTS pg_bigm;
DROP INDEX CONCURRENTLY IF EXISTS idx_issue_properties_bigm;
CREATE INDEX CONCURRENTLY idx_issue_properties_bigm
ON issue USING gin (LOWER(properties::text) gin_bigm_ops);
ANALYZE issue;
The ANALYZE is not optional and not a formality. Building an expression index
does not collect statistics for its expression, and until they exist the
planner has nothing to judge the index by: it falls back to a pattern-length
heuristic that estimates a single-character needle at 5% of the table and
leaves contains on a sequential scan, with the index built, valid and unused.
Migration 447 does this after 446; an out-of-band build has to do it itself.
Keep the expression exactly as written: the predicate is
LOWER(properties::text) LIKE LOWER(...), and an ILIKE-shaped or
non-lowered index would never be used (pg_bigm 1.2 on RDS has no ILIKE index
scan — the constraint migration 036 hit).
Confirm the statistics landed:
SELECT count(*) FROM pg_statistic WHERE starelid = 'idx_issue_properties_bigm'::regclass;