perf(postgres): add index for resource ID prefix filter

[epbensimpson]

Description

A prefix filter in ReadRelationships cannot currently use any of the existing indexes due to using a LIKE in the query, unless the database is configured to use C collation.

This PR adds an index with object_id varchar_pattern_ops which allows it to be used for LIKE queries regardless of collation.

Obviously there is some tradeoff in terms of writes with adding an additional index which needs some consideration. But without the index or appropriate collation the filter is basically unusable on any deployment using postgres with a significant number of tuples.

Testing

I couldn't find any existing tests specifically around index usage so testing was all manual, and was done by capturing the output from the query builder, replacing the specific revision with pg_current_snapshot(), and manually running the following query before and after adding the index manually, on a deployment with ~4.1 million tuples:

SELECT object_id, relation, userset_namespace, userset_object_id, userset_relation, caveat_name, caveat_context, expiration
FROM relation_tuple
WHERE pg_visible_in_snapshot(created_xid, pg_current_snapshot()) = true
  AND pg_visible_in_snapshot(deleted_xid, pg_current_snapshot()) = false
  AND namespace = '<namespace>'
  AND object_id LIKE '<prefix>\_%'
  AND (expiration IS NULL OR expiration > NOW());

Before

Gather  (cost=1000.00..410914.91 rows=21 width=154) (actual time=82.851..1220.622 rows=11 loops=1)
  Workers Planned: 2
  Workers Launched: 2
  Buffers: shared hit=327354 read=32287
  I/O Timings: shared read=2045.173
  ->  Parallel Seq Scan on relation_tuple  (cost=0.00..409912.81 rows=9 width=154) (actual time=446.095..1202.032 rows=4 loops=3)
"        Filter: (((object_id)::text ~~ '<prefix>\_%'::text) AND ((namespace)::text = '<namespace>'::text) AND pg_visible_in_snapshot(created_xid, pg_current_snapshot()) AND (NOT pg_visible_in_snapshot(deleted_xid, pg_current_snapshot())) AND ((expiration IS NULL) OR (expiration > now())))"
        Rows Removed by Filter: 1367386
        Buffers: shared hit=327354 read=32287
        I/O Timings: shared read=2045.173
Planning:
  Buffers: shared hit=1
Planning Time: 0.161 ms
Execution Time: 1220.659 ms

After

Index Scan using ix_relation_tuple_by_resource_id_prefix on relation_tuple  (cost=0.56..2.79 rows=22 width=154) (actual time=0.035..0.052 rows=11 loops=1)
  Index Cond: (((namespace)::text = '<namespace>'::text) AND ((object_id)::text ~>=~ '<prefix>_'::text) AND ((object_id)::text ~<~ '<prefix>`'::text))
"  Filter: (((object_id)::text ~~ '<prefix>\_%'::text) AND pg_visible_in_snapshot(created_xid, pg_current_snapshot()) AND (NOT pg_visible_in_snapshot(deleted_xid, pg_current_snapshot())) AND ((expiration IS NULL) OR (expiration > now())))"
  Buffers: shared hit=15
Planning:
  Buffers: shared hit=49 read=1
  I/O Timings: shared read=0.613
Planning Time: 0.988 ms
Execution Time: 0.635 ms

References

Fixes #3274

[github-actions]

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

[epbensimpson]

I have read the CLA Document and I hereby sign the CLA

[josephschorr]

Adding another index will significantly impact both write performance limits and storage requirements - Can we change the existing forward index to support prefixes somehow?

[epbensimpson]

I'm not an expert on postgres so take this with a grain of salt, but it doesn't seem feasible to me.

Assuming replacing the existing index with a new one is fine (would require a brief exclusive lock on the table which could be a problem under load), using varchar_pattern_ops we'd end up with an index that does not match the collation of the underlying column and would have significant read performance implications.

Using object_id COLLATE "C" in the existing index (rather, a new index that replaces it) is an option but has a large blast radius. Queries would need to be rewritten to also explicitly specify COLLATE "C" and wouldn't be able to use the index if they weren't, which seems like a potential footgun. It would also change result ordering to be byte-ordered which may not be desirable.

[epbensimpson]

We've manually deployed this index to our production instance in the meantime, here's some stats about the index creation:

DB Type: AWS Aurora Postgres db.r6g.2xlarge
Relations: ~159,363,120
Elapsed Time: 19 m 44 s 185 ms
Index Sizes:

pk_relation_tuple,56 GB
uq_relation_tuple_living_xid,53 GB
ix_relation_tuple_alive_by_resource_rel_subject_covering,26 GB
ix_relation_tuple_by_subject,7121 MB
ix_watch_index,2760 MB
ix_relation_tuple_by_resource_id_prefix,2758 MB
ix_relation_tuple_by_subject_relation,2176 MB
ix_gc_index,235 MB
ix_relation_tuple_expired,8192 bytes

[kchenery]

Adding another index will significantly impact both write performance limits and storage requirements - Can we change the existing forward index to support prefixes somehow?

If this is a big concern, I think you can get a bigger win there by swapping the created_xid and deleted_xid columns in the PK and then removing the index uq_relation_tuple_living_xid as this effectively becomes a duplicate then.

I didnt spot any queries that would be negatively impacted by this change as everytime you reference created_xid where an index here is useful, you also reference deleted_xid. Cant be 100% certain of that though as I'm not too familiar with the access patterns. Also I know this is far easier said than done.

As you'll see from Ben's data above that secondary unique index is almost half the table volume.