[db] Missing Index for Transactions Polling

[onelapahead]

In transaction managers with a large number of transactions (successful, failed, or pending) we observed large total query times and table scans:

SELECT pss.query, pss.calls,
       round(pss.mean_exec_time::numeric, 2) AS mean_ms,
       round(pss.total_exec_time::numeric/1000/60, 1) AS total_min
FROM pg_stat_statements pss
JOIN pg_database pd ON pd.oid = pss.dbid
WHERE pd.datname = 'a-db'
ORDER BY total_exec_time DESC
LIMIT 15;
                                                                                                                                         query                                                                                                                                          |  calls  | mean_ms | total_min
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------+---------+-----------
 SELECT seq, id, created, updated, status, delete, tx_from, tx_to, tx_nonce, tx_gas, tx_value, tx_gasprice, tx_data, tx_hash, policy_info, first_submit, last_submit, error_message FROM transactions WHERE (status = $1) ORDER BY seq LIMIT $2                                         | 1600655 |  116.69 |    3113.0
 SELECT seq, id, created, updated, status, delete, tx_from, tx_to, tx_nonce, tx_gas, tx_value, tx_gasprice, tx_data, tx_hash, policy_info, first_submit, last_submit, error_message FROM transactions WHERE (status = $1 AND tx_from NOT IN ($2,$3)) ORDER BY seq LIMIT $4              |  129269 |  154.04 |     331.9
 SELECT pg_advisory_xact_lock($1)                                                                                                                                                                                                                                                       |  461960 |   35.15 |     270.6
 UPDATE transactions SET tx_hash = $1, policy_info = $2, first_submit = $3, last_submit = $4, updated = $5 WHERE id = $6                                                                                                                                                                |  804940 |    7.92 |     106.3
 SELECT seq, id, created, updated, status, delete, tx_from, tx_to, tx_nonce, tx_gas, tx_value, tx_gasprice, tx_data, tx_hash, policy_info, first_submit, last_submit, error_message FROM transactions WHERE (status = $1 AND tx_from NOT IN ($2)) ORDER BY seq LIMIT $3                 |   16972 |  338.62 |      95.8
 INSERT INTO txhistory (id,tx_id,status,action,count,time,last_occurrence,error,error_time,info) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)  RETURNING seq                                                                                                                                 | 1203240 |    4.13 |      82.9
 UPDATE transactions SET status = $1, updated = $2 WHERE id = $3                                                                                                                                                                                                                        |  804940 |    4.24 |      56.9
 INSERT INTO txhistory (id,tx_id,status,action,count,time,last_occurrence,error,error_time,info) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10),($11,$12,$13,$14,$15,$16,$17,$18,$19,$20)  RETURNING seq                                                                                       |  341088 |    8.87 |      50.4
 SELECT seq, id, created, updated, status, delete, tx_from, tx_to, tx_nonce, tx_gas, tx_value, tx_gasprice, tx_data, tx_hash, policy_info, first_submit, last_submit, error_message FROM transactions WHERE (tx_from = $1 AND status = $2) ORDER BY tx_nonce LIMIT $3                   |    4399 |  434.76 |      31.9
 SELECT seq, id, created, updated, status, delete, tx_from, tx_to, tx_nonce, tx_gas, tx_value, tx_gasprice, tx_data, tx_hash, policy_info, first_submit, last_submit, error_message FROM transactions WHERE (status = $1 AND tx_from NOT IN ($2,$3,$4)) ORDER BY seq LIMIT $5           |    6339 |  254.07 |      26.8
 INSERT INTO receipts (id,created,updated,block_number,tx_index,block_hash,success,protocol_id,extra_info,contract_loc) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)  ON CONFLICT DO NOTHING RETURNING seq                                                                                   |  375816 |    4.26 |      26.7
 INSERT INTO txhistory (id,tx_id,status,action,count,time,last_occurrence,error,error_time,info) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10),($11,$12,$13,$14,$15,$16,$17,$18,$19,$20),($21,$22,$23,$24,$25,$26,$27,$28,$29,$30)  RETURNING seq                                             |  110694 |   14.47 |      26.7
 INSERT INTO transaction_completions (id, time, status) VALUES ($1,$2,$3) ON CONFLICT DO NOTHING                                                                                                                                                                                        |  369085 |    4.09 |      25.2
 INSERT INTO txhistory (id,tx_id,status,action,count,time,last_occurrence,error,error_time,info) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10),($11,$12,$13,$14,$15,$16,$17,$18,$19,$20),($21,$22,$23,$24,$25,$26,$27,$28,$29,$30),($31,$32,$33,$34,$35,$36,$37,$38,$39,$40)  RETURNING seq   |   47950 |   20.99 |      16.8
 SELECT seq, id, created, updated, status, delete, tx_from, tx_to, tx_nonce, tx_gas, tx_value, tx_gasprice, tx_data, tx_hash, policy_info, first_submit, last_submit, error_message FROM transactions WHERE (tx_from = $1 AND status = $2 AND tx_nonce > $3) ORDER BY tx_nonce LIMIT $4 |  601326 |    1.36 |      13.6
 
 SELECT relname, seq_scan, seq_tup_read, idx_scan, n_live_tup,
       round(seq_tup_read::numeric / nullif(seq_scan, 0), 0) AS avg_rows_per_seqscan
FROM pg_stat_user_tables
WHERE seq_scan > 0
ORDER BY seq_tup_read DESC
LIMIT 10;
      relname      | seq_scan | seq_tup_read | idx_scan | n_live_tup | avg_rows_per_seqscan
-------------------+----------+--------------+----------+------------+----------------------
 transactions      |     4972 |   2235009130 |  2604790 |    1459340 |               449519
 listeners         |       93 |         6030 |      108 |         66 |                   65
 eventstreams      |        9 |           18 |        6 |          2 |                    2
 schema_migrations |       14 |           14 |        0 |          1 |                    1

And if we did an explain on one of the more frequent queries from the policyloop.go when there no more pending transactions, but a large number of successful/failed transactions:

EXPLAIN (ANALYZE, BUFFERS)
SELECT seq, id, status FROM transactions
WHERE status = 'Pending'  -- substitute actual polled value
ORDER BY seq LIMIT 50;
                                                                     QUERY PLAN
----------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.43..898.97 rows=50 width=85) (actual time=828.230..828.231 rows=0 loops=1)
   Buffers: shared hit=1365960 read=6340
   ->  Index Scan using transactions_pkey on transactions  (cost=0.43..222911.10 rows=12404 width=85) (actual time=828.228..828.229 rows=0 loops=1)
         Filter: ((status)::text = 'Pending'::text)
         Rows Removed by Filter: 1459809
         Buffers: shared hit=1365960 read=6340
 Planning:
   Buffers: shared hit=16 read=19
 Planning Time: 2.286 ms
 Execution Time: 828.269 ms

This confirmed we were doing full table scans on the following query (3k total minutes, 1.6 million calls):

SELECT ... FROM transactions WHERE (status = $1) ORDER BY seq LIMIT $2

Additionally, another query:

WHERE (status = $1 AND tx_from NOT IN ($2,$3)) ORDER BY seq LIMIT $4

also hits the same problem accounting another ~450 minutes of our query time spent.

Additional queries from the enterprise downstream engine, also highlights this need and accounts for the rest of the query time.

The order by seq we apply, means the status index alone is rarely providing the value needed. We are mainly just using transaction_pkey index, and then doing full table scans, otherwise if transaction_status is used, we have to then sort.

So by indexing both together, we optimize for the main policyloop.go query, and provide a better starting point for all other transaction related queries.

[Chengxuan]

For DB index changes, please remember to share the following information:

1. migration time ( an example of a DB with a large volume of transactions would be helpful guidance)
2. metrics of the benefit that's provided by the new index.