Skip to content

feat: add configurable batch size for bulk inserting failed rsources records#6873

Open
mihir20 wants to merge 2 commits intomasterfrom
mihir/bulk-insert-failed-records
Open

feat: add configurable batch size for bulk inserting failed rsources records#6873
mihir20 wants to merge 2 commits intomasterfrom
mihir/bulk-insert-failed-records

Conversation

@mihir20
Copy link
Copy Markdown
Contributor

@mihir20 mihir20 commented Apr 12, 2026

Description

Txn Inserting into rsources_failed_keys_v2_records is taking too much time. The core bottleneck is AddFailedRecords() for rsources_stats. It uses tx.Prepare + a loop of individual stmt.ExecContext calls. Each ExecContext is a separate network round-trip to Postgres. If a batch
has 1,000 aborted jobs with record_ids, that's 1,000 sequential INSERT round-trips all within one transaction.

  • Introduced batch processing for inserting failed records in rsources_failed_keys_v2_records
  • Added FailedRecordsInsertBatchSize configuration to control batch size
  • Implemented batch insert SQL statement construction
  • Updated unit tests to cover multi-batch scenarios
  • Ensured default batch size is set when configuration is invalid

We could have used the COPY command, but the insert needs ON CONFLICT (id, record_id) DO UPDATE SET ts = NOW(), and COPY doesn't support ON CONFLICT

Linear Ticket

Security

  • The code changed/added as part of this pull request won't create any security issues with how the software is being used.

@mihir20 mihir20 force-pushed the mihir/bulk-insert-failed-records branch from a0bc0ea to d838708 Compare April 12, 2026 08:09
@codecov
Copy link
Copy Markdown

codecov bot commented Apr 12, 2026

Codecov Report

❌ Patch coverage is 96.15385% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 79.66%. Comparing base (f0bdc52) to head (e049597).

Files with missing lines Patch % Lines
services/rsources/handler.go 96.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6873      +/-   ##
==========================================
- Coverage   80.08%   79.66%   -0.42%     
==========================================
  Files         565      565              
  Lines       63308    63323      +15     
==========================================
- Hits        50698    50447     -251     
- Misses       9619     9897     +278     
+ Partials     2991     2979      -12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mihir20 mihir20 force-pushed the mihir/bulk-insert-failed-records branch from d838708 to 91e7652 Compare April 12, 2026 08:59
@mihir20 mihir20 requested a review from atzoum April 12, 2026 08:59
@mihir20 mihir20 force-pushed the mihir/bulk-insert-failed-records branch from 91e7652 to 94718f8 Compare April 12, 2026 13:03
batchSize := sh.config.FailedRecordsInsertBatchSize
defaultFailedRecordsBatchSize := 5000
if batchSize <= 0 {
batchSize = defaultFailedRecordsBatchSize
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need an upper limit based on the max number of parameters allowed

…records

🔒 Scanned for secrets using gitleaks 8.28.0
@mihir20 mihir20 force-pushed the mihir/bulk-insert-failed-records branch from 94718f8 to 3da64f6 Compare April 13, 2026 05:55
🔒 Scanned for secrets using gitleaks 8.28.0
@mihir20 mihir20 requested review from atzoum and ktgowtham April 13, 2026 05:58
Copy link
Copy Markdown
Contributor

@ktgowtham ktgowtham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants