fix(source-s3): increase memory for Check Connection to prevent OOM on large buckets#75968
fix(source-s3): increase memory for Check Connection to prevent OOM on large buckets#75968devin-ai-integration[bot] wants to merge 2 commits intomasterfrom
Conversation
…n large buckets Add resourceRequirements.jobSpecific.check_connection with 4096Mi memory to metadata.yaml, matching the pattern used by source-azure-blob-storage and source-sftp-bulk (both cdk:python-file-based connectors). This resolves OOM exit code 137 during Check Connection for customers with large S3 buckets (e.g. 16 Parquet files, ~10.8GB total). Co-Authored-By: bot_apk <apk@cognition.ai>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| @@ -356,6 +356,7 @@ This connector utilizes the open source [Unstructured](https://unstructured-io.g | |||
|
|
|||
There was a problem hiding this comment.
[markdownlint-fix] reported by reviewdog 🐶
Co-Authored-By: bot_apk <apk@cognition.ai>
|
Deploy preview for airbyte-docs ready! ✅ Preview Built with commit 696d431. |
|
|
/publish-connectors-prerelease
|
|
↪️ Triggering Reason: CI all green, prerelease published — advancing to review stage. Session: https://app.devin.ai/sessions/944109c31676472da2e61c4602224cad |
Reviewing PR for connector safety and quality.
|
AI PR Review ReportReview Action: NO ACTION (NOT ELIGIBLE)
📋 PR Details & EligibilityConnector & PR InfoConnector(s): Auto-Approve EligibilityEligible: No Review Action DetailsNO ACTION (NOT ELIGIBLE) — All enforced gates pass, but the Behavioral Changes anti-pattern gate is flagged (
🔍 Gate Evaluation DetailsGate-by-Gate Analysis
Detailed AnalysisPR Hygiene:
Code Security — Diff Hunk Analysis:
Backwards Compatibility — Spec Comparison:
Behavioral Changes — Diff Hunk Keywords:
Live / E2E Tests — Validation Analysis:
📚 Evidence ConsultedEvidence
|
2 participants
What
Resolves https://github.com/airbytehq/oncall/issues/11839:
The source-s3 connector (v4.15.2) hits OOM (exit code 137) during Check Connection for customers with large S3 buckets. The root cause is that
metadata.yamllacks aresourceRequirementssection, so Check Connection runs with the platform default of 800Mi memory — insufficient for buckets with many large files (e.g. 16 Parquet files totaling ~10.8GB).How
Adds a
resourceRequirements.jobSpecificentry forcheck_connectionwithmemory_limit: 4096Miandmemory_request: 4096Mi. This matches the established pattern used by sibling file-based connectors:source-azure-blob-storage—4096Miforcheck_connectionsource-sftp-bulk—4096Miforcheck_connectionNo code changes — metadata-only fix. Version bumped from
4.15.2→4.15.3(patch).Review guide
airbyte-integrations/connectors/source-s3/metadata.yaml— the core fix (resourceRequirements block) and version bumpairbyte-integrations/connectors/source-s3/pyproject.toml— version bumpdocs/integrations/sources/s3.md— changelog entryHuman review checklist
resourceRequirementsYAML block is correctly indented underdata:and placed beforereleases:4096Miis the right value (matchessource-azure-blob-storageandsource-sftp-bulk)4.15.3is consistent acrossmetadata.yamlandpyproject.tomlcheck_connection, not syncTest Coverage
No unit tests added. This is a metadata-only change — no connector code was modified. The
resourceRequirementsfield is consumed by the platform scheduler, not by the connector runtime, so there is no connector-level code path to test.User Impact
Customers with large S3 buckets will no longer experience OOM failures during Check Connection. No negative side effects — this only increases the memory ceiling for the check job type.
Can this PR be safely reverted and rolled back?
Link to Devin session: https://app.devin.ai/sessions/10726618e5e94f188f58ebb3d2dba5ad