Skip to content

Refactor augmentation Lambda to read from S3 instead of SQS body#424

Open
nickclyde wants to merge 1 commit intomainfrom
nickclyde/418-read-ecr-from-s3
Open

Refactor augmentation Lambda to read from S3 instead of SQS body#424
nickclyde wants to merge 1 commit intomainfrom
nickclyde/418-read-ecr-from-s3

Conversation

@nickclyde
Copy link
Copy Markdown
Member

@nickclyde nickclyde commented Apr 3, 2026

Summary

Closes #418

  • Rewrites the augmentation Lambda to be triggered by S3 events (via EventBridge → SQS) instead of receiving inline eICR XML and nonstandard codes in the SQS message body
  • Lambda now reads TTC output from TTCAugmentationMetadataV2/{persistenceId} and the original eICR from eCRMessageV2/{persistenceId}, matching the APHL AIMS eCR pipeline architecture
  • Implements source-bucket routing (extracts bucket name from S3 event, falls back to S3_BUCKET env var)
  • Adds Terraform resources for the full event pipeline: SQS queue + DLQ, EventBridge rule on TTCAugmentationMetadataV2/ prefix, Lambda event source mapping, IAM policy, and S3 EventBridge notification
  • Fixes existing Terraform bug where REGION env var was used instead of AWS_REGION (which lambda_handler.create_s3_client() expects)

Changes

packages/augmentation-lambda/

  • lambda_function.py: Complete rewrite — parses EventBridge S3 events from SQS, reads inputs from S3 via lambda_handler helpers, adds _parse_nonstandard_codes() to validate TTC output entries as NonstandardCodeInstance objects
  • pyproject.toml: Added aws-lambda-powertools, structlog deps; added boto3, moto dev deps
  • tests/conftest.py (new): moto mock_aws fixtures with EventBridge S3 event payloads and pre-populated S3 test data
  • tests/test_augmentation_lambda_function.py: Rewritten with 10 tests covering success, S3 writes, source-bucket routing, missing files, mixed batch, and _parse_nonstandard_codes parsing

terraform/

  • main.tf: Added EICR_INPUT_PREFIX and TTC_OUTPUT_PREFIX env vars, fixed REGIONAWS_REGION, added SQS queue + DLQ, EventBridge rule, event source mapping, and SQS IAM policy
  • s3.tf: Enabled EventBridge notifications on the S3 bucket

@nickclyde nickclyde force-pushed the nickclyde/418-read-ecr-from-s3 branch from 7ec9353 to ad36342 Compare April 3, 2026 20:57
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.49123% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.75%. Comparing base (d031555) to head (ad36342).

Files with missing lines Patch % Lines
...-lambda/src/augmentation_lambda/lambda_function.py 96.49% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #424      +/-   ##
==========================================
- Coverage   95.78%   95.75%   -0.03%     
==========================================
  Files          42       42              
  Lines        2183     2216      +33     
==========================================
+ Hits         2091     2122      +31     
- Misses         92       94       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refactor augmentation Lambda to read from S3 instead of SQS message body

2 participants