feat(source-pylon): true incremental sync via POST /issues/search on updated_at#76083
Conversation
…dated_at The issues stream previously used GET /issues with start_time/end_time query parameters, which filtered by created_at. This meant updates to existing issues (status changes, reassignments, new messages, etc.) were never captured in incremental syncs. Pylon's new POST /issues/search endpoint supports filtering on updated_at, enabling true incremental sync that captures all issue modifications. Changes: - Switch issues stream from GET /issues to POST /issues/search - Change incremental cursor from created_at to updated_at - Add request_body_json with updated_at time filter - Add cursor_paginator_body for POST body pagination - Add updated_at to issues schema - Update test fixtures to reflect new cursor field - Bump connector version to 0.0.6
|
|
|
Note 📝 PR Converted to Draft More info...Thank you for creating this PR. As a policy to protect our engineers' time, Airbyte requires all PRs to be created first in draft status. Your PR has been automatically converted to draft status in respect for this policy. As soon as your PR is ready for formal review, you can proceed to convert the PR to "ready for review" status by clicking the "Ready for review" button at the bottom of the PR page. To skip draft status in future PRs, please include |
👋 Welcome to Airbyte!Thank you for your contribution from usbogie/airbyte! We're excited to have you in the Airbyte community. If you have any questions, feel free to ask in the PR comments or join our Slack community. 💡 Show Tips and TricksPR Slash CommandsAs needed or by request, Airbyte Maintainers can execute the following slash commands on your PR:
Tips for Working with CI
📚 Show Repo GuidanceHelpful Resources
|
Henry Bogardus (usbogie)
changed the title
|
↪️ Triggering Reason: CI is green (33 passed, 0 failed). Human-authored fix for true incremental sync on |
|
Prove-Fix Report:
|
| Detail | Value |
|---|---|
| Connector | source-pylon |
| Version | 0.0.6 |
| PR | #76083 |
| Oncall Issue | airbytehq/oncall#11880 |
| Session | Devin Session |
Outcome: Could Not Run Live Tests (Blocked on Approval)
Live connection testing is blocked pending HITL approval (escalated to PR author and fallback approver, no response after 1.5 hours). Regression tests are blocked due to missing GSM credentials (alpha connector). All pre-flight checks passed and code analysis strongly supports the fix.
Pre-flight Checks
- Viability: Code changes address the reported issue (switches from GET /issues with created_at to POST /issues/search with updated_at)
- Safety: No malicious code, no credential harvesting, no suspicious patterns
- Breaking change: NOT breaking (alpha connector, cursor field change, new POST endpoint with pagination)
- Reversibility: Version bump follows semver (0.0.5 -> 0.0.6), state format compatible
Pre-release
- Published:
0.0.6-preview.6cc204b(workflow)
Regression Tests
- Triggered comparison mode regression tests (workflow)
- Result: FAILED due to missing GSM credentials (no integration test config exists for source-pylon in GSM)
- This is an infrastructure limitation for this alpha connector, not a code issue
Connection Investigation
- 8 active connections found, all TIER_2, all actively syncing and succeeding
- Identified 1 internal (Airbyte-hosted) connection as best test candidate
- Current version: v0.0.5 (not pinned), healthy baseline (approximately 786 records per sync)
Live Connection Tests
- Blocked: awaiting HITL approval to pin 1 internal connection to pre-release
- Escalated to Henry Bogardus (@usbogie) (PR author) at 11:53 UTC - no response
- Escalated to Aaron ("AJ") Steers (@aaronsteers) (fallback) at 12:55 UTC - no response
- Reminder set for follow-up check
Evidence Analysis
Code analysis supports the fix:
- The
issuesstream correctly usesPOST /issues/searchwithupdated_atcursor filter DatetimeBasedCursorconfigured withcursor_field: updated_atand proper datetime formats- Request body uses
subfilterswithtime_is_after/time_is_beforeoperators onupdated_at - Pagination uses
cursor_paginator_body(body-based cursor pagination for POST requests) - Step size of P30D with PT1S granularity is appropriate for incremental sync
- Schema includes
updated_atfield in the inline schema definition
Risk assessment:
- Low risk: alpha connector, 8 total connections, all TIER_2
- Non-breaking: no schema type changes, no field removals, no PK changes
- Reversible: standard semver bump, state format compatible
Next Steps
To complete live testing (when approval is received):
- Pin the internal connection source actor to
0.0.6-preview.6cc204b - Trigger sync and monitor results
- Verify: issues stream returns records,
updated_atcursor works, sync completes successfully - Unpin connection after testing
- Update this report with results
For merge decision without live testing:
- All pre-flight checks passed
- Code analysis confirms the fix correctly implements POST /issues/search with updated_at cursor
- Non-breaking change on an alpha connector with low blast radius (8 connections)
- All existing connections are healthy and succeeding on v0.0.5
Report finalized: 2026-04-04T13:25 UTC | Devin Session
|
|
↪️ Triggering Reason: CI is green (33 passed, 0 failed). Prove-fix report complete (pre-flight checks passed, pre-release published, live testing pending approval). Next pipeline step is AI review. |
Reviewing PR for connector safety and quality.
|
AI PR Review ReportReview Action: NO ACTION (NOT ELIGIBLE)
📋 PR Details & EligibilityConnector & PR InfoConnector(s): Auto-Approve EligibilityEligible: No Review Action DetailsNO ACTION (NOT ELIGIBLE) — Anti-Pattern gates (Forwards Compatibility, Behavioral Changes) are flagged and require human sign-off. The PR is not eligible for auto-approval due to functional code changes. No PR review submitted. Human review required.
🔍 Gate Evaluation DetailsGate-by-Gate Analysis
PR Hygiene — PASS
Code Hygiene — WARNING
Test Coverage — PASS
Code Security — PASS
Backwards Compatibility — PASSSpec Comparison:
Schema Changes:
Stream Changes:
Note: Cursor field change from Forwards Compatibility — FAIL (Blocks Auto-Approve)Keywords found in diff hunks of
Concerns:
Requires human sign-off to verify that the state transition is acceptable for this alpha connector (8 connections, all TIER_2). Behavioral Changes — FAIL (Blocks Auto-Approve)Keywords found in diff hunks of
Concerns:
These are intentional changes per the PR description — the new endpoint supports filtering on CI Checks — PASSAll core checks completed successfully:
Community PR note: Test [No Creds] passed, indicating tests that don't require credentials are green. Live / E2E Tests — UNKNOWNValidation Required: Yes — this PR matches sync behavior changes (cursor field, incremental_sync, paginator, record_selector changes in manifest.yaml) and is linked to issue #76084. Evidence Check (Priority 2 — CI/Labels fallback):
Result: UNKNOWN — pre-release exists and pre-release checks passed, but no live connection test results available yet. Live testing was attempted via Recommendation: When HITL approval is granted, complete the live validation cycle. Alternatively, a maintainer can approve based on code analysis and the alpha connector's low blast radius (8 connections, all TIER_2). 📚 Evidence ConsultedEvidence
❓ How to RespondProviding Context or JustificationYou can add explanations that the bot will see on the next review: Option 1: PR Description (recommended) ## AI PR Review Justification
### Forwards Compatibility
[Your explanation here — e.g., why the state migration from created_at to updated_at is acceptable]
### Behavioral Changes
[Your explanation here — e.g., confirming the endpoint change is intentional and tested]Option 2: PR Comment After adding your response, re-run Note: For Anti-Pattern gates (Forwards Compatibility, Behavioral Changes), justifications provide context but still require human sign-off. These gates block auto-approval by design — a maintainer must review and approve the change. State Migration RecommendationThe cursor field change from |
Existing connections have state with created_at as the cursor key. Without migration, upgrading to 0.0.6 would cause a full re-sync. This CustomStateMigration renames the state key so existing connections resume from their last cursor position.
3 participants
Summary
Closes #76084
issuesstream fromGET /issues(filtered bycreated_at) toPOST /issues/search(filtered byupdated_at) for true incremental syncPOST /issues/searchendpoint supports filtering onupdated_at, so incremental syncs now capture all issue modificationscursor_paginator_bodyto handle pagination via POST request body (the search endpoint acceptscursor/limitin the body, not query params)updated_atto the issues schemaCustomStateMigrationto renamecreated_at→updated_atin existing connection state, preventing a full re-sync on upgradeTest plan
source-declarative-manifest checkpasses with valid Pylon API tokensource-declarative-manifest readon the issues stream returns records withupdated_atpopulatedupdated_atcursor{"created_at": "..."}state correctly migrated, records resume from cursor positionairbyte-cdk connector testAI PR Review Justification
Forwards Compatibility
A
CustomStateMigration(MigrateIssuesCursorFromCreatedAtToUpdatedAt) handles the cursor field rename fromcreated_attoupdated_at. When existing connections upgrade from v0.0.5, the migration detectscreated_atin state and renames it toupdated_at, so connections resume from their last cursor position without a full re-sync. Unit tests cover all edge cases (old state, new state, empty state, both keys present).Behavioral Changes
The endpoint change from
GET /issuestoPOST /issues/searchand the cursor field change fromcreated_attoupdated_atare intentional. The previous endpoint only supported filtering by creation time, which made "incremental" sync miss all modifications to existing issues. The new search endpoint supportsupdated_atfiltering, enabling true incremental sync. The response format (dataarray,paginationobject) is identical between both endpoints. Tested locally against a production Pylon instance with ~120k issues.