Skip to content

fix(extract CDK): User defined cursor Incremental partition #74124

Merged
Rodi Reich Zilberman (rodireich) merged 23 commits intomasterfrom
15906-cdk-source-oracle-incremental-sync-failing
Apr 6, 2026
Merged

fix(extract CDK): User defined cursor Incremental partition #74124
Rodi Reich Zilberman (rodireich) merged 23 commits intomasterfrom
15906-cdk-source-oracle-incremental-sync-failing

Conversation

@rodireich
Copy link
Copy Markdown
Contributor

@rodireich Rodi Reich Zilberman (rodireich) commented Mar 2, 2026

The logic for incremental partitions was not splitting the table correctly, leading to possible failures when reading large tables incrementally with user defined cursor (no CDC).
This changes the default implementation of incremental partition to non splittable.
It may be possible to rewrite incremental partitions correctly to be able to split it correctly and account to all edge cases, but the utility of that may not be that high as typically the incremental reads are smaller and user a more expensive WHERE clause, which makes concurrent read less effective.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 2, 2026

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • 🛠️ Quick Fixes
    • /format-fix - Fixes most formatting issues.
    • /bump-version - Bumps connector versions, scraping changelog description from the PR title.
  • ❇️ AI Testing and Review (internal link: AI-SDLC Docs):
    • /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
    • /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
    • /ai-review - AI-powered PR review for connector safety and quality gates.
  • 🚀 Connector Releases:
    • /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
    • /bump-progressive-rollout-version - Bumps connector version with an RC suffix (2.16.10-rc.1) for progressive rollouts (enableProgressiveRollout: true).
      • Example: /bump-progressive-rollout-version changelog="Add new feature for progressive rollout"
  • ☕️ JVM connectors:
    • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
      Example: /update-connector-cdk-version connector=destination-bigquery
    • /bump-bulk-cdk-version bump=patch changelog='foo' - Bump the Bulk CDK's version. bump can be major/minor/patch.
  • 🐍 Python connectors:
    • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
    • /poe source example lock - Alias for /poe connector source-example lock.
    • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
    • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
  • ⚙️ Admin commands:
    • /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
      Example: /force-merge reason="CI is flaky, tests pass locally"
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Mar 2, 2026

Deploy preview for airbyte-kotlin-cdk ready!

✅ Preview
https://airbyte-kotlin-1l323sluy-airbyte-growth.vercel.app

Built with commit 705e44a.
This pull request is being automatically deployed with vercel-action

@rodireich
Copy link
Copy Markdown
Contributor Author

Rodi Reich Zilberman (rodireich) commented Mar 31, 2026

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (5c290fa)

@rodireich Rodi Reich Zilberman (rodireich) changed the title test Incremental partition Mar 31, 2026
@rodireich Rodi Reich Zilberman (rodireich) changed the title Incremental partition User defined cursor Incremental partition Mar 31, 2026
@rodireich Rodi Reich Zilberman (rodireich) marked this pull request as ready for review March 31, 2026 17:56
@rodireich Rodi Reich Zilberman (rodireich) changed the title User defined cursor Incremental partition fix(extract CDK): User defined cursor Incremental partition Mar 31, 2026
@rodireich
Copy link
Copy Markdown
Contributor Author

Rodi Reich Zilberman (rodireich) commented Apr 2, 2026

/format-fix

Format-fix job started... Check job output.

✅ Changes applied successfully. (901f4b4)

@rodireich
Copy link
Copy Markdown
Contributor Author

Matt Bayley (@mwbayley) I rebased with master's postgres CDK changes. If I can ask for another pass

Copy link
Copy Markdown
Contributor

@mwbayley Matt Bayley (mwbayley) left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just need to rebase on master, make the version 1.1.2 and adjust the changelog.

@rodireich Rodi Reich Zilberman (rodireich) merged commit 8292bb2 into master Apr 6, 2026
45 of 49 checks passed
@rodireich Rodi Reich Zilberman (rodireich) deleted the 15906-cdk-source-oracle-incremental-sync-failing branch April 6, 2026 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants