fix(extract CDK): User defined cursor Incremental partition #74124
Conversation
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
|
Deploy preview for airbyte-kotlin-cdk ready! ✅ Preview Built with commit 705e44a. |
|
/format-fix
|
|
/format-fix
|
|
Matt Bayley (@mwbayley) I rebased with master's postgres CDK changes. If I can ask for another pass |
Matt Bayley (mwbayley)
left a comment
There was a problem hiding this comment.
Looks good. Just need to rebase on master, make the version 1.1.2 and adjust the changelog.
8292bb2
into
master
The logic for incremental partitions was not splitting the table correctly, leading to possible failures when reading large tables incrementally with user defined cursor (no CDC).
This changes the default implementation of incremental partition to non splittable.
It may be possible to rewrite incremental partitions correctly to be able to split it correctly and account to all edge cases, but the utility of that may not be that high as typically the incremental reads are smaller and user a more expensive
WHEREclause, which makes concurrent read less effective.