Skip to content

implement lazy schema discovery for streams with table configuration#5

Open
Vishal-Sharma-Acquia wants to merge 5 commits intomainfrom
Schema-Discovery
Open

implement lazy schema discovery for streams with table configuration#5
Vishal-Sharma-Acquia wants to merge 5 commits intomainfrom
Schema-Discovery

Conversation

@Vishal-Sharma-Acquia
Copy link

This pull request introduces lazy schema discovery for streams in the tap_sumologic project, improving how schemas are handled and loaded during stream processing. Instead of discovering or loading schemas during stream initialization, schemas are now loaded or inferred only when the stream is actually processed. This change enhances performance, flexibility, and robustness, especially for streams with dynamic or unknown schemas.

Schema discovery improvements:

  • Added a table_config parameter to the SearchJobStream class and updated its initialization to support lazy schema discovery, including handling cases where the schema is not provided and initializing a placeholder schema. (tap_sumologic/streams.py) [1] [2] [3]
  • Modified the get_records method to perform schema discovery just before processing records if the schema has not already been discovered, updating schema and primary keys as needed. (tap_sumologic/streams.py)

Configuration and workflow changes:

  • Updated the discover_streams method to defer schema loading and inference until stream processing, passing None for schema and the full table config for lazy evaluation. (tap_sumologic/tap.py) [1] [2]
  • Changed log messages in discover_streams to reflect the new lazy schema discovery process. (tap_sumologic/tap.py)

CI workflow fix:

  • Fixed the matrix definition in .github/workflows/ci_workflow.yml by quoting the Python version 3.10 for consistency. (.github/workflows/ci_workflow.yml)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant