Skip to content

Dagster integration#378

Draft
giomrella wants to merge 19 commits intomainfrom
gio-dagster
Draft

Dagster integration#378
giomrella wants to merge 19 commits intomainfrom
gio-dagster

Conversation

@giomrella
Copy link
Copy Markdown
Collaborator

Draft PR for the Dagster integration for curious minds.
I had to update cfa-config-generator to have the generate_configs function actually return the config dictionary, but that's only used to display metadata in the Dagster UI. I also updated the R code to use managed identities for simplified testing, but that is also not strictly necessary. This is currently running on the normal wednesday schedule, but outputting to the test container: http://dagster.apps.edav.ext.cdc.gov/locations/cfa-epinow2-pipeline/schedules/weekly_rt_pipeline_schedule

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Dec 5, 2025

Thank you for your contribution @giomrella 🚀! Your pkgdown-site is ready for download 👉 here 👈!
(The artifact expires on 2025-12-26T21:48:41Z. You can re-generate it by re-running the workflow here.)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
giomrella added a commit to CDCgov/cfa-config-generator that referenced this pull request Dec 19, 2025
This PR does three things:
1. adds a generate_local_config function that returns the generated
configs without uploading to azure
2. returns the generated configs from generate_configs()
3. uses the DefaultAzureCredential instead of AzureCliCredential

These changes allow the library to be used without accessing Azure, and
enable Azure use via managed identities. This will streamline Dagster
for Rt pipeline: CDCgov/cfa-epinow2-pipeline#378

---------

Co-authored-by: Nate McIntosh <natemcintosh@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants