List pipeline-level config keys in CLI help (closes #320) by mmcdermott · Pull Request #381 · mmcdermott/MEDS_transforms

mmcdermott · 2026-04-17T10:35:31Z

Summary

Closes #320. Users had no way to discover pipeline-level keys (input_dir, output_dir, parallelize.*, stages, description) from -h; they had to grep runner.py.

Both CLIs now append the same table after their usage:

Pipeline-level configuration keys (settable in the pipeline YAML; `--overrides` is accepted by `MEDS_transform-pipeline`):
  input_dir                    Root MEDS directory to read from ...
  output_dir                   Root directory to write stage outputs into.
  stages                       List of stage configs defining the pipeline ...
  description                  Optional free-text pipeline description.
  parallelize.n_workers        Number of parallel workers to launch per stage (default 1).
  parallelize.launcher         Hydra launcher to use (e.g. 'joblib', 'slurm').
  parallelize.launcher_params  Launcher-specific keyword arguments forwarded to Hydra.

The table is defined once in src/MEDS_transforms/_cli_help.py (PIPELINE_CONFIG_KEYS) and consumed by both entrypoints. runner.main() imports only _cli_help so the pipeline runner's help path stays free of the Hydra/ConfigStore imports that __main__ pulls in.

Test plan

Manual: MEDS_transform-pipeline -h and MEDS_transform-stage -h both show the new block.
test_pipeline_help extended and new test_stage_help_lists_pipeline_keys asserts the block.
Full non-parallel suite passes.

Closes #320

🤖 Generated with Claude Code

Both MEDS_transform-pipeline -h and MEDS_transform-stage -h now list the accepted pipeline-level configuration keys (input_dir, output_dir, stages, description, parallelize.*) alongside the argparse/stage usage, so users can discover them without reading runner.py. Closes #320.

codecov · 2026-04-17T10:37:48Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.24%. Comparing base (daeecca) to head (cb256da).
⚠️ Report is 28 commits behind head on dev.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev     #381      +/-   ##
==========================================
+ Coverage   98.16%   98.24%   +0.07%     
==========================================
  Files          54       55       +1     
  Lines        2510     2619     +109     
==========================================
+ Hits         2464     2573     +109     
  Misses         46       46

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copilot

Pull request overview

Adds discoverability for pipeline-level YAML/--overrides keys by surfacing a shared “Pipeline-level configuration keys …” table in both the pipeline runner (MEDS_transform-pipeline -h) and the stage dispatcher help.

Changes:

Define PIPELINE_CONFIG_KEYS and a formatter helper to render a consistent help-text block.
Append the help block to MEDS_transform-stage help output.
Add the help block as an argparse epilog for MEDS_transform-pipeline -h.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
`src/MEDS_transforms/runner.py`	Adds an argparse epilog to include the pipeline-level config keys table in `-h` output.
`src/MEDS_transforms/__main__.py`	Defines the shared keys table + help-block builder and prints it in stage CLI help.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Move PIPELINE_CONFIG_KEYS and the help-block formatter into a new, dependency-light module src/MEDS_transforms/_cli_help.py. runner.main() no longer imports __main__ (and therefore Hydra) just to format its epilog; __main__.print_help_stage imports the helper lazily at call time. - Rename the function to pipeline_keys_help_block (dropped the leading underscore): it's deliberately a cross-module helper. - Extend test_pipeline_help to assert the new block appears, and add test_stage_help_lists_pipeline_keys for the stage dispatcher.

mmcdermott · 2026-04-17T13:45:13Z

All three comments addressed in a64a03d:

Private helper imported from another module: moved PIPELINE_CONFIG_KEYS and the formatter into a new dependency-light module src/MEDS_transforms/_cli_help.py. Renamed the function to pipeline_keys_help_block (no leading underscore) — it's deliberately a cross-module helper.
Heavy import chain for runner -h: runner.main() now imports _cli_help instead of main. main still pulls Hydra for the stage dispatcher, but that's no longer in the pipeline runner's help path.
Missing help-block test coverage: extended test_pipeline_help and added test_stage_help_lists_pipeline_keys so both CLI help surfaces regress-check for the keys block.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Reword the help-block header so it no longer implies --overrides works for MEDS_transform-stage. New phrasing: keys are settable in the pipeline YAML; --overrides is specifically the pipeline runner's flag.

mmcdermott · 2026-04-17T15:18:27Z

Both round-2 comments addressed:

Header implies --overrides works for the stage CLI (bde4b11): reworded to "settable in the pipeline YAML; --overrides is accepted by MEDS_transform-pipeline" — no longer implies the stage CLI accepts that flag.
PR description / code mismatch: updated the PR description to say _cli_help.PIPELINE_CONFIG_KEYS instead of __main__.PIPELINE_CONFIG_KEYS.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Clarify in the help-block header that dot-notation (e.g. `parallelize.n_workers`) is the Hydra override path accepted by `--overrides`, and that pipeline YAML needs nested mappings (`parallelize:` → `n_workers:`). Keeps the compact table while removing the implication that `parallelize.n_workers:` is valid YAML. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

mmcdermott · 2026-04-20T03:24:23Z

+    is written as a nested mapping (``parallelize:`` with ``n_workers:`` indented beneath).
+
+    Examples:
+        >>> block = pipeline_keys_help_block()


just show the block.

Done in cb256da — doctest now calls print(pipeline_keys_help_block()) and locks in the full rendered block verbatim. Makes the expected output self-documenting and catches any accidental wording/spacing drift.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Replaces the three piecewise assertions (startswith / in / in) with a single `print(pipeline_keys_help_block())` that shows the whole block. Locks in the exact rendered text and doubles as documentation. Also split the header into three wrapped lines so long CLI help renders without line-wrapping midword; this is the form the doctest now shows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

mmcdermott requested a review from Copilot April 17, 2026 12:59

Copilot started reviewing on behalf of mmcdermott April 17, 2026 12:59 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Comment thread src/MEDS_transforms/runner.py

Comment thread src/MEDS_transforms/__main__.py Outdated

Comment thread src/MEDS_transforms/runner.py Outdated

mmcdermott requested a review from Copilot April 17, 2026 14:59

Copilot started reviewing on behalf of mmcdermott April 17, 2026 15:00 View session

Copilot AI reviewed Apr 17, 2026

View reviewed changes

Comment thread src/MEDS_transforms/_cli_help.py Outdated

Comment thread src/MEDS_transforms/_cli_help.py

Address Copilot round-2 on PR #381

bde4b11

Reword the help-block header so it no longer implies --overrides works for MEDS_transform-stage. New phrasing: keys are settable in the pipeline YAML; --overrides is specifically the pipeline runner's flag.

mmcdermott requested a review from Copilot April 20, 2026 02:56

Copilot started reviewing on behalf of mmcdermott April 20, 2026 02:57 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

Comment thread src/MEDS_transforms/_cli_help.py

mmcdermott requested a review from Copilot April 20, 2026 03:22

Copilot started reviewing on behalf of mmcdermott April 20, 2026 03:23 View session

mmcdermott commented Apr 20, 2026

View reviewed changes

Copilot AI reviewed Apr 20, 2026

View reviewed changes

mmcdermott merged commit e1e31d6 into dev Apr 20, 2026
9 checks passed

mmcdermott deleted the fix/pipeline-args-in-help branch April 20, 2026 03:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

List pipeline-level config keys in CLI help (closes #320)#381

List pipeline-level config keys in CLI help (closes #320)#381
mmcdermott merged 5 commits intodevfrom
fix/pipeline-args-in-help

mmcdermott commented Apr 17, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Apr 17, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mmcdermott commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

mmcdermott commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

mmcdermott Apr 20, 2026

Uh oh!

mmcdermott Apr 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

mmcdermott commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

codecov Bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mmcdermott commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

mmcdermott commented Apr 17, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

mmcdermott Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

mmcdermott Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mmcdermott commented Apr 17, 2026 •

edited

Loading

codecov Bot commented Apr 17, 2026 •

edited

Loading