feat(test): add unit test suite and GitHub Actions CI#112
Open
rederik76 wants to merge 6 commits into
Open
Conversation
Introduce shared test fixtures, minimal JSON fixtures, and a unit/ integration test layout. Cover config loading, bundle discovery, logging, token substitution, secrets, and the dataflow spec pipeline (builder, template expansion, mapping, transformers, and expectations). Add broader template_processor tests for validation, YAML templates, parameter defaults and types, caching, and failure paths. Register pytest markers for spark, integration, and bdd scenarios. Add pytest-cov for optional coverage reports and ignore coverage artifacts in git.
Restore PR checks on main with path-filtered jobs for docs spelling, HTML build, and sample dataflow validation. Document local equivalents in tests/README.md and contributor_dev_steps.rst.
PySpark imports distutils, which requires setuptools on Python 3.12+. Regenerate requirements-dev.lock with --allow-unsafe and fix paths-filter SHA.
1 task
Add dataflow unit tests, fixtures, and integration tests that validate every sample bundle spec and exercise DataflowSpecBuilder on feature-samples. Fix validate_dataflows.py to load version mappings from config/default so legacy specs pass with mapping applied. Fail docs spelling CI on misspellings, update contributor docs for trunk-based workflow, refresh script examples for current sample bundles, remove obsolete develop-build workflow, and bump VERSION to v0.19.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces the first automated unit test suite and PR CI for the Lakeflow Framework.
Unit test harness
tests/conftest.py,tests/helpers.py, minimal JSON/YAML undertests/fixtures/tests/unit/for fast isolated tests,tests/integration/for script-level checkspytest tests/ -m "not integration and not spark"(~158 tests, < 30s)integration,spark,bdd(latter two skipped in CI)Unit test coverage
Core
src/modules, using fixtures instead of fullsamples/trees:Integration tests for
validate_dataflows.pyare behind theintegrationmarker.GitHub Actions CI
New
.github/workflows/ci.ymlonmainwith parallel, path-filtered jobs:docs/**changed — HTML fails above 19 warnings (scripts/ci/docs_html_check.sh)samples/**changed —python scripts/validate_dataflows.py samples/Contributor docs
tests/README.md— layout, markers, local commands, CI matrixcontributor_dev_steps.rst— unit testing, sample validation, docs checks; trunk-based workflow targetingmainCloses #[issue-number]
Test plan
pip install --require-hashes --no-deps -r requirements-dev.lockpytest tests/ -m "not integration and not spark" -qpytest tests/ -m integration -q— optional locallypython scripts/validate_dataflows.py samples/— requiresjsonschemamake -C docs spellingandbash scripts/ci/docs_html_check.sh 19— if docs changedmain: unit tests always; docs and sample jobs path-filteredmain