Merge pull request #14 from hotosm/infra/dev-k8s

Infra/dev k8s

Author: kshitijrajsharma

.github/workflows/validate-stac.yml

@@ -4,16 +4,18 @@ on:
     branches:
       - master
     paths:
-      - "models/**/stac-item.json"
+      - "models/**"
       - "fair/schemas/**"
       - "fair/stac/validators.py"
+      - "fair/zenml/model_validator.py"
   pull_request:
     branches:
       - master
     paths:
-      - "models/**/stac-item.json"
+      - "models/**"
       - "fair/schemas/**"
       - "fair/stac/validators.py"
+      - "fair/zenml/model_validator.py"
 defaults:
   run:
     shell: bash
@@ -22,7 +24,7 @@ concurrency:
   cancel-in-progress: true
 jobs:
   validate:
-    name: validate stac items
+    name: validate stac items & models
     runs-on: ubuntu-latest
     steps:
       - name: Clone repo
@@ -36,3 +38,5 @@ jobs:
         run: uv sync --group dev
       - name: Validate STAC items
         run: make validate-stac
+      - name: Validate model pipelines
+        run: make validate-models

.gitignore

@@ -223,4 +223,7 @@ data/sample/predict/predictions
 stac_catalog
 
 # hatch-vcs generated version file
-fair/_version.py
+fair/_version.py
+
+# dev infra
+infra/dev/.port-forward.pids

Makefile

@@ -52,3 +52,6 @@ pre-commit:
 validate-stac:
 	uv run python scripts/validate_stac_items.py
 
+validate-models:
+	uv run python scripts/validate_model.py
+

README.md

@@ -304,7 +304,7 @@ A user might run 10 experiments, promote 3 of them over time. Only those 3 appea
 | **ZenML** | Pipeline orchestration, version tracking | SQLite (`~/.config/zenml/`) | ZenML Server (PostgreSQL) |
 | **Orchestrator** | Runs pipeline steps | `local` | Kubernetes |
 | **Artifact Store** | Weights, datasets, artifacts | local filesystem | S3 |
-| **Experiment Tracker** | Metrics logging | W&B | W&B |
+| **Experiment Tracker** | Metrics logging | Mlflow | Mlflow |
 | **Container Registry** | Model runtime images | local Docker | ghcr.io |
 | **fAIr Backend** | User-facing orchestration layer | -- | -- |
 
@@ -360,3 +360,9 @@ ZenML stack components NOT used:
 5. **YAML-based training & inference**: Every pipeline run is driven by a generated YAML config (STAC defaults + user overrides). Logged as a ZenML artifact. Re-run any experiment by re-running its YAML.
 
 6. **MLM Processing Expression for dispatch**: `pre_processing_function` and `post_processing_function` use `format: "python"` with entrypoint strings (e.g., `ramp.pipeline:preprocess`). Each model defines its own pre/post processing; the system does not assume a fixed pipeline shape. System considers each model having its own runtime , preprocessing , postprocessing and training pipeline.
+
+7. W&B self-hosted requires MySQL + Redis + a commercial license for team use. Mlflow is fully open source.
+
+8. **Pipeline contract**: Every model under `models/` must export `training_pipeline` and `inference_pipeline` as `@pipeline`-decorated functions in its `pipeline.py`. CI validates this via AST parsing (`scripts/validate_model.py`) — no runtime dependencies required. This ensures the orchestration layer can discover and dispatch any contributed model uniformly.
+
+9. **S3 data resolution via UPath/fsspec**: Production training data lives in S3 (MinIO in dev, AWS in prod). `fair/utils/data.py` provides helpers (`list_files`, `resolve_path`, `resolve_directory`) built on `universal-pathlib` (UPath) over fsspec/s3fs — no boto3 or GDAL env vars. fsspec reads `AWS_ENDPOINT_URL` natively for MinIO. Caching is available via fsspec URL-chaining (`simplecache::s3://`, `filecache::s3://`, `blockcache::s3://`) — model developers opt in as needed. Local paths pass through unchanged. Data is never baked into Docker images.