-
Notifications
You must be signed in to change notification settings - Fork 5
Restructure orchestration models into typed profiles and tasks #381
Copy link
Copy link
Open
Labels
backendAssociated with the backend repositoryAssociated with the backend repositorydata mgmt appAssociated with the data management appAssociated with the data management app
Milestone
Description
Summary
Replace the generic Task model (with task_type discriminator) and DataConnection model with per-type profile and task tables. Remove OrchestrationSystem entirely — the internal Celery worker is the only orchestration runner.
Backend
- Remove
OrchestrationSystemmodel and all references - Keep
DataConnectionas the ETL profile (holds extractor/transformer/loader config + notification recipients) - Add new models:
AggregationProfile,AggregationTask,AggregationMapping(with typed fields:source_datastreamFK,target_datastreamFK,statistic,timezone_mode,timezone_offset) - Add stub models for
DerivedProfile,DerivedTask,DerivedMapping,DerivedMappingSource - Add stub models for
QualityProfile,QualityTask - Migrate existing
Taskrows intoETLTaskorAggregationTaskbased ontask_type - Separate
TaskRuntables per type (or shared with discriminator — TBD) - Add per-type Celery task functions (
run_etl_task,run_aggregation_task, etc.) - Add per-type API endpoints with strongly typed schemas
- Update
send_orchestration_notificationsto iterate all profile types - Update hydroserverpy client with typed namespaces (
hs.etl.connections,hs.aggregation.profiles,hs.aggregation.tasks, etc.)
Frontend
- Redesign orchestration page: profiles as the primary grouping, type-specific task table columns
- Type-specific task creation forms with datastream pickers (replace UUID text fields)
- Stepped creation flow (profile → schedule → mappings)
- Profile detail/edit slide-over (view details, edit, add task — all as explicit buttons)
- Empty state cards for new workspaces explaining each orchestration type
- Type filter in toolbar replacing the data connection toggle panel
References
- Derived Datastream Orchestration #368 (Derived datastream orchestration)
- Support Automated Data Quality Management #373 (Automated data quality management)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
backendAssociated with the backend repositoryAssociated with the backend repositorydata mgmt appAssociated with the data management appAssociated with the data management app