Skip to content

Restructure orchestration models into typed profiles and tasks #381

@daniel-slaugh

Description

@daniel-slaugh

Summary

Replace the generic Task model (with task_type discriminator) and DataConnection model with per-type profile and task tables. Remove OrchestrationSystem entirely — the internal Celery worker is the only orchestration runner.

Backend

  • Remove OrchestrationSystem model and all references
  • Keep DataConnection as the ETL profile (holds extractor/transformer/loader config + notification recipients)
  • Add new models: AggregationProfile, AggregationTask, AggregationMapping (with typed fields: source_datastream FK, target_datastream FK, statistic, timezone_mode, timezone_offset)
  • Add stub models for DerivedProfile, DerivedTask, DerivedMapping, DerivedMappingSource
  • Add stub models for QualityProfile, QualityTask
  • Migrate existing Task rows into ETLTask or AggregationTask based on task_type
  • Separate TaskRun tables per type (or shared with discriminator — TBD)
  • Add per-type Celery task functions (run_etl_task, run_aggregation_task, etc.)
  • Add per-type API endpoints with strongly typed schemas
  • Update send_orchestration_notifications to iterate all profile types
  • Update hydroserverpy client with typed namespaces (hs.etl.connections, hs.aggregation.profiles, hs.aggregation.tasks, etc.)

Frontend

  • Redesign orchestration page: profiles as the primary grouping, type-specific task table columns
  • Type-specific task creation forms with datastream pickers (replace UUID text fields)
  • Stepped creation flow (profile → schedule → mappings)
  • Profile detail/edit slide-over (view details, edit, add task — all as explicit buttons)
  • Empty state cards for new workspaces explaining each orchestration type
  • Type filter in toolbar replacing the data connection toggle panel

References

Metadata

Metadata

Labels

backendAssociated with the backend repositorydata mgmt appAssociated with the data management app

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions