Skip to content

Reuse MOAR artifacts across optimization runs (persistent workspace) #496

Description

@shreyashankar

Right now everything MOAR learns gets thrown away when the process exits:

  • The first layer runs the sample pipeline once per model to measure cost/accuracy. This is a big chunk of the search cost, and we redo it every time — even for the same pipeline on the same dataset.
  • The expansion prompt has an "Action Performance History" section that's supposed to tell the agent how each directive has performed across queries, but the stats reset every run, so it always says "Unknown (never tried)".
  • We only save the frontier (yaml path + cost + accuracy) in pareto_frontier.json. The cost/accuracy of everything else we evaluated is lost, so a new search on the same pipeline re-discovers plans we already paid for.

It would be nice to have an opt-in workspace — something like docetl.workspace = "./.docetl", a directory with a sqlite file in it (in-process, no server) — that persists:

  1. model stats per (pipeline, dataset), so repeat optimizations can skip the first layer
  2. directive stats globally, keyed by directive name, so the agent has real priors across different queries/pipelines
  3. explored plans with their cost/accuracy, so a new search can warm-start from a previous frontier, and results can be reloaded after the process exits

Default off — if you don't set a workspace, nothing changes.

Open questions: how to cheaply decide "same pipeline" / "same dataset" for safe reuse, and the fact that we can't detect when someone changes their eval_fn (stored accuracies would silently be stale).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions