Right now everything MOAR learns gets thrown away when the process exits:
- The first layer runs the sample pipeline once per model to measure cost/accuracy. This is a big chunk of the search cost, and we redo it every time — even for the same pipeline on the same dataset.
- The expansion prompt has an "Action Performance History" section that's supposed to tell the agent how each directive has performed across queries, but the stats reset every run, so it always says "Unknown (never tried)".
- We only save the frontier (yaml path + cost + accuracy) in
pareto_frontier.json. The cost/accuracy of everything else we evaluated is lost, so a new search on the same pipeline re-discovers plans we already paid for.
It would be nice to have an opt-in workspace — something like docetl.workspace = "./.docetl", a directory with a sqlite file in it (in-process, no server) — that persists:
- model stats per (pipeline, dataset), so repeat optimizations can skip the first layer
- directive stats globally, keyed by directive name, so the agent has real priors across different queries/pipelines
- explored plans with their cost/accuracy, so a new search can warm-start from a previous frontier, and results can be reloaded after the process exits
Default off — if you don't set a workspace, nothing changes.
Open questions: how to cheaply decide "same pipeline" / "same dataset" for safe reuse, and the fact that we can't detect when someone changes their eval_fn (stored accuracies would silently be stale).
Right now everything MOAR learns gets thrown away when the process exits:
pareto_frontier.json. The cost/accuracy of everything else we evaluated is lost, so a new search on the same pipeline re-discovers plans we already paid for.It would be nice to have an opt-in workspace — something like
docetl.workspace = "./.docetl", a directory with a sqlite file in it (in-process, no server) — that persists:Default off — if you don't set a workspace, nothing changes.
Open questions: how to cheaply decide "same pipeline" / "same dataset" for safe reuse, and the fact that we can't detect when someone changes their eval_fn (stored accuracies would silently be stale).