MultiProgressManagers.jl

A lightweight progress tracking system for parallel tasks. Implemented in pure Julia with a Tachikoma.jl-based dashboard and SQLite persistence.

Features

• 📊 Tachikoma Dashboard: Clean 2-tab terminal UI for monitoring experiments
• 💾 SQLite Persistence: Current state stored in SQLite
• 🔄 Multi-Task Support: Track parallel sub-tasks within a single experiment
• ⚡ Simple API: update!, finish!, fail!
• 🔀 Distributed & Threads: Single DB writer on the master; local and remote workers overwrite a per-task slot so update!(task) does not stall compute. Remote IPC is coalesced (~10 ms) over a RemoteChannel

Demo

Installation

Package (for use in Julia scripts/REPL):

using Pkg
Pkg.add("MultiProgressManagers")

App / CLI (optional, for the mpm dashboard from the shell):

using Pkg
Pkg.Apps.add("MultiProgressManagers")

Ensure ~/.julia/bin is on your PATH. Then run mpm <folder> (the directory that contains your .db files) or mpm --help.
You can also open the dashboard from Julia without the app: view_dashboard(folder) (see Quick Start).

Quick Start

Basic Usage (same process)

using MultiProgressManagers

# Create a multi-task experiment with 5 parallel tasks
N = 5
parameter_values = rand(N)
manager = ProgressManager(
    "Training Run", N;
    description = "Epoch 1-10 of ResNet training",
    db_path = "./progresslogs/experiment1.db",
    task_descriptions = ["parameter=$(val)" for val in parameter_values],
)

# Update progress for each task as it runs
for (task_num, param_val) in enumerate(parameter_values)
    steps = rand([100, 200, 300])
    update!(manager, task_num; total_steps = steps, message = "Starting run with $steps steps")
    for step in 1:steps
        do_work(task_num, param_val, step)
        update!(manager, task_num; step = step, message = "step $step")
    end
    finish!(manager, task_num)
end

# Mark entire experiment as complete
finish!(manager)

Worker-based progress (threads or Distributed)

When work runs on other threads or processes, only the master touches the DB. Workers get a ProgressTask and can call update! every step without waiting on SQLite. Local tasks (:local) write an overwrite-latest slot in-process. Remote tasks (:remote) write a process-local slot and send the latest state over a RemoteChannel at most every 10 ms (and always on finish! / fail!). Load Distributed before requesting :remote tasks so the remote-worker extension is available:

using MultiProgressManagers
using Distributed  # for pmap

@everywhere using MultiProgressManagers

# Master: create experiment and get task handles
manager = ProgressManager("Distributed Run", 8; db_path = "./progresslogs/dist.db")
tasks = [get_task(manager, i, :remote) for i in 1:8]  # :local for @spawn threads

# Workers: send progress via the task handle (no DB access)
@everywhere function run_worker(task::ProgressTask, total_steps::Int)
    for step in 1:total_steps
        do_work(step)
        update!(task; step = step, total_steps = total_steps, message = "step $step")
    end
    finish!(task)
end

# Run and finish
pmap(i -> run_worker(tasks[i], 100), 1:8)
finish!(manager)

See examples/multithreading.jl (threads + ProgressTask via get_task(..., :local)) and examples/distributed_pmap.jl (Distributed + ProgressTask via get_task(..., :remote)).

Drill.jl training callbacks

Add Drill.jl to your environment, load it with using Drill, then build a training callback from a progress task (often a :remote task for distributed workers):

using MultiProgressManagers
using Drill
using Distributed  # when using :remote tasks

num_parallel_tasks = 8
manager = ProgressManager("my_study", num_parallel_tasks; db_path = default_db_path("my_study"))
task_index = 1
task = get_task(manager, task_index, :remote)
callback = create_drill_callback(task)

Viewing the Dashboard

From the shell: mpm ./progresslogs opens the dashboard for every .db file in that folder (requires the app; see Installation).
From Julia: using MultiProgressManagers; view_dashboard("./progresslogs") does the same.

Dashboard Tabs

Tab 1: Runs

Shows all experiments in the database:

• Experiment name and description
• Status (running/completed/failed)
• Overall progress across all tasks
• Start time and duration
• Automatically selects the newest experiment at the top of the list

Tab 2: Details (Running)

Shows detailed view of selected experiment:

• Individual task progress bars
• Task status (pending/running/completed/failed)
• Current step / total steps for each task
• Message column (from update!(...; message=...))
• Speed calculation (steps per second)

API Reference

Creating an Experiment

ProgressManager(name::String, num_tasks::Int;
                description::String = "",
                db_path::Union{String,Nothing} = nothing,
                task_descriptions::Vector{String} = String[])

Parameters:

• name: Human-readable experiment name (shown in dashboard)
• num_tasks: Number of parallel sub-tasks in this experiment
• description: Optional longer description
• db_path: Optional path to SQLite database file. When omitted, the package derives a default path from the experiment name.
• task_descriptions: Optional vector of per-task labels (length must equal num_tasks).

Returns: A ProgressManager instance for tracking this experiment.

Updating Task Progress (in-process)

update!(
    manager::ProgressManager, task_number::Int;
    step::Union{Int, Nothing} = nothing,
    total_steps::Union{Int,Nothing} = nothing,
    message::String = ""
)

Records progress for a specific task. Report progress by supplying step. Set total steps for the task with total_steps. Update the current task message with message. The message is shown in the dashboard Details tab. When total_steps is omitted, the previously stored total is reused. It is recommended to set total_steps at the beginning of a task. If total_steps is not supplied, it is updated automatically as step is updated. This path writes SQLite immediately; for hot loops on threads or Distributed workers, use a ProgressTask instead (see below).

Parameters:

• manager: The ProgressManager for this experiment
• task_number: 1-indexed task number (1 to total_tasks)
• step: Current progress step for this task
• total_steps: Optional; set it once and later updates may omit it
• message: Optional; shown in the "Message" column (e.g. phase or status)

ProgressTask: worker-based updates

When work runs on other threads or processes, workers must not call manager-side update! or touch the DB. Instead they use a ProgressTask:

get_task(manager::ProgressManager, task_number::Int, type = :local) -> ProgressTask

Returns a handle for one task. type:

• :local — per-task overwrite-latest slot (same process, e.g. @spawn or @threads)
• :remote — process-local overwrite-latest slot in front of a RemoteChannel (for Distributed / pmap)

update!(task::ProgressTask;
    step::Union{Int,Nothing} = nothing,
    total_steps::Union{Int,Nothing} = nothing,
    message::String = "",
)
finish!(task::ProgressTask)
fail!(task::ProgressTask; message::String = "Task failed")

Workers call update! during the loop and finish!(task) when the task is done. Frequent update! calls are cheap: they overwrite a per-task slot and do not wait on SQLite. A poller on the master process persists the latest per-task state to the DB (about every 10 ms). As with manager-side update!, total_steps only needs to be supplied when it changes or is first established. An empty message does not replace the last non-empty message.

Drill training integration

create_drill_callback(task::ProgressTask)

Returns a Drill callback that reports training progress through task. Load Drill.jl first with using Drill.

Finishing a Task (in-process)

finish!(manager::ProgressManager, task_number::Int)

Explicitly mark a task as completed. When doing multithreaded or distributed tasks, use finish!(task) on the ProgressTask instead.

Finishing an Experiment

finish!(manager::ProgressManager; message::String = "Completed successfully")

Mark the entire experiment as completed. This sets the experiment status to "completed" and marks all remaining tasks as done. Optional message is stored as the experiment's final message.

Failing a Task or Experiment

fail!(manager::ProgressManager, task_number::Int; message::String = "Task failed")
fail!(manager::ProgressManager; message::String = "Experiment failed")

Mark either a specific task or the entire experiment as failed with a message. The code also provides overloads that take an Exception or a positional error_message::String; these ultimately set the same message used in the DB.

Progress reporting performance

Same-process update!(manager, task_number; ...) writes SQLite on every call.

ProgressTask updates are designed not to stall compute:

• :local — update! only overwrites a per-task slot (typically around a microsecond). Callers never wait on SQLite or other workers.
• :remote — update! overwrites a process-local slot and sends at most one RemoteChannel message every 10 ms, plus a flush on finish! / fail!.
• The master poller copies dirty slots and writes the latest per-task state. Intermediate steps are not stored. An empty message does not replace the last non-empty message.

Call update!(task; ...) as often as you like. For threads or Distributed workers, always use get_task instead of manager-side update!.

See benchmark/ for AirspeedVelocity suites that compare these paths against a no-manager baseline.

Keyboard Shortcuts

Global

• q / Q: Quit
• r: Refresh (reload data from database)

Tabs

• 1 or F1: Runs tab
• 2 or F2: Details (Running) tab
• ← / h: Previous tab
• → / l: Next tab

Runs tab

• ↑ / ↓ or Ctrl+k / Ctrl+j: Move selection; the highlighted experiment is the one shown in the Details tab
• f: Mark selected running experiment as failed (opens confirmation modal)

Confirmation modal (after pressing f)

• Enter: Confirm (mark as failed) or cancel, depending on highlighted option
• ← / h or → / l: Switch between Cancel and Confirm
• Escape or c: Cancel and close modal

Details (Running) tab

• ↑ / ↓ or Ctrl+k / Ctrl+j: Scroll the task list
• a / d: Shrink or grow the message column width

Configuration

Database Location

If you omit db_path, the package stores the experiment under the default progress-log directory using a filename derived from the experiment name. Because each experiment gets its own DB file, duplicate names at the default path are rejected; pass an explicit db_path if you want to reopen an existing experiment file.

# Example: Create database in specific directory
manager = ProgressManager("My Experiment", 3; db_path = "./logs/run1.db")

License

MIT

Contributing

Contributions welcome! Please open an issue to discuss changes before submitting PRs with new features.