Hub
Docs

Plesty Documentation

Runs, Data & Resume

What an experiment run persists — plan, journal, per-step results — and how crash-safe resume works.

Every run() creates its own checkpoint directory under run_root:

<run_root>/<run_id>/
├── plan.json        # frozen schedule + config + content hash
├── journal.jsonl    # append-only event log (status, resume source)
├── records.jsonl    # one line per completed step: its result document
└── data/            # blobs the records reference (arrays, images)
    └── step_0003.npy

The run id is derived from the experiment name and a timestamp, e.g. demoexperiment_20260708-101500.

run_root defaults to PLESTY_DATA_MOUNT when a share is configured, and to ./runs otherwise — so on a real bench the runs land on the share beside the frames.

The journal

The journal is a JSON-Lines file — one event per line — recording the life of the run: run_started, step_started, step_completed, step_failed, run_canceled, run_aborted, run_completed. Every append is flushed and fsynced, so the journal survives crashes; at worst the final line is torn, which replay tolerates by skipping it.

There is no mutable "status" file: the current run status is always derived by replaying the journal — exactly like reconstructing training progress from a deep-learning run log.

The records

records.jsonl is the measurement. One line per completed step, appended only once that step is fully persisted — so the write is the commit: a line exists, or the step did not finish. That property is what lets a monitor tail the file while the run is still going (see Live Views).

Each line is a result document carrying the step's value and its provenance verbatim — step id, operation, parameters — so a row is traceable to the exact plan step that produced it without opening the plan beside it.

What the value holds depends on what the step returned:

  • any JSON-serializable value → inlined in the line itself
  • a PlestyArray → written to data/step_NNNN.npy, with the line carrying the path plus name, unit, description and shape
  • encoded bytes (a camera image) → the same way, a binary blob beside the line that references it

Frames a device writes are different again: they never travel back over the network at all, so the record carries only the path on the share. See Runs & Data for that rule and the two share spellings it requires.

Runs written by plesty-lib ≤ 0.3.4 used one data/step_NNNN.json per step and no records.jsonl. They are still readable — the library detects the older layout — but new runs are written as above.

Resume

uv run python -m plesty.scan_exp --resume demoexperiment_20260708-101500

On resume, the base class:

  1. Rebuilds the plan with build_plan() and compares its content hash against the persisted plan.json — a mismatch (changed config or schedule) is refused with PlanMismatchError
  2. Replays the journal to find the completed step ids
  3. Skips completed steps and continues from the first unfinished one, appending to the same journal

Retries

A failed step is retried up to max_retries times (constructor argument, default 3), sleeping retry_sleep seconds between attempts. Only when a step exhausts its retries does the run abort — journaled as run_aborted, ready to be resumed after the cause is fixed.