Weco Observe

Weco Observe lets you track experiments from any optimization loop - whether it's an LLM agent running autoresearch, a custom training script, or a manual experimentation workflow. Your experiments appear in the same Weco dashboard as regular runs, with tree visualization, code diffs, and metric tracking.

When to use Observe

Use weco observe when you're running your own optimization loop and want to track results in the Weco dashboard. This is different from weco run, which drives the optimization itself.

Quick start

First, authenticate if you haven't already:

weco login

Use the weco skill to monitor the experiments.

WECO_RUN_ID=$(weco observe init \
  --name "my-experiment" \
  --metric val_bpb \
  --goal min \
  --source train.py)

# Log baseline result (step 0)
weco observe log \
  --run-id "$WECO_RUN_ID" \
  --step 0 \
  --status completed \
  --description "baseline" \
  --metrics '{"val_bpb": 2.366, "memory_gb": 0.0}' \
  --source train.py

# Log an experiment
weco observe log \
  --run-id "$WECO_RUN_ID" \
  --step 1 \
  --status completed \
  --description "increase batch size to 32K" \
  --metrics '{"val_bpb": 2.261, "memory_gb": 0.0}' \
  --source train.py

# Log a failed experiment
weco observe log \
  --run-id "$WECO_RUN_ID" \
  --step 2 \
  --status failed \
  --description "double model depth (OOM)" \
  --metrics '{"val_bpb": 0.0, "memory_gb": 0.0}' \
  --source train.py

CLI reference

weco observe init

Create a new external run for tracking.

Prints the run ID to stdout so it can be captured with $().

weco observe log

Log a step (experiment) to an existing run.

Branching and tree structure

By default, each step automatically chains to the last successful (non-failed) step, forming a linear sequence. When you discard a failed experiment and try a different approach, use --parent-step to branch correctly:

# Step 2 is the current best
# Step 3 failed - we revert to step 2 and try something different
weco observe log --run-id "$WECO_RUN_ID" --step 3 --status failed \
  --description "tried X, didn't work" --metrics '{"val_bpb": 2.5}'

# Step 4 branches from step 2, not the failed step 3
weco observe log --run-id "$WECO_RUN_ID" --step 4 --parent-step 2 \
  --status completed --description "tried Y instead" --metrics '{"val_bpb": 2.1}'

This produces a tree in the dashboard:

[Baseline (step 0)]
  +-- Step 1 (kept)
        +-- Step 2 (kept)
              +-- Step 3 (failed)
              +-- Step 4 (kept, branched from 2)

Python SDK

For scripts with a Python loop, use the SDK directly instead of shell commands:

from weco.observe import WecoObserver

obs = WecoObserver()
run = obs.create_run(
    name="sweep v3",
    source_code={"train.py": open("train.py").read()},
    primary_metric="val_bpb",
    maximize=False,
)

for i, result in enumerate(experiments):
    run.log_step(
        step=i,
        status="completed" if result.kept else "failed",
        description=result.description,
        metrics={"val_bpb": result.val_bpb, "memory_gb": result.memory_gb},
        code={"train.py": open("train.py").read()},
    )

Run lifecycle

External runs are managed by the dashboard, not the CLI:

• Runs stay active as long as steps are being logged
• The dashboard shows "Latest step X ago" for running external runs
• Mark as Complete from the dashboard actions menu to close a run
• Logging a new step to a closed run silently reopens it
• Runs with no activity for 24 hours are automatically archived

Design notes

• Step 0 is the baseline - created automatically by init with your source code. Log step 0 after your baseline run to attach metrics.
• Idempotent - re-posting the same step number updates the existing step instead of creating a duplicate.
• Multi-metric - the --metrics JSON can contain any number of metrics. The primary metric (specified in init) is used for the chart and "best metric" calculations.
• Code diffs - pass --source on each step to see code diffs between experiments in the dashboard.