Control and tunables

Commands mutate a running job only at safe points you declare — never inside a step, never across threads.

Built-in controls (`launchhelm.torch`)

tr.watch(model, optimizer, scheduler=None, scaler=None, eval_fn=None) registers:

Control	Input	Effect
`lr`	`{scale > 0}` or `{lr > 0}`, optional `group`	Scales or sets the learning rate; scales the watched scheduler's base/initial/max lrs proportionally so schedules stay coherent.
`weight_decay`	`{value >= 0}`, optional `group`	Sets weight decay.
`grad_clip`	`{max_norm > 0}` or `{max_norm: null}`	Sets or lifts the cap `clip_grad_norm_()` applies.
`dropout`	`{0 <= p < 1}`	Sets `p` on every `nn.Dropout*` module; returns the count changed.
`checkpoint`	`{}`	Saves and uploads a checkpoint; returns the artifact id.
`rollback`	`{checkpoint}`, a verified checkpoint artifact of the run	Saves where the run is, then restores that checkpoint on every rank in place: model, optimizer, scheduler, scaler, step, RNG, tunables. Training goes on with the next batches (the data loader is not rewound). Recorded as a rewind on the run's charts; "Undo this change…" restores the checkpoint saved first. A checkpoint that can't be read is refused before anything changes.
`pause`	optional `{max_seconds}`, 1 to 86,400, default 1,800	Holds every rank at the step boundary inside `tr.step()`. The run keeps its session and still takes controls (captures at `after_step`, checkpoints, rollbacks, lr). Resumes by itself after `max_seconds`. Collectives on the control group stay short, so a long pause doesn't trip their timeout.
`resume`	`{}`	Ends a pause; training continues with the next step.
`eval`	`{}`	Runs your `eval_fn` on all ranks, logs metrics under `eval/`.

Tunables

tr.tunable("name", default, minimum=..., maximum=...) for numbers or bools, choices=[...] for strings. Read .value in the loop; the change applies on all ranks at the same step. Each tunable registers a set_<name> command.

Custom commands

run.command(safe_point) turns a typed function into a UI-issuable command — annotations become the input form:

class Schedule:
    """The learning rate this fake loop is using."""

    def __init__(self) -> None:
        self.lr = 0.0003

    def set_lr(self, lr: float) -> None:
        """Store the requested learning rate."""
        self.lr = lr


schedule = Schedule()
with launchhelm.init("quickstart", config={"lr": schedule.lr}) as run:
    run.command("after_step")(schedule.set_lr)
    for step in range(60):
        loss = 2.0 / (step + 1)
        run.log({"loss": loss, "lr": schedule.lr}, step=step)
        run.sync("after_step", step=step)
        # Real steps take seconds; the pause leaves a window to issue
        # set_lr from the run's Controls tab.
        time.sleep(0.5)

python-change_learning_rate.py · verbatim from sdks/python/examples/change_learning_rate.py

For richer control use run.on(safe_point, capability, handler) with an explicit Capability — typed input and output schemas, which the UI renders as a form. A handler receiving a context: CommandContext parameter gets the safe-point coordinates (the step it applied at). The experiment example under Examples shows two commands sharing one safe point.

Issuing from the UI

Open the run's Controls tab, pick a capability, fill the form, submit. The command is claimed by the run, applied at the next sync for that safe point, and lands in a terminal outcome: applied, rejected, failed, cancelled or outcome_unknown. run.pending(safe_point) checks locally whether one is waiting (used by distributed collectives).

Rules that matter

sync runs at most one waiting command per call, in claim order.
Handlers run only at their declared safe point, on the training thread.
outcome_unknown is never retried automatically — see command outcomes.
In DDP every rank must reach tr.step at the same cadence; the control protocol is collective. See distributed.