Skip to content
LaunchHelm

Telemetry and control for training runs

Observe the mission. Steer the outcome.

LaunchHelm charts your training runs as they log and lets you change them while they train. Add the Python SDK to your loop for live metrics, controls, checkpoints and alerts.

Create account Read the quickstart

pip install launchhelm

image-classifier run-024

Live
Step
21,000
Started
3h ago
Last update
Now
Controls applied
1

Controls · History

lr

Applied

“Loss plateaued”

  1. Requested you
  2. Allowed
  3. Picked up
  4. Applying
  5. Applied step 18000
lr
0.0005

Available controls

  • lr
  • weight_decay
  • grad_clip
  • dropout
  • checkpoint

Participants

  • rank-0 Connected
  • rank-1 Connected

01Live telemetry

See the run as it happens.

Metrics land in LaunchHelm the moment your loop logs them. Scalars chart per stream. Rich values render in place.

train/loss

2.48

at step 18000

optim/lr/group0

0.0005

at step 18000

perf/samples_per_s

1,284

at step 18000

Scalars

Loss, learning rate, throughput and host metrics on live charts, with nanosecond observation times.

  • null
  • missing

Faithful signals

Missing and explicit-null points stay visible as gaps. Smoothing is a view you turn on.

Exact data in view
step val/loss
3000 3.02
6000 Missing
9000 2.78
12000 2.61

Exact reads

Query any stream's stored points back in bounded pages, or let an agent read them over MCP.

text

epoch 3

loss 0.25, eval queued

media
table
trial lr loss
a 0.001 0.42
b 0.003 0.31
histogram

Rich values

Text, images, tables and histograms log from the same run and render beside its charts.

02Live control

Steer at the points you declare.

Your loop declares safe points and typed controls. LaunchHelm renders each one as a form. A change applies at the next safe point, between steps.

1You write a typed function

change_learning_rate.py Python 
class Schedule:
    def set_lr(self, lr: float) -> None:
        """Store the requested learning rate."""
        self.lr = lr

with launchhelm.init("quickstart", config={"lr": schedule.lr}) as run:
    run.command("after_step")(schedule.set_lr)
    for step in range(60):
        loss = 2.0 / (step + 1)
        run.log({"loss": loss, "lr": schedule.lr}, step=step)
        run.sync("after_step", step=step)

2The run's Controls tab shows the form

set_lr

Send a control
lr Number
0.0005
Reason
Loss plateaued

Applied the next time the run reaches “after_step”.

  1. Requested
  2. Allowed
  3. Pick up at after_step
  4. Apply
  5. Result

Every control ends in one recorded outcome.

An unknown outcome is never retried for you. The run keeps what was requested and what was applied, so the next move is your decision.

Control and tunables
Applied
The handler ran and reported success.
Rejected
The run refused it: bad input, wrong safe point or a closed run.
Failed
The handler raised.
Cancelled
Withdrawn before the run claimed it.
Outcome unknown
The result could not be recorded.

03Resilience

Lose a process, keep the run.

  1. Checkpoint saved step 64
  2. Session lost step 352
  3. Run resumed restored at step 64

Checkpoints that resume

One call saves model, optimizer, scheduler, step and RNG state. A relaunched process resumes the same run and restores the latest checkpoint.

Checkpoints and resume
  • train/loss nonfinite
  • eval/accuracy absence
  • run session_lost
  • In-app
  • Email
  • Webhook

Alerts that reach you

Rules watch streams and sessions: a NaN loss, a stream that stops producing, a session that drops. Notifications go to the in-app feed, email or a webhook.

Alerts and notifications

  1. Control applied

    step 18000

    lr · 0.0005

  2. Checkpoint uploaded

    step 64

    artifact · checkpoint

  3. Session started

    rank-0

A durable record

Points logged before a crash persist on the training machine and upload when the run resumes. Commands, artifacts and sessions stay attached to the run.

Concepts

04Integrations

Works where your training runs.

The SDK runs inside your process, on your machine or a cloud job. Use it in a plain loop, across distributed ranks or through a framework callback.

PyTorch

launchhelm.torch

Watch a model and optimizer once, then log, clip and apply controls inside tr.step. Tunables become dashboard fields.

  • lr
  • weight_decay
  • grad_clip
  • dropout
  • checkpoint

pip install "launchhelm[torch]"

PyTorch guide
torch_quickstart.py Python 
tr = lht.init("quickstart", name="torch-quickstart", config={"lr": 0.01})
tr.watch(model, optimizer)
dropout_p = tr.tunable("dropout_p", 0.0, minimum=0.0, maximum=0.9)

for step in range(20):
    optimizer.zero_grad()
    loss = torch.nn.functional.mse_loss(model(x), y)
    loss.backward()
    tr.clip_grad_norm_()
    optimizer.step()
    tr.step(step=step, loss=loss.item())
Distributed

DDP under torchrun

Every rank joins the same run as its own participant. A control applies on every rank at the same step.

  • rank-0
  • rank-1
Distributed training
torch_ddp.py Python 
tr = lht.init("quickstart", name="ddp-run", system_metrics=True)
tr.watch(model, optimizer)

for step in range(20):
    optimizer.zero_grad()
    loss = torch.nn.functional.mse_loss(model(x), y)
    loss.backward()
    optimizer.step()
    # step() is the collective control point: call it on every rank at
    # the same cadence.
    tr.step(step=step, loss=loss.item())
Hugging Face

Trainer callback

One callback opens the run, logs on every logging step, applies dashboard controls at step end and uploads each checkpoint save.

pip install "launchhelm[hf]"

Hugging Face and Lightning
hf_trainer.py Python 
from launchhelm.torch.hf import LaunchHelmCallback

trainer = Trainer(
    model=Tiny(),
    args=args,
    train_dataset=Rows(),
    callbacks=[LaunchHelmCallback("quickstart")],
)
trainer.train()
Lightning

Lightning callback

The same callback pattern for a Lightning Trainer. Your self.log calls flow through to LaunchHelm streams.

pip install "launchhelm[lightning]"

Hugging Face and Lightning
lightning_trainer.py Python 
from launchhelm.torch.lightning import LaunchHelmCallback

trainer = Trainer(
    max_epochs=1,
    callbacks=[LaunchHelmCallback("quickstart", name="lightning-example")],
    enable_progress_bar=False,
    enable_model_summary=False,
    logger=False,
    default_root_dir="/tmp/lh-lightning-example",
)
trainer.fit(Module(), train_dataloaders=DataLoader(Rows(), batch_size=4))
Hosts

System metrics

CPU, memory, disk, network and per-GPU stats stream alongside your metrics.

pip install "launchhelm[system]"

Examples
system_metrics.py Python 
import launchhelm
import launchhelm.system

with launchhelm.init("quickstart", name="system-demo") as run:
    launchhelm.system.start(run, interval=2.0)
    for step in range(8):
        run.log({"loss": 2.0 / (step + 1)}, step=step)

05Agents

Give an agent the same run.

LaunchHelm serves a Model Context Protocol endpoint at /mcp. Connect an agent client by signing in or with an API key.

  • Read only or full access

    Read only lets an agent browse projects, runs, metrics and controls history. Full access lets it send controls.

  • One project if you choose

    Limit a connection to a single project when you approve it.

  • Disconnect any time

    Connected apps are listed under Settings → API keys.

MCP guide

Connect an app

Organization
Vision Lab
Access
  • Read only
  • Full access
Project
image-classifier

Tools the agent can call

  • launchhelm.get_context read
  • launchhelm.list_documents read
  • launchhelm.run_query read
  • launchhelm.read_samples read
  • launchhelm.list_commands read
  • launchhelm.submit_command full access

06Workspace

One workspace for every run.

Projects, runs and people live in one organization. Allowances are shared across members.

baseline
0.713
lr-sweep
0.0703

Compare runs

Pick a metric to see where every run that logs it stands.

  • Member
  • Observer

Invite your team

Invite by email or share a link. Choose Member or Observer, set an expiry and limit a link to your email domains.

  • run-024 archive

Export and import

Download a run as an archive. Import an archive into a project.

  • Training runs
  • One project

Scoped API keys

Create a key for all projects or limit it to one. Each key is shown once.

07Get started

Log from your training host, steer from the dashboard.

  1. 1

    Create an API key

    Open Settings → API keys and create a key. It is shown once with the lines to export. Save it before you leave the page.

  2. 2

    Install the SDK

    pip install launchhelm on Python 3.11+ for macOS, Linux and Windows.

  3. 3

    Log a run

    The run appears in your project with its loss stream.

log_run.py Runnable 
import launchhelm

with launchhelm.init("quickstart", name="first-run", config={"lr": 0.0003}) as run:
    for step in range(8):
        loss = 2.0 / (step + 1)
        run.log({"loss": loss, "lr": 0.0003}, step=step)

Python

Logging, a durable local queue, safe-point commands, artifacts, checkpoints and the PyTorch adapter.

Python guide

More SDKs

TypeScript, Rust and C++ SDKs are coming soon.

Start

Your next run can be live.

Create an account, make an API key and run pip install launchhelm. Your run charts as soon as your loop logs.

Create account Read the quickstart Compare plans