Reference

Troubleshooting

Errors you can hit, and the fix for each.

LaunchHelmError: unauthorized / forbidden

The API key is missing, expired, revoked, or limited to another project. Check the LAUNCHHELM_* variables are set in the process environment (print the names, not the values) and that the key is listed under Settings → API keys. A key limited to one project needs LAUNCHHELM_PROJECT set to that project's UUID, not its name.

invalid scope when querying or resuming

You passed a run id without the project it belongs to, or an org id that does not match the key's. Set LAUNCHHELM_PROJECT to the run's project.

forbidden on an artifact or checkpoint upload

Your plan doesn't include artifact uploads. Check Settings → Plan & usage in the app.

Run did not appear / closed run conflicts

init("name") reuses an existing project of that exact name. resume= needs a run id or exact title in that project. If a previous process was killed without finish(), the run may still show open until its session expires — resume it with resume= and the run id.

A command shows rejected

The runtime refused it locally: the input failed the capability schema, the safe point did not match, or the run was closed. The run's Controls tab shows the rejection reason. Fix the input and reissue — rejected commands were never applied, so reissuing is safe.

outcome_unknown

The result was not recorded — the handler may have run. Check the command's detail before reissuing it. In DDP this raises DivergedStateError on every rank; fix the handler and rerun the job.

DivergedStateError in distributed runs

A control applied on some ranks but not all. With the default on_partial_failure="raise" every rank fails together so you rerun the job. on_partial_failure="continue" exists for experiments where divergence is tolerable.

Notifications not arriving by email

• Recipients need a verified email address; unverified recipients get in-app notifications instead.
• A flapping condition is rate-limited by the rule's cooldown_seconds.

torch errors on import launchhelm

The base package never imports torch — torch errors come from launchhelm.torch; check that import instead.

No system metrics

launchhelm.system without psutil/NVML logs one status line and continues. Install the extra: pip install "launchhelm[system]".

CheckpointError on an FSDP model

save_checkpoint expects a full state dict — configure a full state dict first, or save one state dict per rank shard yourself.

Telemetry missing after a crash

Points recorded by run.log() upload asynchronously. run.flush(delivered=False) persists them on the local machine (default state_dir .launchhelm/), and they upload on the next start that reuses it. Keep state_dir stable between retries and call finish() at exit — it waits for delivery.

pip install launchhelm fails

• Python older than 3.11 — pip reports No matching distribution found or requires a different Python. Check python --version, then install under 3.11+ (python3.11 -m pip install launchhelm, or recreate the environment).
• Out-of-date pip — python -m pip install -U pip, then retry.
• Unquoted extras — pip install launchhelm[torch] treats the brackets as a shell glob (zsh fails with no matches found). Quote them: pip install "launchhelm[torch]".