Alerts and notifications

Rules watch runs and fire notifications when their condition holds. You author them in the app on the organization's Alerts page.

Trigger kinds

Trigger	Fires when
`threshold`	A stream's latest sample satisfies a comparison — `gt`, `gte`, `lt`, `lte`, `eq`, `ne` — or `nonfinite` (NaN/Inf, the classic loss-explosion tripwire).
`absence`	A stream stops producing points.
`stale`	A run or participant stops heartbeating while still expected to be alive.
`session_lost`	A session ends without finishing — the crashed-trainer signal.
`event`	A named event occurs.
`schedule`	On a time schedule (for periodic checks).

Rules carry a cooldown_seconds so a flapping condition notifies once per window, not once per sample. session_lost dedups per session.

Windows and trends

A threshold trigger compares the latest sample unless you give it a span on the query's axis:

sustained_span — the breach must hold for this many steps, ending at the latest point: every point in that span must breach, and the history must reach back to the span's start.
min_points — the span must contain at least this many points.
deviation — compare to a trend instead of a fixed value: a straight line fitted to the points within baseline_span steps before the span, extended across it. A point breaches when it sits more than sigmas standard deviations of that fit above it (op: gt), below it (lt) or either side (ne). With fewer than three baseline points nothing fires.

A missing or non-finite latest sample never breaches, so when a metric stops the rule stays quiet; pair it with a stale rule for silence. The composer offers this as "moves away from its trend" and "For at least (steps)".

Checking a rule against history

Before saving, "Check against history" in the composer replays the condition over the run in scope, or the project's five newest runs, and lists where it would have fired: "run-a: would have fired once, at step 21,368." Nothing is sent and no rule is saved; the rule's query is saved (or an identical one reused) so the check reads exactly what the rule will. The API is POST /v1/organizations/{org}/rule-checks with a threshold trigger (its saved query included) and up to 20 run_ids. Each sample is judged as the evaluator judges the latest one, with the same windows and trends, and a firing is reported where a breach begins. Only the newest 10,000 points per run are read (truncated says when older history was skipped), and cooldown_seconds and sustained_seconds are wall-clock and not replayed. Other trigger kinds can't be checked.

Offering controls in an alert

A notify action can list up to four offers. Each names a control (a capability name) and can give a button label and an input object of at most 4 KiB. The alert shows each offer as a button on the run page. Pressing it opens that control's request form for the run the alert is about, with the input and the reason ("Alert: …") filled in. Nothing is sent until the person reviews it and presses Send, so the usual approval and permission rules apply. When the run has no control with that name, or the run isn't connected, the button is shown disabled with the reason. In the composer this is "Offer controls in the alert": it lists the controls registered by runs in the alert's scope, and each value is edited with that control's own form.

Silencing a rule

Anyone who can edit a rule can silence it for one run or for every run, for up to a week:

POST /v1/organizations/{org}/rules/{rule}/silences   {"run_id": "…", "seconds": 3600}
GET  /v1/organizations/{org}/rules/{rule}/silences

seconds: 0 lifts the silence for that run, or for every run when run_id is left out. While a silence holds, the rule does not fire for that run: its series are skipped, so other runs still alert. Other trigger kinds record the firing as suppressed with "suppressed": "silenced", and an event they would have fired on is used up rather than saved for later. Silences are kept when the rule is edited, end on their own, and each change is recorded as a rule.silenced event. On the run page, "Silence 1 h" on an alert silences its rule for that run and dismisses the alert.

Notifications from rules reference the run and stream they fired about (refs of kind run and stream), and the notification list takes ref_id to list one run's alerts.

Channels

in_app — the notification feed inside the dashboard.
email — a recipient with an unverified address falls back to in-app until they verify.
webhook — POSTs the firing payload to a URL for paging/chat glue.
push — a browser notification, to one browser per person. In Settings → Notifications, "Turn on in this browser" asks the browser for permission, subscribes it, and saves the subscription as your push address; "Send a test" sends you a test notification (POST /v1/organizations/{org}/notifications/test, at most one a minute per channel, and it ignores your severity floor and quiet hours). Clicking a push notification opens the organization's notifications. The server needs LAUNCHHELM_VAPID_PRIVATE_KEY and LAUNCHHELM_VAPID_SUBJECT; without them GET /v1/push-key reports push as unavailable and the settings page says so. On Android, Chrome and Firefox receive it. iPhones don't yet: Safari on iOS delivers push only to sites installed as Home Screen web apps, and LaunchHelm doesn't ship a web app manifest.

A working example

The most useful first rule: a threshold rule on train/loss with op nonfinite — fires the moment a NaN or Inf sample lands. Pair it with session_lost on the run and the two classic silent failures (NaN and crash) both reach you.