§13. Incidents and the diagnoser (crates incident, diagnose)

planarian @ a924e09 · Spec v4.6 · snapshot main · all versions

Incident lifecycle

• Open when a unit is expected (desired condition holds, no override) and its derived status is Failed, Degraded, Stopped-but-desired-Running, Detected-but-desired-Bound, or Absent-but-desired-present, with fresh evidence for two consecutive ticks.
• Defer to systemd first. If the unit has Restart= and systemd is within its own StartLimitBurst, the incident waits for systemd’s attempt (up to the unit’s RestartSec plus settle) before Planarian acts, so two controllers don’t restart the same service.
• Attempts are transactions; each outcome is appended. Budget 3 per incident, persisted in the journal (survives daemon restart). Backoff between attempts: 10 s, 60 s, 300 s.
• Closed when the target satisfies its desired predicate (Running, Bound, Stopped, or Absent as desired) with fresh evidence for a stabilization window (5 min).
• Regressed: if the same target diverges again within 30 min of Close, the incident reopens with count+1 and a longer backoff; at count = 3 it goes straight to Escalated (report to user; cooldown 1 h; no autonomous action on that target until a human acts or the cooldown ends).
• Cooldown after budget exhaustion: 1 h, then one more attempt, then 6 h.
• An override on the target closes the incident (I13).
• A person acting on the target (release, override, stop, a manual repair) reopens the incident with a fresh attempt and no cooldown — but only when the action was authorized and succeeded. A request that is refused, parked, or fails leaves the incident record exactly as it was: budget, cooldown, escalation and next attempt unchanged. Rejection grants no indirect control over the repair loop. Gym case: an unauthorized planarian release sshd during sshd’s cooldown changes nothing about the incident.
• Quiet mode. While Foreground holds, incidents on units the foreground work does not depend on wait (§12f); their budgets and backoff do not advance while they wait.

Diagnoser

1. From the open incidents, walk Requires edges (and Diagnostic edges with confidence ≥ 0.7) to the deepest divergent unit; that is the target. Ordering and Wants edges are never followed: a failed wanted unit is its own incident and is never assumed to be the cause of the wanting unit’s failure. Cycles: the walked subgraph is first collapsed into strongly connected components; “deepest” is defined on the resulting DAG; within a component, the target is the unit with the earliest definite Observed::Failed timestamp, else the one with the most failed Readiness probes, else lexical order. Gym case: an optional wanted service fails while its wanting service stays healthy (two incidents at most, no false root cause).
2. Enumerate candidate transitions, depth ≤ 2, whose precondition holds and whose prediction closes the divergence. Exclude exhausted and cooling targets.
3. Order by: tier ascending, then disruption ascending, then recovery (Idempotent < Revertible), then cost. Rollback last.
4. Submit one at a time as Proposal { source: Diagnoser }. Stop at the first Committed whose expectation held.
5. Exhausted → DiagnoserExhausted → model (B) or user report (A).

Phase A ApplyChange candidates are limited to catalog entries tagged repair: (re-enable a predicted option; reapply a previously-committed change for the same target — the latter is in the catalog from v4.6; it was specified and not shipped in v4.5).