§12. Executor and transaction kernel (crates exec, txn, watchdog)

planarian @ a924e09 · Spec v4.6 · snapshot main · all versions

Journal: append-only, write-ahead, fsync’d, in a preallocated file at /var/lib/planarian/journal/ (§12c). Read-only backend calls (current, boot_entries, last_booted, observe, stage, evaluate, build — build writes only to the Nix store and activates nothing, so it is treated as read-only with respect to system state, but its output is recorded) may precede Prepared. Every mutating call (activate_live, publish_boot, protect, restart, stop, snapshot, restore, file promotion) requires a durably written Prepared record and a current lease epoch. Prepared is written on the Allow path as well as the AskFirst path.

Ownership and fencing. Each transaction has exactly one owner at a time (Lease). Mutating backend calls carry the lease epoch; the backend layer reads the journal’s current epoch for that transaction and refuses the call if it differs (Outcome::Failed("fenced")). Mutations that take time (activation, revert) run in a worker (a thread in A, a subprocess recorded in the lease from D; see “Worker processes” below) that inherits the epoch. Taking over a transaction means: write a new lease with epoch + 1 (durably), stop the previous worker where one can be addressed (SIGTERM then SIGKILL to its process group via pidfd, from D), then establish confirmed quiescence. Every mutation attempt first journals a durable operation id (OpId, an OpBegin { txn, epoch, seq } record written and fsynced before the backend call, matched by an OpEnd with the outcome after it; “acknowledged” below means the OpEnd exists), and the backend call is tagged with it (for systemd: the job’s unit is started with a transient property X-Planarian-Op=<OpId> via SetUnitProperties in the same D-Bus transaction, or, where the call cannot carry a tag, the daemon records the pre-call snapshot of pending_jobs() as the op’s baseline). Quiescence then covers both the jobs in the old lease’s jobs list and every job discovered by jobs_tagged(op) for each journaled OpId whose acknowledgment was never recorded. Every such job must be cancelled (systemctl cancel <job> where the supervisor supports it) and confirmed gone from pending_jobs(), and any activation worker’s own child processes reaped; the new owner waits up to the job-drain limit (60 s) for that confirmation, and if quiescence cannot be confirmed, or jobs_tagged returns None for an unacknowledged OpId (the backend cannot say whether undiscovered work remains), it writes RecoveryFailed and blocks all further mutation rather than proceeding. A drain timeout is never permission to proceed, and neither is an empty list of known jobs. Gym case: accept an activation job, withhold its reply, kill the owner before the job id is journaled, start recovery; recovery must discover or fence the accepted work before any conflicting mutation. Only after confirmed quiescence may the new owner advance the journal. Lease transfer is atomic: the lease is a file /var/lib/planarian/txn/<txn>/lease replaced by atomic rename of a temp file whose name embeds the candidate epoch, after flock on the transaction directory; two processes attempting takeover serialize on the lock and the second sees the first’s epoch and backs off. An old owner that wakes up finds its epoch stale on its next call and exits. Only the current lease holder may advance the journal or commit. Gym case: kill and immediately restart the daemon while a deliberately slow activation worker survives; assert activation and revert never overlap and the old owner cannot commit afterward.

Protocol for a Proposal:

1. Observe S0. If proposal.observed != S0.digest() → Reject Stale, return the current digest.

2. Admission before work. Before any staging, evaluation or build, check what can be checked cheaply: the caller’s identity and capabilities against the transition’s authorization rule (§11), the catalog membership of an ApplyChange, and the recovery interlock (§12c). A caller who cannot be authorized for the transition causes zero stage/evaluate/build calls (Reject(Unauthorized) here). Then, for ApplyChange: run the handler, stage, evaluate, and build (read-only path); for SwitchGeneration/Revert/Rollback: dry-activate the target (§11). Derive facts from the evaluated candidate. Work in this step is charged to the caller and bounded (one build slot per caller class; §16a).

3. Refinement check: proposal.expected MUST agree with spec.predict() on every unit the spec predicts, MUST NOT broaden may_change, MUST NOT set others_unchanged = false where the spec sets true. Violation → Reject. Executor-owned requirements (§9) are appended. The transition’s RecoveryStrategy and revert_to are fixed here by the spec, not the proposer.

4. policy.decide (with authorization). Reject → log, alert if the source was the model. Otherwise create the transaction atomically: mkdir /var/lib/planarian/txn/<txn>/ (fails if it exists), write base.nix and the initial lease (epoch 1, this daemon) into it, then journal Prepared (durable) with facts, expected, revert_to, recovery strategy, handler, staged_source, base/candidate override hashes, gc_root, candidate path, policy version, and the lease. The transaction does not exist until the Prepared record is durable; the directory alone is garbage the next startup removes. AskFirst → request a token bound as in §8, park. On token: re-derive facts, re-run decide, recompute cumulative hash and candidate verification; any difference in bound_to invalidates the token and re-requests. Tokens expire in 24 h (CLOCK_BOOTTIME, §12h), are single-use, and do not survive a daemon restart (owner decision, 30 Oct): a restart invalidates every unused token, the parked change re-requests approval, and the checkpoint carries consumed nonces only, for replay protection. The approval is recorded: the token-use journal record and the transaction carry the approver’s PrincipalId, bound_to and the transaction id, and the approved execution writes its own log record (I7).

5. Check precondition against the current state (TransitionSpec::precondition; §11). A failed precondition writes Rejected(Precondition); an expired or invalidated approval that is not re-granted within its window, or a policy decision that changes to Reject on re-evaluation, writes the terminal Rejected(reason); all release the GC root. A user withdrawing a parked proposal writes Cancelled. All are terminal and reopen the incident if one was waiting.

6. (The lease was created with the transaction in step 4.) Snapshot if facts.sensitivity ∋ UserData and supported; record. If unsupported in Phase A, log NoSnapshot and continue; from C, AskFirst.

7. Execute, by recovery strategy, which is fixed per transition and independent of tier:

   Transition	Handler	Recovery	Notes
   RestartUnit, StopUnit, StopAndSuppress, SetOverride, LiftOverride	UnitOp / IntentOp	ReobserveOnly	candidate = None; never a generation operation
   SetRuntimeOption, CollectGarbage	UnitOp / GcOp	ReobserveOnly	same
   ApplyChange, SwitchGeneration, Revert, LoadModel	GenerationOp	RevertGeneration	full journal path below
   Rollback	GenerationOp	RevertGeneration	tier may be Runtime, but it is a generation operation: serialized (I6), owned, with candidate = known_good (or accepted_baseline), publish_boot, and override restoration to the override content recorded for that generation; its own revert_to is the generation it left
   BindDriver	DeviceOp	UnbindDevice	a device operation via the inventory backend (bind/unbind on the sysfs driver interface); no generation is built or activated; on failure or mismatch, unbind and rebind the previously bound driver (recorded at Prepared)
   RestoreSnapshot	SnapshotOp	RestoreSnapshotThenRevert	C

   • UnitOp / IntentOp / GcOp (ReobserveOnly): journal Activating → backend call in-process → Checking → settle → observe → compare → journal Committed with the outcome recorded, whether or not the expectation held; a mismatch informs the incident and never triggers generation reversion. A Partial/Unknown outcome means re-observe and record, nothing more. StopAndSuppress journals the suppression before calling stop.
   • GenerationOp (RevertGeneration), boot-only candidates (§12f: the change takes effect only at boot): verify_candidate → journal Publishing (durable, before any boot mutation) → publish_boot_once(candidate) as a one-shot next-boot entry, with the running generation (the one publishing the candidate; always at or above the security floor) kept as the default and fallback (owner decision, 30 Oct; known_good is a manual fallback per I1, never the automatic one) → journal AwaitingBoot. At the next boot, the candidate boots once; the admission check and the confirmation checkpoint of §10 apply: admitted → the candidate is made the default and the override promoted (§9 step 4) → Promoted; confirmed at 10 min → post-commit bookkeeping → Committed (boot-confirmed, known_good); not admitted, or the boot does not reach ready: the following boot falls back to the publishing generation → Reverted; admitted but not confirmed (canonical probes fail at the checkpoint) → Reverting: publish_boot(publishing generation) as default, restore the base override, the notice says a restart will complete it → Reverted (the machine runs the candidate until that restart; the daemon treats the window as a live-tested-but-unconfirmed generation). A daemon restart without a reboot leaves AwaitingBoot untouched and never re-prepares it.
   • GenerationOp (RevertGeneration), live: wait for an activation window if §12f requires one (the transaction stays in Built, bounded per §12f) → when the window opens: stale check, re-derive, re-decide (I8) → from B, confirm a passing sandbox run for this staged_source where §12g requires one → verify_candidate → journal Activating → worker: activate_live(candidate) → journal Checking → run applicable Readiness probes from both the candidate’s UnitDefs and the installed core, plus the expectation (per probe, §9) → pass: publish_boot(candidate) → journal Published → promote (§9 step 4) → journal Promoted → post-commit bookkeeping (§12e floor derivation, the committed generation’s override content, session accounting; part of the commit, I25) → journal Committed (same record marks live_tested). Fail: journal Reverting → worker: activate_live(revert_to) → publish_boot(revert_to) → restore base override (§9 step 4) → re-probe → Reverted, or RecoveryFailed if revert probes fail (alerts; stops all autonomous action until a human acts).
   • DeviceOp (UnbindDevice): journal Activating with the previously bound driver recorded → bind → Checking → probes → pass: Committed; fail: Reverting → unbind, rebind previous → Reverted/RecoveryFailed. Gym case: a failed driver bind restores the previous driver and attempts no generation rollback.
   • RestoreSnapshotThenRevert (RestoreSnapshot): restore, then as above.

8. Outcomes. Partial/Unknown are never success. In Activating with RevertGeneration: re-observe; candidate active → Checking; else → Reverting. In Checking: probes decide. In Published/Promoted on the live path: the candidate is already live and tested; recovery completes the commit (re-run publish_boot, re-run the promotion rename — both idempotent) rather than reverting, unless probes now fail, in which case revert. In Promoted on the boot-only path the candidate is admitted, not confirmed: recovery waits for the confirmation checkpoint of this boot and then commits or reverts as above; it never writes Committed before it. The watchdog may complete these steps because they are idempotent and were authorized by the transaction; it still cannot write Committed from any state before Checking.

9. Log (I7) with DerivedFacts, PolicyVersion, probe evidence, decision, outcome, lease epoch.

Crash-recovery table (what the lease holder does on finding a transaction in each state):

Gym cases: kill the daemon during a Runtime-tier Rollback (recovers as a generation operation) and during StopAndSuppress (recovers as a unit operation; no generation rollback attempted); an old backend job that outlives the 60 s drain (takeover ends in RecoveryFailed, never overlapping mutation); daemon and watchdog attempt takeover simultaneously (exactly one wins); daemon stays down after Published and after Promoted (watchdog completes the commit).

Recovery watchdog. planarian-recovery.service: separate unit, Restart=always, started before planariad, no dependency on it. Every 10 s it reads the journal; for a transaction in Activating/Checking/Publishing/Published/Promoted (live)/Reverting whose lease is unrenewed past deadline or whose owner pid is dead, it takes over the lease (fencing and confirmed quiescence as above) and runs the crash-recovery table. From Checking it does one of two complete things: if the authorized Readiness probes pass, it runs the entire authorized commit path (publish_boot → promote → Committed), because that path was authorized by the transaction and every step is idempotent; otherwise it reverts. It never stops partway. The earlier restriction is restated precisely: the watchdog may not write Committed from any state before Checking; from Checking, Published or Promoted (live) it completes the authorized path, bookkeeping included (I25). For a daemon that is alive and renewing its lease, the watchdog does nothing. Its binary and config are Sensitivity::Core.

Daemon restart reconciliation. On start, planariad reads the journal. If the watchdog holds a lease on a non-terminal transaction, the daemon waits until that transaction is terminal (it does not compete). Otherwise it takes over the lease (fencing) and runs the crash-recovery table. Either way it writes a reconciliation record (what it found, what it did, why) shown in the console on next open. Recovery steps that may activate a generation (a transaction found in Activating, Checking, Publishing, Published or Reverting) run just after planarian-ready.target is reached: the takeover and the reconciliation record come before it (owner decision, Nov 1). Activation reloads and restarts units ordered after the target, which would otherwise wait on planariad’s own start (#9).

Self-protection during activation. The NixOS units for planariad and planarian-recovery are declared with restartIfChanged = false and stopIfChanged = false, which NixOS’s switch honors by emitting X-RestartIfChanged=false/X-StopIfChanged=false, so no activation restarts or stops them. Independently, a candidate whose evaluated unit files for these two services differ from the running generation’s is touches_core → Reject; upgrading Planarian itself is the core update path (Phase E), and in A–D a manual nixos-rebuild by root outside Planarian. Networking changes set deadline to 90 s because loss of the daemon’s control path is likely. Boot ordering, readiness, and the response to the daemon failing are in §12d.

Concurrency: I6. Runtime transitions may run during build but not during Activating/Checking/Publishing/Published/Promoted/Reverting, except a boot-only Promoted, during which unit and device repairs run (owner decision, 31 Oct). This list and I6’s are the same list; schema carries it once.

Worker processes. v4.1–v4.5 described a worker subprocess whose pid the lease records. In A the activation runs in a worker thread and quiescence is established through jobs_tagged and the transient-unit tag alone, which is sufficient; the Lease.worker_pid and Lease.jobs fields are reserved for D’s supervisor and are empty in A (recorded in docs/deviations.md).