Orchestrator spec and reference impl

[rusty1968]

No description provided.

[leongross]

Do you plan to realize your state machine implementation based on https://github.com/typeconstructor/rot-reducer?

[chrysh]

@rusty1968 Should we also add multi-level cascades, concurrent faults, and double-fault recovery tests? Or is this not necessary?

[chrysh]

Are we trying to restore the golden image several times or just once?

[chrysh]

All the other crates have way fewer lines in lib.rs. Would it be better to put it into a separate .rs file?

[rusty1968]

The split is model.rs, tests.rs and lib.rs

[chrysh]

What if the passive device does not manage to boot up? How will we notice that?

[rusty1968]

CSA boot-progress checkpointing in 6d5f956

[chrysh]

Why does overflow mean a logic bug? Silently dropped does not sound like something I want in my security relevant device.

[chrysh]

Device should be in reset when rechecking. A live device is already executing code it loaded before the check, so the recheck says nothing about what's running. It can rewrite its own flash right after the check passes. Either the re-walk re-asserts reset on every non-held component first (recovery = platform re-boot), or re-checking live components becomes an explicitly weaker, differently-named operation instead of the same VerifyFirmware.

[chrysh]

Why do we always go back to the golden image? Should we try the last known good before update instead?

[rusty1968]

Your intuition is sharp here: the reason we trust golden image is because it is the one trustworthy - it comes from an immutable partition. Previous could perhaps buy us an availability optimization. When/if we have a stringent availability requirement, we can revisit this, but now perhaps is premature optimization.

[chrysh]

Advancing the SVN floor before we have a successful boot feels a bit early. If we only advance the SVN when we have a device that booted up, we could go back to the last version of the firmware if the boot fails. 5 years in, the golden image will be very old, so it could be a good idea to jump back to a more recent firmware version first?

[rusty1968]

Advancing the SVN floor before we have a successful boot is a real CSA compliance issue. Addressed in c353501

[chrysh]

What is a shell in this context? It's easy to confuse with bash or debug shell in the context of firmware. Is there a better word? platform layer vs board layer or so?

[rusty1968]

I changed it to platform driver.

[chrysh]

Is one failed and one retry_count enough for the whole platform? Should it be per device?

[rusty1968]

Yes, the retry budget is per component.

[chrysh]

Even if we only ever update one device at a time, §5.3.3 background polling can report corruption of device B while device A is mid-update or mid-recovery. What does the machine do then?

[chrysh]

Maybe each device should have their own state machine?

[rusty1968]

Even if we only ever update one device at a time, §5.3.3 background polling can report corruption of device B while device A is mid-update or mid-recovery. What does the machine do then?

Excellent question - The scenario is covered by tests:

• corruption_in_updating_triggers_recovery / corruption_during_update_discards_staged — if a device is found corrupt while another device is being updated, the update is dropped and the machine switches to fixing the corrupt device, throwing away the half-finished update image.

• corruption_while_recovering_retargets_to_new_component / restored_for_wrong_component_does_not_advance_recovery — if a second device fails while we're still fixing the first, we switch to fixing the second; and if the first one later says "I'm fixed," we ignore it, since we'd already moved on.

• cascading_runtime_corruption_cascades_transitively — when the failed device is one we're allowed to simply switch off (rather than one we must stop everything to fix), we isolate it and everything that depends on it, and keep going instead of interrupting the current work.

[rusty1968]

Maybe each device should have their own state machine?

Your intuition is directionally correct — but what genuinely matters here is per-device state , and the design already carries it (ComponentStatus, retry_budget, isolation markers,etc). What stays singular is the ordering: we only fix one device at a time, and we only update one device at a time. The machine keeps a single "who am I fixing right now" slot (State::Recovering(id)), so it can't be pointed at two devices at once. That one-at-a-time behavior is on purpose, not a limitation we ran into.

[rusty1968]

Do you plan to realize your state machine implementation based on https://github.com/typeconstructor/rot-reducer?

Correct. typeconstructor is another of my github accounts

[chrysh]

Maybe each device should have their own state machine?

Your intuition is directionally correct — but what genuinely matters here is per-device state , and the design already carries it (ComponentStatus, retry_budget, isolation markers,etc). What stays singular is the ordering: we only fix one device at a time, and we only update one device at a time. The machine keeps a single "who am I fixing right now" slot (State::Recovering(id)), so it can't be pointed at two devices at once. That one-at-a-time behavior is on purpose, not a limitation we ran into.

What happens if we get another update firmware request for device B while we are fixing the firmware if device A?

[rusty1968]

Maybe each device should have their own state machine?

Your intuition is directionally correct — but what genuinely matters here is per-device state , and the design already carries it (ComponentStatus, retry_budget, isolation markers,etc). What stays singular is the ordering: we only fix one device at a time, and we only update one device at a time. The machine keeps a single "who am I fixing right now" slot (State::Recovering(id)), so it can't be pointed at two devices at once. That one-at-a-time behavior is on purpose, not a limitation we ran into.

What happens if we get another update firmware request for device B while we are fixing the firmware if device A?

TL;DR

The UpdateRequest is silently dropped. Recovery is not interrupted, no effect is emitted, and nothing is queued. The requester gets no response at all.

That drop is deliberate as a policy (recovery outranks update — "fix first, then update"), but the silence is the part worth a second look: the requester cannot tell "refused, busy recovering" apart from "lost."

CSA Alignment

The CSA mandates that failures be reported and models updates as answered requests, which is exactly the gap I am fixing with 57fedb0

This is a sketch of how we use the report effect.

impl Platform for PlatformAst  {
    fn execute(&mut self, effect: Effect) -> Result<(), EffectError> {
        match effect {
            // ... device I/O effects (ReadFirmware, ReleaseReset, ...) ...
            Effect::ReportIsolated(id)       => self.mgmt.report_isolated(id),
            Effect::ReportRecoveryFailed(id) => self.mgmt.report_recovery_failed(id),
            // Pure protocol reply — no device I/O. Answer the pending PLDM
            // FW-update request so the requester gets a definite "busy, retry".
            Effect::ReportUpdateDeferred =>
                self.pldm.respond_update(PldmCompletion::RetryLater),
            // ...
        }
    }
}

The CSA does not mandate update-vs-recovery arbitration. Recovery is described as autonomous and separate from update; the interaction of an in-flight recovery with an incoming update is left to the implementation. Our single-flight, recovery-priority choice is therefore a legitimate implementation-defined policy, neither required nor forbidden by the CSA.

[chrysh]

@rusty1968 Should we add those test?:

/// INV9: the iRoT gate must hold even when the eRoT walk finishes first.
/// C0's `ComponentReady` is still outstanding when the last verdict
/// arrives; completing the walk must not bypass the gate.
#[test]
fn chain_exhaustion_does_not_bypass_irot_gate() {
    let (_effects, state) = drive(
        chain(&[
            (C0, ComponentAttrs::active_required()),
            (C1, ComponentAttrs::passive_required()),
        ]),
        &[
            BOOT,
            Event::VerificationPassed(C0), // C0 released; its iRoT still owes ComponentReady
            Event::VerificationPassed(C1), // walk done — but C0 never reported
        ],
    );
    assert_eq!(state, State::AwaitingReady(Some(C0)));
}

/// Trust invariant: a verdict only means something for the component under
/// verification. A replayed `VerificationPassed` for a component that has
/// since been isolated must not re-release it.
#[test]
fn replayed_verdict_does_not_release_isolated_component() {
    let (effects, state) = drive(
        chain(&[
            (C0, ComponentAttrs::passive_isolable()),
            (C1, ComponentAttrs::passive_required()),
        ]),
        &[
            BOOT,
            Event::VerificationPassed(C0), // released; walk moves to C1
            Event::CorruptionDetected(C0), // Isolable → AssertReset + Isolated
            Event::VerificationPassed(C0), // replayed / spurious verdict
        ],
    );
    let releases_c0 = effects
        .iter()
        .filter(|&&e| e == Effect::ReleaseReset(C0))
        .count();
    assert_eq!(releases_c0, 1, "isolated component was re-released");
    assert_ne!(state, State::Ready, "C1's verdict is still outstanding");
}

/// A healthy Active component must not be dragged into recovery by a stale
/// boot-progress watchdog. After a re-walk, C0's `ComponentReady` arrives
/// while the machine is back in `PreSupervision`; the report must clear its
/// watchdog (as `Booted` does there), making the later `Timeout` stale.
#[test]
fn component_ready_during_rewalk_clears_watchdog() {
    let (effects, state) = drive(
        chain(&[
            (C0, ComponentAttrs::active_required()),
            (C1, ComponentAttrs::passive_required()),
        ]),
        &[
            BOOT,
            Event::VerificationPassed(C0), // C0 released, watchdog armed
            Event::VerificationFailed(C1), // C1 fails → Recovering(C1)
            Event::Restored(C1),           // restore ok → re-walk from top
            Event::ComponentReady(C0),     // C0's iRoT reports in — discarded
            Event::Timeout(C0),            // stale watchdog fires
        ],
    );
    assert!(
        !effects.contains(&Effect::RestoreGoldenImage(C0)),
        "healthy, ready C0 was sent to recovery by a stale watchdog"
    );
    assert_ne!(state, State::Recovering(C0));
}