CI flake: Test Suite / io::fallocate::local_durability_crashes_on_new_segment_if_not_enough_space

[clockwork-labs-bot]

Detected by CI Gremlin after a failed GitHub Actions attempt succeeded on rerun.

Tests:

• io::fallocate::local_durability_crashes_on_new_segment_if_not_enough_space

Error:
thread 'tokio-runtime-worker' (237979) panicked at /home/runner/actions-runner/_work/SpacetimeDB/SpacetimeDB/crates/durability/src/imp/local.rs:289:18:
commitlog write failed: Os { code: 28, kind: StorageFull, message: "No space left on device" }

Failed job runs:

• https://github.com/clockworklabs/SpacetimeDB/actions/runs/29014698368/job/86107482262 (Test Suite, attempt 1 -> 2)

Timeout detection:

• none

Context:

• none

[clockwork-labs-bot]

Diagnosis for the linked job attempt:

I inspected only the exact linked failed job: run 29014698368, job 86107482262, Test Suite, attempt 1.

The job failed in the fallocate durability pass:

cargo test -p spacetimedb-durability --features fallocate -- --test-threads=1

The failing test was:

io::fallocate::local_durability_crashes_on_new_segment_if_not_enough_space - should panic

This test intentionally creates a 512 MiB loop-mounted filesystem, opens local durability with 256 MiB commitlog segments, writes enough transactions to force a new segment, and expects a later append to panic with local durability: actor vanished after the durability actor crashes.

In this attempt, the actor did hit the expected ENOSPC path while creating/writing the new commitlog segment:

thread 'tokio-runtime-worker' (...) panicked at crates/durability/src/imp/local.rs:289:18:
commitlog write failed: Os { code: 28, kind: StorageFull, message: "No space left on device" }

Immediately after that, the test logged:

[INFO  main::io::fallocate] trying append on crashed durability
note: test did not panic as expected at crates/durability/tests/io/fallocate.rs:76:10

Root cause for this exact attempt: the test reached the intended low-disk condition and the durability actor crashed on StorageFull, but the caller-visible append after the actor crash did not panic as the #[should_panic = "local durability: actor vanished"] test expects. So the flake is in the durability/fallocate crash-propagation test behavior, not runner disk exhaustion and not a generic CI setup failure.

Suggested title:

CI flake: fallocate durability actor hits ENOSPC but append does not panic

Duplicate check:

Likely duplicate:

• CI flake: local_durability_crashes_on_new_segment_if_not_enough_space did not panic #5231: same test, same final harness failure: local_durability_crashes_on_new_segment_if_not_enough_space did not panic as expected.

I did not find another closer Test Flakes duplicate. The No space left on device line is part of this test's intentional setup; #5231 is the top duplicate candidate.