[server] Orphan log segments from full truncation can prevent tablet server recovery

[swuferhong]

Search before asking

• I searched in the issues and found nothing similar.

Fluss version

main (development)

Please describe the bug 🐞

Repeated calls to truncateFullyAndStartAt may leave old active segment files on disk after they have been removed from the in-memory segment map.

During restart, LogLoader loads these orphan files as valid segments. If the segments belong to different log generations, writer-state rebuilding may encounter discontinuous batch sequences and throw OutOfOrderSequenceException, preventing the TabletServer from starting.

Root causes

1. LocalLog.createAndDeleteSegment deletes the old segment files only when
   the replacement uses the same base offset. For a different base offset, the
   old segment is removed from LogSegments but not deleted from disk.

2. Writer-state recovery still validates sequence continuity after writer state
   has been established. The recovery exemption only covers an initial batch
   whose current sequence is NO_BATCH_SEQUENCE; gaps such as 828 -> 1110
   still fail recovery.

Impact

• Orphan segment files accumulate indefinitely.
• Missing lazy indexes are repeatedly rebuilt during restart.
• Stale log generations may be loaded as current data.
• Writer-state recovery can fail with OutOfOrderSequenceException.
• TabletServer may enter a restart loop.

Expected behavior

• Full truncation must physically delete every removed segment.
• Recovery replay must not reject historical batches because of sequence gaps.

Solution

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!