Apply blocks at production round entry

[heifner]

Summary

Companion to #320 (strong-QC lockout fix). Where #320 handles the case where our applied head is on a different branch from the one that will win fork-choice, this PR handles the case where our applied head is on the same branch but behind it -- i.e., blocks from the canonical chain are already in fork_db but haven't been applied by the time our producing slot starts.

Plus a follow-on optimization to controller::is_head_descendant_of_pending_lib since it sits adjacent to this work, and a latent timer-correlation race in schedule_delayed_production_loop that becomes load-bearing once the apply loop runs in producing mode.

This PR is stacked on #320; merge target will rebase to master after #320 lands.

Why now -- continuing the Vaulta diagnosis

Incident 2 in the Vaulta operator's reports showed exactly this case. When titan's slot ended, blocks 408-411 were in fork_db on the canonical branch by 41.567, but the BP's applied head was still at 408 (apply was interrupted by the next block's arrival, and in_producing_mode then suppressed the retry). The BP entered its own slot at 41.962 with chain.head() == 408 and fork_db_head == 411, both on the canonical chain. Production proceeded on the stale head; all 12 produced blocks orphaned at slot end.

The strong-QC lockout fix in #320 doesn't trigger here -- titan's branch is our branch, just further along, so locks_out_branch_of correctly returns false. We need a separate condition for "same chain, behind."

Mechanism

producer_plugin::start_block already has a process_pending_blocks loop that fires apply_blocks when in_speculating_mode or when our head is not a descendant of pending_savanna_lib (an off-canonical-chain check). This PR extends the disjunct with a third condition: fork_db_head is ahead of chain.head() and fork_db_head.extends(chain.head().id()) (same branch, behind). When that fires, the existing time-bounded apply runs once, head advances, and start_block proceeds normally on the now-current head.

The bail-out condition is widened: the loop now exits whenever apply_blocks reports zero blocks applied, regardless of whether the status is complete (queue empty) or incomplete (the block at the front of the queue couldn't complete within deadline -- e.g. an infinite trx). Re-entering the loop in that case would just hit the same wall while starving the main thread of net_plugin and fork-choice updates that could resolve the situation. In producing mode the bail falls through to produce_block, building a competing block at the same height; under Savanna fork choice the producer's competing block wins by timestamp once committed and the unapplyable block is orphaned. interrupt_trx_test exercises this path.

Adds block_handle::extends(block_id_type) -- a thin wrapper around finality_core::extends. Used both by the new producer_plugin condition and by the optimization below.

Latent race in schedule_delayed_production_loop

Pre-existing bug, latent on master: schedule_delayed_production_loop checks _timer_corelation_id in its timer callback before posting schedule_production_loop to the executor, but the posted lambda itself doesn't recheck. Between the timer callback firing and the executor running the post, another scheduling call (e.g. schedule_maybe_produce_block) can bump _timer_corelation_id. The stale post then runs anyway, calling schedule_production_loop -> start_block -> the next deadline-interrupt schedule -> another cid bump -- which preempts the just-set produce_block timer (whose own posted lambda does check cid and skips).

schedule_maybe_produce_block already mirrors the cid check in its posted lambda. Apply the same pattern to schedule_delayed_production_loop.

Latent because today the apply-blocks loop in start_block doesn't fire in producing mode, so the deadline interrupt's schedule_production_loop post doesn't race against schedule_maybe_produce_block. The new same-chain-behind condition makes that combination reachable. Without this fix, interrupt_trx_test deadlocks: every apply attempt on the bad block is deadline-interrupted, the producer never reaches produce_block, and the chain stalls.

Savanna context: the drop-late-blocks property is gone

Spring/EOSIO has historically had a "drop late blocks" property: under DPoS, the BP that propagated late was the one that suffered, because the next BP would refuse their blocks and produce on top of their own (longer-by-chain-length) view. Timing accountability stayed with the offending BP.

Savanna fork choice does not have this property. Whether a chain wins is decided by the finalizer set's QC voting, not by chain length at any individual node. By the time titan's late chain has a strong QC, no local action by the next BP can recover the slot -- the only choices are "apply and accept the loss" or "don't apply and add a fork-switch loss on top."

This change accepts that trade-off: it can't restore the drop-late-blocks defense (which would require a protocol change), but it minimizes the local damage by ensuring the BP at least produces on the right chain when it does produce. Same-chain-behind is the most clearly-this-is-pointless case -- there's no alternative branch we'd be sticking with by refusing to apply, we're just sitting on an older block of the chain everyone else already has.

Optimization: is_head_descendant_of_pending_lib via core.extends

The previous implementation walked the fork_db index from chain_head following previous() links until it hit pending_savanna_lib_id (or went below it), under the fork_db mutex -- O(N) in head-vs-pending-lib distance with mutex contention.

Reimplemented using chain_head.extends(pending_id), which reads the canonical block_ref at pending_lib's height directly out of the head's finality_core. The core tracks block_refs across [last_final_block_num(), current_block_num()) on the chain leading to the head, so extends answers the same question in O(1) without any fork_db lock. The head == pending_lib edge case is handled with an explicit equality check.

Removes the now-unused fork_db.is_descendant_of_pending_savanna_lib (and impl) since this was its only caller.

Consensus invariants

1. No new signed artifact -- only chain.head() advances (it walks blocks already in fork_db, which got there via the network-validated accept_block path).
2. No fork-choice override -- fork_database's comparator still picks the branch; this code only governs whether we apply blocks on the chosen branch promptly.
3. No finalizer voting change -- consider_voting fires per applied block as it does today.
4. Time-boundedness preserved -- apply_blocks uses the existing slot deadline.
5. Schedule transitions during apply are handled by the existing re-determine path: if the apply advances head past a producer schedule change that reassigns our upcoming slot, determine_pending_block_mode will correctly transition to speculative and we won't produce in someone else's slot.

Spring

Same change applies cleanly to Spring; not opening that PR here. Spring's maintainers may decide independently whether and when to adopt.

[brianjohnson5972]

Looks good, a few comments, nothing needed though

[brianjohnson5972]

From Claude:
schedule_delayed_production_loop cid recheck (producer_plugin.cpp:2880-2890)

Mirrors the existing pattern in schedule_maybe_produce_block at line 2898-2906. Correct fix for the described race. The PR description's explanation of why this becomes
load-bearing now (the apply-loop deadline-interrupt path becomes reachable in producing mode) is convincing.

Related issue not addressed: schedule_production_loop's own failure-retry path at line 2832-2838 posts unconditionally without an inner cid check. It has the same shape
and could in principle suffer the same race, though the retry interval (block_interval/10) makes it less likely to interleave. Worth flagging as follow-up.

[brianjohnson5972]

Looks good, but there is one nit:
_timer_corelation_id should be _timer_correlation_id
but doesn't have to be done