[nexus] instance update saga: retry "inherit lock", too #10801
Conversation
Similarly to #10166, a duplicate execution of the `siu_become_updater` saga action, which attempts to inherit the lock from the parent `start_instance_update` saga, must never fail if a previous execution of that node has already marked the instance record as locked. Like in PR #10177, which fixed #10166, we can resolve this by adding a retry loop that should retry any transient database errors indefinitely. And, as we discovered in #10785, the exception to this rule is cases where the instance record has been deleted, so we handle that gracefully here as well (or at least we will, once #10787 actually fixes the lock inheriting query's handling of deleted instance records...).
nickelization
left a comment
There was a problem hiding this comment.
This is only my third day at Oxide, so there's a chance you might want to wait for someone more experienced to review this, but for what it's worth I looked it over pretty carefully and everything here makes sense to me and looks good!
Really appreciated the detailed comments by the way - as a newbie they were definitely helpful to me in understanding what's going on here, and also helped solidify my thinking around this stuff in general a bit too.
Thanks, I'm glad you found them helpful! I try my best. :) |
Similarly to #10166, a duplicate execution of the
siu_become_updatersaga action, which attempts to inherit the lock from the parent
start_instance_updatesaga, must never fail if a previous execution ofthat node has already marked the instance record as locked. Like in PR
#10177, which fixed #10166, we can resolve this by adding a retry loop
that should retry any transient database errors indefinitely. And, as we
discovered in #10785, the exception to this rule is cases where the
instance record has been deleted, so we handle that gracefully here as
well (or at least we will, once #10787 actually fixes the lock
inheriting query's handling of deleted instance records...).