Fix stale mapping and respect delete in rsync_from_disk#5357
Open
PauloVLB wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
b/530149063
Problem
ProtoFuzzTargetCorpus.rsync_from_diskkeeps a stale internal mapping of files to delete. Since thecorpus_pruningtask calls this method twice (once after initial pruning and again after cross-pollination), the second pass attempts to delete the exact same files again, causing a flood of 404 warnings. It was also completely ignoring thedeleteargument.Link for logs
Proposed Solution
Update the internal mapping after deletion to remove the files that were just deleted. Also, wrap the deletion logic to respect the
deleteflag.Validation
Added a new unit test
test_rsync_multiple_callsincorpus_manager_test.pyto confirm that multiple calls don't try to delete the same files.Also validated in dev environment: those are logs from a
corpus_pruningtask successfully executing without the mentioned warnings.Link
Why weren't these warnings visible before?
The bug was always there but silent. Before switching to threads, we used processes. On Linux, the background thread responsible for flushing logs to Cloud Logging dies in forked child processes. Thus, warnings from child processes (like these 404s) were swallowed. Threads keep execution in the same process, making the warnings visible.