Skip to content

Fix stale mapping and respect delete in rsync_from_disk#5357

Open
PauloVLB wants to merge 1 commit into
masterfrom
fix-corpus-pruning-404
Open

Fix stale mapping and respect delete in rsync_from_disk#5357
PauloVLB wants to merge 1 commit into
masterfrom
fix-corpus-pruning-404

Conversation

@PauloVLB

@PauloVLB PauloVLB commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

b/530149063

Problem

ProtoFuzzTargetCorpus.rsync_from_disk keeps a stale internal mapping of files to delete. Since the corpus_pruning task calls this method twice (once after initial pruning and again after cross-pollination), the second pass attempts to delete the exact same files again, causing a flood of 404 warnings. It was also completely ignoring the delete argument.

Link for logs

image

Proposed Solution

Update the internal mapping after deletion to remove the files that were just deleted. Also, wrap the deletion logic to respect the delete flag.

Validation

Added a new unit test test_rsync_multiple_calls in corpus_manager_test.py to confirm that multiple calls don't try to delete the same files.

Also validated in dev environment: those are logs from a corpus_pruning task successfully executing without the mentioned warnings.

Link

image

Why weren't these warnings visible before?

The bug was always there but silent. Before switching to threads, we used processes. On Linux, the background thread responsible for flushing logs to Cloud Logging dies in forked child processes. Thus, warnings from child processes (like these 404s) were swallowed. Threads keep execution in the same process, making the warnings visible.

@PauloVLB PauloVLB marked this pull request as ready for review July 9, 2026 03:14
@PauloVLB PauloVLB requested a review from a team as a code owner July 9, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant