[Bug] Concurrent tag creation and snapshot expiration may delete files referenced by the new tag

[LiangRuoShui]

Search before asking

• I searched in the issues and found no issue covering this exact create-tag versus snapshot-expiration race.

Paimon version

• 1.3.1
• The same race is still present on master commit 54e64e3.

Compute Engine

Java API. The problem is in paimon-core and is compute-engine independent.

Minimal reproduce step

Preconditions:

1. Create several snapshots.
2. Choose an old snapshot S inside the range that will be expired.
3. Ensure a later snapshot has deleted or replaced files that are still visible in S.
4. Run tag creation and snapshot expiration concurrently.

A deterministic test can inject a latch immediately after ExpireSnapshotsImpl collects tagged snapshots:

1. Thread A enters ExpireSnapshotsImpl.expireUntil.
2. Thread A executes tagManager.taggedSnapshots in 1.3.1, or collectTaggedSnapshots on master, and is paused. Snapshot S has no tag at this point.
3. Thread B calls table.createTag with tag name tag-s and snapshot id S. The tag file is written successfully.
4. Thread B returns success.
5. Resume Thread A.
6. Thread A continues data-file and manifest cleanup using the old tagged-snapshot list, which does not contain tag-s.
7. Thread A finally deletes the snapshot metadata for S.
8. Read tag-s using tag time travel.

The tag file remains present, but its manifest list, manifest, or data files may have been deleted, causing FileNotFoundException or an unreadable tag.

The relevant ordering in release 1.3.1 is:

• ExpireSnapshotsImpl reads tagManager.taggedSnapshots once.
• The same fixed list is used by createDataFileSkipperForTags and findSkippingTags.
• cleanUnusedDataFiles and cleanUnusedManifests run.
• Snapshot metadata is deleted last.

Tag creation independently does:

• AbstractFileStoreTable.findSnapshot loads S.
• TagManager.createTag writes the tag metadata through FileIO.overwriteFileUtf8.

There is no shared lock, GC epoch, revalidation, or second tag collection between these paths.

Current master still collects tagged snapshots once before planning and deleting data and manifest files, and deletes snapshot metadata last.

What does not meet your expectations?

A successfully created tag should remain readable and must protect every file referenced by its tagged snapshot.

Snapshot expiration currently protects only tags visible when expiration initially collects tag metadata. A tag created after that point can return success while the same expiration operation deletes files referenced by it. This is a TOCTOU race that can cause data loss from the perspective of the new tag.

Checking snapshot existence before and after creating the tag is not sufficient because snapshot metadata is deliberately deleted after data and manifest cleanup. Both checks can succeed before the expiration operation removes the snapshot metadata and referenced files.

Are you willing to submit a PR?

Not committed at this time.