borg 2.0.0b20 corruption

[tve]

I'm having difficulties with a repo that I created on a ZeroFS filesystem backed by Hetzner object storage. When I run borg check I get a couple of consistency errors like:

Repo object ee194bd9f35d06e777aae2c261c9139d1eea6ff75ab0c337b61e689707ad64d1 is corrupted: metadata does not match checksum.
Repo object ee194bd9f35d06e777aae2c261c9139d1eea6ff75ab0c337b61e689707ad64d1 is corrupted: data does not match checksum.

What type of corruption do these errors indicate? Are they bit flips or could they mean that the file got truncated or zeroed?

The file in question is 7.8MB in size, is there a way I can verify the checksum manually or run the check just on this file as opposed to having to rerun on the whole repository (which takes >12h).

[ThomasWaldmann]

        def check_object(obj):
            """Check if obj looks valid."""
            hdr_size = RepoObj.obj_header.size
            obj_size = len(obj)
            if obj_size >= hdr_size:
                hdr = RepoObj.ObjHeader(*RepoObj.obj_header.unpack(obj[:hdr_size]))
                meta = obj[hdr_size : hdr_size + hdr.meta_size]
                if hdr.meta_size != len(meta):
                    log_error("metadata size incorrect.")
                elif hdr.meta_hash != xxh64(meta):
                    log_error("metadata does not match checksum.")
                data = obj[hdr_size + hdr.meta_size : hdr_size + hdr.meta_size + hdr.data_size]
                if hdr.data_size != len(data):
                    log_error("data size incorrect.")
                elif hdr.data_hash != xxh64(data):
                    log_error("data does not match checksum.")
            else:
                log_error("too small.")

[ThomasWaldmann]

Layout: header (with xxh64 "checksums") + metadata + data

So, that means:

• the header is not truncated (otherwise it would say "too small")
• the metadata is also not truncated (otherwise it would say "metadata size incorrect")
• the data is also not truncated (otherwise it would say "data size incorrect")

Could be bit flips or more severe corruption (like due to bad RAM or ranges of zeros/crap read/written due to some malfunction).

[tve]

Am I correct that borg writes files only once? I.e. a file is never updated/appended to?

[ThomasWaldmann]

Yes. If you use the filesystem storage, it will use that:

https://github.com/borgbackup/borgstore/blob/master/src/borgstore/backends/posixfs.py#L184

If you just want S3 storage, borgstore also supports that directly.

[ThomasWaldmann]

class RepoObj:
    # Object header format includes size information for parsing the object into meta and data,
    # as well as hashes to enable checking consistency without having the borg key.
    obj_header = Struct("<II8s8s")  # meta size (32b), data size (32b), meta hash (64b), data hash (64b)
    ObjHeader = namedtuple("ObjHeader", "meta_size data_size meta_hash data_hash")

[tve]

The corruption may be due to a bug in the 9p remote file protocol: Barre/ZeroFS#346

[ThomasWaldmann]

Guess you are double-beta-testing. :-)