Skip to content

odb: add write_packfile, for_each_unique_abbrev, convert_object_id#2074

Open
MayCXC wants to merge 1 commit intogitgitgadget:masterfrom
MayCXC:ps/series-1-vtable-v3
Open

odb: add write_packfile, for_each_unique_abbrev, convert_object_id#2074
MayCXC wants to merge 1 commit intogitgitgadget:masterfrom
MayCXC:ps/series-1-vtable-v3

Conversation

@MayCXC
Copy link

@MayCXC MayCXC commented Mar 26, 2026

This adds three ODB source vtable methods that were not part of the
recent ps/odb-sources and ps/object-counting series, plus caller
routing for object-name.c and fast-import.c.

New vtable methods:

  • write_packfile: Ingest a pack from a file descriptor. The files
    backend chooses between index-pack (large packs) and
    unpack-objects (small packs below fetch.unpackLimit). Options
    cover thin-pack fixing, promisor marking, fsck, lockfile capture,
    and shallow file passing. Non-files backends can handle pack
    ingestion through their own mechanism.

  • for_each_unique_abbrev: Iterate objects matching a hex prefix for
    disambiguation. The files backend searches loose objects via
    oidtree, multi-pack indices, then non-MIDX packs.

  • convert_object_id: Translate between hash algorithms using the
    loose object map. Used during SHA-1 to SHA-256 migration.

Caller routing:

  • object-name.c: The abbreviation and disambiguation paths
    (find_short_object_filename, find_abbrev_len_packed, and
    find_short_packed_object) directly access files-backend internals
    (loose cache, pack store, MIDX). These are converted to dispatch
    through the for_each_unique_abbrev vtable method, so that
    non-files backends participate through proper abstraction rather
    than being skipped.

  • fast-import.c: end_packfile() replaced direct pack indexing,
    registration, and odb_source_files_downcast() with a call to
    odb_write_packfile(). gfi_unpack_entry() falls back to
    odb_read_object() when the pack slot is NULL (non-files backends
    ingest packs without registering them on disk).

This addresses Patrick's feedback on the previous submission [1]:
the correct fix for downcast sites is proper vtable abstraction, not
skipping non-files backends.

Additional:

  • ODB_SOURCE_HELPER added to the source type enum
  • odb/source-type.h extracted to avoid circular includes with
    repository.h
  • OBJECT_INFO_KEPT_ONLY flag for backends that track kept status
  • self_contained_out output field on odb_write_packfile_options

Motivation: These methods are needed by the local helper backend
series [2], which delegates object and reference storage
to external git-local- helper processes. sqlite-git [3] is a
working proof of concept that stores objects, refs, and reflogs
in a single SQLite database with full worktree support.

CC: Junio C Hamano gitster@pobox.com, Patrick Steinhardt ps@pks.im

[1] https://github.com/gitgitgadget/git/pull/2068.patch
[2] https://github.com/gitgitgadget/git/compare/master...MayCXC:git:ps/series-2-helpers-v3.patch
[3] https://github.com/MayCXC/sqlite-git

@gitgitgadget
Copy link

gitgitgadget bot commented Mar 26, 2026

There is an issue in commit 3d511c2:
odb: add write_packfile, for_each_unique_abbrev, convert_object_id

  • Commit not signed off

@MayCXC MayCXC force-pushed the ps/series-1-vtable-v3 branch 2 times, most recently from 785108f to 146c7ed Compare March 26, 2026 13:00
@MayCXC
Copy link
Author

MayCXC commented Mar 26, 2026

/submit

@gitgitgadget
Copy link

gitgitgadget bot commented Mar 26, 2026

Submitted as pull.2074.git.1774530437562.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-2074/MayCXC/ps/series-1-vtable-v3-v1

To fetch this version to local tag pr-2074/MayCXC/ps/series-1-vtable-v3-v1:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-2074/MayCXC/ps/series-1-vtable-v3-v1

@MayCXC MayCXC force-pushed the ps/series-1-vtable-v3 branch 2 times, most recently from 6e569fc to 5b3e9a8 Compare March 26, 2026 13:32
@MayCXC
Copy link
Author

MayCXC commented Mar 26, 2026

/submit

@gitgitgadget
Copy link

gitgitgadget bot commented Mar 26, 2026

Submitted as pull.2074.v2.git.1774532383055.gitgitgadget@gmail.com

To fetch this version into FETCH_HEAD:

git fetch https://github.com/gitgitgadget/git/ pr-2074/MayCXC/ps/series-1-vtable-v3-v2

To fetch this version to local tag pr-2074/MayCXC/ps/series-1-vtable-v3-v2:

git fetch --no-tags https://github.com/gitgitgadget/git/ tag pr-2074/MayCXC/ps/series-1-vtable-v3-v2

@MayCXC MayCXC force-pushed the ps/series-1-vtable-v3 branch 6 times, most recently from 47c9431 to 729b6f2 Compare March 26, 2026 18:44
Add three vtable methods to odb_source that were not part of the
recent ps/odb-sources and ps/object-counting series:

 - write_packfile: ingest a pack from a file descriptor. The files
   backend chooses between index-pack (large packs) and
   unpack-objects (small packs below fetch.unpackLimit). Options
   cover thin-pack fixing, promisor marking, fsck, lockfile
   capture, and shallow file passing.

 - for_each_unique_abbrev: iterate objects matching a hex prefix
   for disambiguation. Searches loose objects via oidtree, then
   multi-pack indices, then non-MIDX packs.

 - convert_object_id: translate between hash algorithms using the
   loose object map. Used during SHA-1 to SHA-256 migration.

Also add ODB_SOURCE_HELPER to the source type enum, preparing for
the helper backend in the next commit.

The write_packfile vtable method replaces the pattern where callers
spawn index-pack/unpack-objects directly. fast-import already uses
odb_write_packfile() and this allows non-files backends to handle
pack ingestion through their own mechanism.

Signed-off-by: Aaron Paterson <apaterson@pm.me>
@MayCXC MayCXC force-pushed the ps/series-1-vtable-v3 branch from 729b6f2 to a69b8d3 Compare March 26, 2026 19:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant