Cleanup, performance, and fixes for #60 + #53#73
Merged
Conversation
Rationale: we need source-level access to CUE4Parse to debug API quirks, apply local patches, and pick up upstream fixes without waiting for a package release. Changes: - UnrealReZen.csproj: drop PackageReference "CUE4Parse" 4.28.0; add ProjectReference to ..\external\CUE4Parse\CUE4Parse\CUE4Parse.csproj - .gitmodules: pin external/CUE4Parse to FabianFG/CUE4Parse - Program.cs / Compressions.cs: adapt to the current CUE4Parse API (OodleHelper.OodleFileName constant, ref-path DownloadOodleDll signature)
…y paths
WriteProgressBar no longer divides by zero on single-file packs
- Clamp denominator with Math.Max(total, 1) so passing one file to pack
doesn't throw DivideByZeroException.
Replace the broken MemoryMappedHelpers pattern with ChunkSource
- CreateMemoryMappedFileFromByteArray used CreateNew inside a using block
then OpenExisting after dispose, so the mapping was already released
when callers tried to read it. Deleted MemoryMappedHelpers.cs.
- New ChunkSource abstraction (FromFile / FromBytes, ReadInto, Length)
gives the packer a single type that works for both on-disk inputs
(memory-mapped) and the in-memory dependencies blob.
Stream AES encryption instead of loading the entire .ucas into memory
- Old path: File.ReadAllBytes -> EncryptAES(byte[]) -> File.WriteAllBytes,
which OOM'd on large packs (>2 GB).
- New path: CryptoStream from src .ucas to a temp file, then File.Move
replaces the original.
Cross-platform path handling
- Path.GetRelativePath + Replace('\','/') instead of
Replace("\\","") string math that only happened to work on Windows.
Type safety
- Replace `dynamic b = dep` with provider.Files.OfType<FIoStoreEntry>();
avoids runtime DLR dispatch and the NRE it throws on unexpected types.
- Tighten nullable annotations on DirIndexWrapper helpers
(dentry / lastDentry now correctly typed FIoDirectoryIndexEntry?).
CUE4Parse API migration
- Oodle: OodleHelper.OODLE_DLL_NAME -> OodleHelper.OodleFileName,
DownloadOodleDll(string) -> DownloadOodleDll(ref string? path).
Cache Zlib / Oodle native instances in static Lazy<T> - Compressions.cs previously constructed a fresh Zlibng or Oodle wrapper (which reloads the native DLL) for every 64 KB compression block. On a 1 GB pack that was ~16 000 DLL init cycles. - Now held in Lazy<Zlibng> / Lazy<Oodle> with isThreadSafe=true so the DLL is loaded exactly once per process. Fold SHA1 into the compression loop (single read pass) - Old: mmf.SHA1Hash() read the whole file, then the compression loop read it a second time block-by-block. - New: one loop reads each block once, feeds it into sha1.TransformBlock(..), compresses, writes. TransformFinalBlock at the end produces the chunk hash. Halves the bytes read from MMF. Build a utoc entry lookup once instead of scanning per-input - Old BuildManifest walked provider.Files (~all entries across all containers) for every file in the repack list: O(n*m). - New BuildUtocEntryLookup produces a Dictionary<string, List<FIoStoreEntry>> keyed by entry.Path, case-insensitive, then each repack input is an O(1) probe: O(n+m) overall. Minor - Hoist the 16-byte stackalloc pad buffer out of the inner file loop (silences CA2014, allocates once per pack instead of once per file). - Drop ChunkSource.ComputeSha1; hashing moved into the one-pass loop.
Renames (content also cleaned up) - UnrealReZen/Core/FIoConstants.cs -> Constants.cs (class was already named Constants; file name now matches) - UnrealReZen/Core/FIoUToc.cs -> UToc.cs (kept only UTocHeader, AssetMetadata, FIoContainerID; see below) Dead code removed - UToc wrapper class: never read back by anything; the packer only writes the UTOC bytes, so the reader-style wrapper was unused. - FIoPerfectHashSeeds: unreferenced. - ManifestData (Core/FIoDependency.cs): unreferenced. - MemoryStreamHelpers: removed Read/Seek/Skip/Tell/Pos extensions that were never called. Only Write extensions remain. Program.cs split from one long Main into focused helpers - EnsureNativeDlls, TryParseEngineVersion, ValidateCliOptions - LogOptionsSummary, TryLoadProvider - BuildManifest, BuildUtocEntryLookup Main/Run now read top-to-bottom as a pipeline. API hygiene - DefaultFileProvider: switch from the (string, SearchOption, bool, VersionContainer?) obsolete ctor to (string, SearchOption, VersionContainer, StringComparer). - FIoStoreTocEntryMeta.ChunkHash marked required so the implicit non-null contract is enforced by the compiler.
Local Claude Code settings (.claude/settings.local.json) were tracked by accident in the initial snapshot. They are per-user, per-machine data and shouldn't live in the repo. Added .claude/ to .gitignore and untracked the existing file.
Symptom - "CUE4Parse.Compression.OodleException: Oodle decompression failed: not initialized" at provider load time, even though oo2core / oodle dll was present next to the exe. Users had to manually register the instance from their own code to work around it. Root cause - EnsureNativeDlls was calling OodleHelper.DownloadOodleDll and ZlibHelper.DownloadDll, which only *fetch* the DLL. Neither sets OodleHelper.Instance / ZlibHelper.Instance, so CUE4Parse's decoder threw the moment it tried to read an Oodle/Zlib-compressed source container. Fix - Use OodleHelper.Initialize / ZlibHelper.Initialize, which download (if missing) AND register the Instance. Verify Instance is non-null afterwards and fail fast with a clear message if not.
#53) Symptom - Passing --aes-key (required to read encrypted source archives) also silently encrypted the output .ucas and set the EncryptedContainerFlag in the .utoc. The tool only encrypted the .ucas body though -- it never encrypted the .utoc directory index -- so the resulting container was structurally invalid for games that honor the flag (FModel flagged "IsEncrypted for no reason", games crashed on load). Design change - --aes-key is now strictly "key for decrypting the source game". - New --encrypt-output flag (default false) opts in to encrypting the generated .ucas and setting the EncryptedContainerFlag. Most UE4/5 games accept plain archives for mods, so this matches the common case and the behavior of community forks (GhostyPool, matyamod). Code - Packer.PackToCasToc now takes a nullable FAesKey? outputAes. Null means "don't encrypt, don't set the flag"; non-null means "encrypt UCAS with this key and set the flag". - ConstructUtocFile takes an explicit bool isEncrypted instead of inferring it from the AES key string. - Program.Run only forwards the key when opts.EncryptOutput is true. Known limitation - When --encrypt-output is passed, the .utoc directory index itself is still not encrypted. Games that strictly require an encrypted directory index will reject the container. Proper directory-index encryption is tracked separately; this commit fixes the common "don't encrypt at all" path.
What changed and why - .NET 7 -> 8. The project now targets net8.0 (csproj), so the old setup-dotnet@v3 with 7.0.x would fail to restore. - Checkout with submodules: false, then an explicit "git submodule update --init external/CUE4Parse". A plain submodules: recursive dies on a broken catch2 pin buried under ACL/rtm's transitive submodules; we don't need those anyway. CUE4Parse's native CMake step is non-fatal (it logs a warning and continues if the natives don't build), so skipping ACL is safe - our tool downloads its own Oodle/Zlib DLLs at runtime. - Dropped the "dotnet test" step. There is no test project in the repo, so the step was always a no-op-or-fail depending on SDK version. - Bumped actions/checkout v3 -> v4, actions/setup-dotnet v3 -> v4. - Removed the **.cs / **.csproj path filter. A change to the workflow file, .gitmodules, or the submodule pin should also retrigger CI. - Matrix left with a single entry (windows-latest) but kept as a matrix so future platforms can be added without restructuring.
This was referenced Apr 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multi-commit pass over the codebase covering bug fixes, performance wins, a structural cleanup, and two reported issues.
Fixes
EnsureNativeDllswas callingDownloadOodleDll/DownloadDll, which only fetch the native DLL; neither setsOodleHelper.Instance/ZlibHelper.Instance, so the CUE4Parse decoder threw the moment it hit an Oodle/Zlib-compressed source container. Switched toInitialize(...), which downloads and registers the instance.EncryptedContainerFlagset on the resulting UTOC even when the archive wasn't properly encrypted.--aes-keynow only unlocks the source archives; a new opt-in--encrypt-outputis required to encrypt the generated.ucasand set the flag. Default behavior is plain (matches community forks). A known limitation — the UTOC directory index itself still isn't encrypted when--encrypt-outputis used — is called out in that commit's body.Other commits in this PR
WriteProgressBaron single-file packs, MMF-from-bytes pattern that disposed the mapping before callers read it, OOM when AES-encrypting a large.ucasviaFile.ReadAllBytes,Replace(\"\\\\\",\"\")path math that only worked on Windows,dynamicdispatch replaced withOfType<FIoStoreEntry>(), and the Oodle API migration for the current CUE4Parse (OodleFileName,ref-pathDownloadOodleDll).Lazy<T>(was constructed per 64 KB block), SHA1 folded into the compression loop so source is read once instead of twice, per-input provider scan replaced with a dictionary lookup (O(n·m) → O(n+m)), stackalloc pad buffer hoisted out of the inner loop.FIoConstants.cs→Constants.cs,FIoUToc.cs→UToc.cs; removed the never-read-backUTocwrapper,FIoPerfectHashSeeds,ManifestData, and unusedRead/Seek/Skip/Tell/Posextensions onMemoryStream; splitProgram.csinto focused helpers (EnsureNativeDlls,TryParseEngineVersion,ValidateCliOptions,LogOptionsSummary,TryLoadProvider,BuildManifest,BuildUtocEntryLookup); switchedDefaultFileProviderto its non-obsolete constructor..claudedirectory — local editor settings shouldn't live in the repo.Review notes
Each commit stands on its own and has a detailed body. Easier to review per-commit than as a squash.
Build: clean (
dotnet build UnrealReZen/UnrealReZen.csproj). No runtime regression tests exist in the repo; the two GitHub issue fixes were verified by reading the CUE4Parse API paths they touch.