Download base package sources from posit-dev/oak-r-sources

[DavisVaughan]

This is probably one of the more technically impressive PRs I've ever done 😛

Previously in oak_source::get_r("4.5.0") we would download the 100mb R 4.5.0 tarball from CRAN and store the whole thing on disk in the cache so we could have access to base package source files. We would repeat this for any other R version the user uses.

We don't really use much of that downloaded tarball right now, just src/library/{package}/R/{files}.R to have the original source files for the base R package of interest.

@lionel- suggested that maybe we could vendor the base packages' R files, starting with the full files for R 4.2.0 (our last supported version of R) and shipping git diffs for all subsequent patch versions that could be applied on the fly. Claude and I have come up with something even better 😄.

Zstd magic

Instead we inline a single 1.5 MB base-sources.tar.zst with include_bytes!() into the ark binary that we decompress and untar on the fly. This contains the complete R/ sources for every base package for every patch release since R 4.2.0, all ~20 of them. This is an incredible technical achievement, as you are about to see.

• 1.5 MB base-sources.tar.zst
• 127.3 MB base-sources.tar decompressed (by ✨ magic ✨ )
• 142.7 MB base-sources/ folder fully untarred

So how the hell did we pack 142.7 MB into 1.5 MB? Good question!

You see, a single R version's worth of src/library/{package}/R/{files}.R files weigh in at about 8 MB. Tarred and compressed, this comes in at about 1.3 MB for just that one R version (very close to our 1.5 MB!).

Between R versions, the {files}.R don't actually change all that much! We take advantage of this by leveraging a zstd feature called the compression window. Zstd maintains an adjustable "lookback window" of bytes that it uses to look for repeating patterns in the blob you are compressing. If it finds patterns, it can highly compress them.

We leverage this by placing the same file across R versions adjacent to each other in the .tar. For example, rather than tarring in this naive order:

4.5.0/base/R/a.R
4.5.0/base/R/b.R
4.5.0/methods/R/a.R

4.6.0/base/R/a.R
4.6.0/base/R/b.R
4.6.0/methods/R/a.R

we instead sort the files like this before tarring:

4.5.0/base/R/a.R
4.6.0/base/R/a.R

4.5.0/base/R/b.R
4.6.0/base/R/b.R

4.5.0/methods/R/a.R
4.6.0/methods/R/a.R

This places 4.5.0/base/R/a.R right next to 4.6.0/base/R/a.R in the tar, and they are basically the same file between R versions, so when zstd sees this it basically removes all of the duplication in the compressed result.

This means that we can store a whopping 20 versions of R sources for nearly the same price as 1, and the size is small enough that we can just vendor it in to ark itself.

Binary size

What I don't understand is what cargo build is doing. With a debug build, you can "see" the difference in the binary size:

cargo build

95,214,024 (95.2 MB) # before
97,698,232 (97.6 MB) # after

cargo build --release

24,543,568 (24.5 MB) # before
24,526,064 (24.5 MB) # after

It got...smaller?? I have no idea how that could happen, but I tried this multiple times and got the same result? I even did a cargo clean between branch switches. I'm surely doing something wrong.

But it's definitely working!

Screen.Recording.2026-07-09.at.8.59.21.AM.mov

Updating

In terms of updating, we will need to update the bundled base-sources.tar.zst after every patch R release. All we have to do is:

• Update versions.txt with the new version
• Run just vendor-r-sources, which will re-download all the R sources from CRAN and recreate an updated base-sources.tar.zst that we'd commit

Some other things

• If a user has a newer R than we have sources for, like r-devel, then we fall back to our newest version of the R sources. This is probably pretty darn good, considering that they don't change much between versions. And I think it is better than doing nothing at all.
• This all still goes through our standard Cache, so once a particular R version's sources are unpacked from the compressed tarball and stored in the cache, they are never unpacked again. That said, those sources will be subject to the same "eviction" rules as any other Cache, so if you don't use an R version's sources for 4 months, it will be evicted (freeing up around 8 MB of space).

[lionel-]

Nice!

One minor comment: I know that ty has ty_vendored as well, but I'd have included these in oak_sources rather than in a separate oak_vendored crate. Fine either way.

Larger concern: Including the sources in a single binary blob of size ~N means that the repo will increase by N for every patch/minor/major release we add support for. By contrast, storing a root source of size N then separate diffs of sizes ~D means we're only increasing the repo size by D when a new release comes out.

That's not a deal breaker though. I do like when a git clone goes fast, but I leave it up to you to decide which approach to go for.

[DavisVaughan]

Due to to licensing concerns (which we aren't fully sure of the ramifications of), we didn't want to embed R's GPL-2 licensed files directly in the MIT licensed ark, even though we effectively are just an "installer" of those files and don't run them ourselves, which should be fine under GPL.

---

Regardless, we have now set up posit-dev/oak-r-sources to hold the infrastructure for making the compressed r-source.tar.zst. That repo is under GPL-2.

At each new R release, we will go over and do a new release of oak-r-sources, regenerating the r-source.tar.zst with the additional R release sources and attaching that as an Asset to the GitHub Release. Like this:
https://github.com/posit-dev/oak-r-sources/releases/tag/v1

I have purposefully chosen a versioning scheme of v1 rather than the R version of 4.6.1 here in case we have bug fixes between R releases, or if we want to change things to add additional base folders beyond R/ and need to do a feature release between R releases.

On the Ark side, after a new oak-r-sources release we update OAK_R_SOURCES_ASSET_VERSION and OAK_R_SOURCES_LATEST_R_VERSION to align with the oak-r-sources release, and that's it!

---

In Ark, the cache insert for base R packages works in 2 stages:

• We first need the r-source.tar.zst asset tied to our OAK_R_SOURCES_ASSET_VERSION. If it isn't in the cache, we go download it from the GitHub Release and cache it under a key of {OAK_R_SOURCES_ASSET_VERSION}_archive/.

• Once we have the r-source.tar.zst, we extract the R version of interest (with all of its base package sources) into {OAK_R_SOURCES_ASSET_VERSION}_{version}/.

When a user changes R versions, we already have r-source.tar.zst cached, so we can jump straight to unpacking.

There's no possible way to have an "outdated" version of r-source.tar.zst. If you have one sitting in cache/v1_archive/r-source.tar.zst and you update ark to a version where we've bumped oak-r-sources to v2, then ark will go out and download the latest r-source.tar.zst and place it in cache/v2_archive/r-source.tar.zst, and then then v1_archive/ will be a candidate for future cleanup. This structure of "only ever use the archive tied to OAK_R_SOURCES_ASSET_VERSION" simplified a lot of potential problems.

[lionel-]

Due to to licensing concerns (which we aren't fully sure of the ramifications of), we didn't want to embed R's GPL-2 licensed files directly in the MIT licensed ark, even though we effectively are just an "installer" of those files and don't run them ourselves, which should be fine under GPL.

Here are some pointers on the matter:

• https://www.gnu.org/licenses/gpl-faq.html#MereAggregation
• https://www.gnu.org/licenses/old-licenses/gpl-2.0-faq.en.html#GPLCompatInstaller