Skip to content

feat: Incremental Build#1267

Draft
RandomByte wants to merge 210 commits intomainfrom
feat/incremental-build-4
Draft

feat: Incremental Build#1267
RandomByte wants to merge 210 commits intomainfrom
feat/incremental-build-4

Conversation

@RandomByte
Copy link
Copy Markdown
Member

Implementation of RFC 0017 Incremental Build

This PR supersedes previous PoC: #1238

JIRA: CPOUI5FOUNDATION-1174

@RandomByte RandomByte marked this pull request as draft January 7, 2026 12:28
@coveralls
Copy link
Copy Markdown

coveralls commented Jan 7, 2026

Coverage Status

coverage: 99.566% (+4.9%) from 94.658%
when pulling b727ebc on feat/incremental-build-4
into cb29ec1 on main.

@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch 3 times, most recently from 5224cd2 to 4904c84 Compare January 9, 2026 09:22
@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch from 950fc6d to 41eed91 Compare January 14, 2026 15:28
@maxreichmann maxreichmann force-pushed the feat/incremental-build-4 branch from bb39565 to 2a21507 Compare January 20, 2026 10:01
@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch 3 times, most recently from 6233816 to f858659 Compare January 20, 2026 16:58
@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch from 71db1d0 to a2c371f Compare January 26, 2026 09:54
@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch from 7364b4b to cf43f0c Compare February 9, 2026 13:32
@maxreichmann maxreichmann force-pushed the feat/incremental-build-4 branch 2 times, most recently from 252b966 to 874a943 Compare February 16, 2026 17:17
Comment thread packages/project/test/fixtures/application.a/custom-tasks/custom-task-2.js Outdated
Comment thread packages/project/test/fixtures/application.a/custom-tasks/custom-task-0.js Outdated
@maxreichmann maxreichmann force-pushed the feat/incremental-build-4 branch 5 times, most recently from df275e5 to 0345502 Compare February 27, 2026 10:34
Comment thread packages/project/test/fixtures/application.a/task.dependency-change.js Outdated
@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch from 9fc2509 to 20ba653 Compare March 5, 2026 16:23
@maxreichmann maxreichmann force-pushed the feat/incremental-build-4 branch from 20ba653 to 9fc2509 Compare March 5, 2026 16:34
@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch from 9fc2509 to 940376d Compare March 5, 2026 16:40
@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch from 9197670 to 44d1107 Compare March 20, 2026 15:05
@maxreichmann maxreichmann force-pushed the feat/incremental-build-4 branch from 44d1107 to d7c402c Compare March 26, 2026 14:05
@maxreichmann
Copy link
Copy Markdown
Member

maxreichmann commented Mar 26, 2026

Rebased onto origin/main

Comment thread internal/e2e-tests/test/build.js Fixed
Comment thread internal/e2e-tests/test/version.js Fixed
maxreichmann and others added 14 commits April 14, 2026 16:21
…ly some dependencies, not all)

`+` Fix some typos in comments
`+` Extend FixtureTester to assert seen projects (not only built projects)
`+` Fix some copy/paste leftovers ("cleanDest: false")
`+` Replace console.log with @ui5/logger Log
* Add test for "ui5 --version"
* Add test for "ui5 build" (currently testing a TS application)

Currently, those tests are not included in any CI pipeline and also need a manual "npm install" (no node_modules included yet).
+ Include "npm install" in test runtime
+ Refactor test environment for better reuse
+ Extend typescript test to cover Incremental Build
+ Add "application.a" (Javascript project fixture)
…ently FAILING)

+ Refactor "ui5-tooling-modules" test
@RandomByte RandomByte force-pushed the feat/incremental-build-4 branch from aa3a2c1 to 17bfe1c Compare April 15, 2026 07:58
…gistry.flush

Split upsertResources into two phases: concurrent I/O resolution
via Promise.all (getIntegrity, getSize, matchResourceMetadataStrict)
followed by serial tree mutation. This avoids sequential file reads
when processing many resources.

Pre-resolve getIntegrity and getSize for all pending resources
concurrently via Promise.all before the serial tree-mutation loop.
This avoids redundant sequential I/O across the triple-nested
directory x tree x resource loop.
Restructure updateIndices() into 3 phases: collect all unique paths
across request sets, batch-fetch in parallel via Promise.all, then
process synchronously from cache. Also parallelize refreshIndices()
node processing since SharedHashTree operations schedule atomically
into the TreeRegistry.
Add unchangedNodes Set in TreeRegistry.flush() to track resource nodes
already confirmed unchanged via matchResourceMetadataStrict. Subsequent
trees sharing the same node skip the full comparison and only check tags,
eliminating N-1 async calls per shared resource across N trees.
Overlap stage cache I/O by prefetching metadata for the next task
while the current task is executing. This reduces idle time between
sequential task executions in warm-cache builds.
Add structured timing and counters to identify I/O hotspots in the
build cache validation flow. Instrumentation covers refreshIndices,
updateIndices, flushTreeChanges, TreeRegistry.flush phases, and
matchResourceMetadataStrict. All gated behind @ui5/logger perf level
or UI5_CACHE_PERF env var.
Add perf-level timing to the build orchestration layer to identify
the ~6.5s gap observed in warm-cache builds between source index
initialization and per-project cache validation. Instruments:
- getRequiredProjectContexts (total + per-project context creation)
- prepareProjectBuildAndValidateCache (getDependenciesReader vs cache)
- #flushPendingChanges (source index vs dependency index updates)
- ProjectBuilder.#build per-project timing
Defer source index initialization from ProjectBuildCache.create() to a
separate initSourceIndex() method, allowing BuildContext to initialize
all project source indices concurrently via Promise.all instead of
sequentially in the dependency discovery loop.
During dependency index updates, the cache proxy reader eagerly called
cacache.get.info() for every byPath() request, even though the callers
(hash tree upserts) only need resource metadata. With 2550+ parallel
calls, this created severe filesystem I/O contention (~2.2s).

Defer the cache path resolution to first content access using a lazy
singleton promise pattern. Also replace O(n) Array.includes() with O(1)
object property lookup and fix the size/byteSize parameter mismatch.

sap.ui.layout updateDependencyIndices: 2466ms -> 59ms
Total warm-cache build: 2.85s -> 315ms
…ing cache writes

Track integrity hashes from restored stage metadata in a Set and skip
cacache.get.info() calls for resources already known to be in CAS.
Reduces cache write time from ~1,400ms to ~100ms for stale-cache builds
by eliminating ~15,000 redundant CAS existence checks.
… import

On first CLI invocation, #importStages treated all restored stage
resources as "changed" because #currentStageSignatures was empty.
This caused ~3000+ resource paths to be propagated to dependents,
triggering expensive updateDependencyIndices calls (~108ms total).

The imported stages represent the already-cached state, not actual
changes. Skip writtenResourcePaths accumulation when this is the
initial import (empty #currentStageSignatures), since dependents'
dependency indices were restored from the same persistent cache.
…pagated

When restoring from cache, dependency indices are already populated
via BuildTaskCache.fromCache. If no dependency changes were propagated
from upstream projects, the cached indices are already correct and
_refreshDependencyIndices can be skipped entirely. This avoids
fetching ~2738 resources per dependent just to confirm nothing changed,
saving ~130ms on warm-cache builds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants