feat: interactive Iceberg Mode Explorer with real C++→WASM engine - #404
Open
JakeSCahill wants to merge 3 commits into
Open
feat: interactive Iceberg Mode Explorer with real C++→WASM engine#404JakeSCahill wants to merge 3 commits into
JakeSCahill wants to merge 3 commits into
Conversation
Turn the standalone v26.2 TOI explorer into a docs-integrated, versioned tool whose rule-bearing output is computed by Redpanda's real code, so it can't drift from the product. - src/js/27-iceberg-explorer.js: hydrates the `[iceberg-explorer]` block into the interactive tool (controls, config string, schema, sample record). Theme-aware CSS in src/css/iceberg-explorer.css. - iceberg-editor/: compiles Redpanda's real C++ Avro->Iceberg schema mapper (iceberg::type_to_iceberg from `dev`) to WebAssembly via Emscripten, with a thin Seastar shim. wasm-spike/ is the reproducible feasibility proof; wasm/ is the browser engine build (build.sh + Embind bindings). Fetched sources, deps, and build outputs are git-ignored (reproducible from the scripts). - Version-aware engine loader: loads iceberg-engine-<version> for the page's doc version, falling back to a bundle asset then to the interim JS engine. <body data-version> added in default.hbs to expose the version to JS. - Config DSL string stays JS but is pinned to Redpanda's own iceberg_mode_test.cc format vectors via tests/iceberg-dsl (fixed two drift bugs: missing value `layout`, wrong option order). - CI: build-iceberg-engine.yml builds a versioned engine per Redpanda release tag; iceberg-dsl-conformance.yml runs the DSL vectors and diffs upstream. Requires the `[iceberg-explorer]` block from redpanda-data/docs-extensions-and-macros. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
❌ Deploy Preview for docs-ui failed. Why did it fail? →
|
Contributor
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughAdds an interactive Iceberg Mode Explorer with deterministic DSL generation, schema and record rendering, and lazy loading of a versioned WebAssembly translation engine. The engine compiles production C++ Avro-to-Iceberg mapping code and exposes JSON through Embind. New scripts provide dependency fetching, source vendoring, staged WASM builds, and smoke tests. Documentation, preview wiring, static assets, styling, npm conformance tests, and GitHub Actions workflows are included. Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Browser
participant Explorer
participant WASMEngine
participant IcebergMapper
Browser->>Explorer: hydrate explorer
Explorer->>WASMEngine: load versioned engine
Explorer->>WASMEngine: avroToIcebergJson(schema)
WASMEngine->>IcebergMapper: map Avro schema to Iceberg type
IcebergMapper-->>WASMEngine: fields or error
WASMEngine-->>Explorer: JSON result
Explorer-->>Browser: render schema and translated record
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 9
🧹 Nitpick comments (1)
iceberg-editor/wasm-spike/shim/seastar/util/defer.hh (1)
32-35: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winStore a decayed callable in the scope guard.
deferpreserves lvalue references, so a guard that outlives or is moved beyond its callable can invoke a dangling reference. Returndeferred_action<std::decay_t<Func>>instead.Proposed fix
+#include <type_traits> + template<typename Func> -deferred_action<Func> defer(Func&& f) { - return deferred_action<Func>(std::forward<Func>(f)); +deferred_action<std::decay_t<Func>> defer(Func&& f) { + return deferred_action<std::decay_t<Func>>(std::forward<Func>(f)); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@iceberg-editor/wasm-spike/shim/seastar/util/defer.hh` around lines 32 - 35, Update the defer function to decay Func when constructing and returning the deferred_action, so the scope guard owns the callable value rather than preserving an lvalue reference. Use std::decay_t<Func> consistently for the return type and deferred_action construction while preserving perfect forwarding of f.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/build-iceberg-engine.yml:
- Line 35: Update the workflow step defining REF and the downstream version
handling to pass event-controlled values through step-level env variables rather
than interpolating them in run scripts; validate both ref and derived version
against the accepted safe format before writing GITHUB_OUTPUT. Also set checkout
persist-credentials to false and restrict workflow permissions to the minimum
required, preserving the existing build behavior for valid inputs.
In @.github/workflows/iceberg-dsl-conformance.yml:
- Line 31: Update the actions/checkout@v4 step in the workflow to disable
credential persistence by setting persist-credentials to false. Leave the
checkout behavior otherwise unchanged.
In `@iceberg-editor/wasm-spike/env.sh`:
- Around line 10-11: The Redpanda engine inputs are not pinned to one immutable,
reproducible revision. In iceberg-editor/wasm-spike/env.sh lines 10-11, require
an explicit immutable tag or commit instead of defaulting to mutable dev and
record the resolved commit; in
iceberg-editor/wasm-spike/third_party/fetch-deps.sh lines 20-22, resolve the
Avro SHA and patch blobs from that same revision or consume a per-release
lockfile with checksums.
In `@iceberg-editor/wasm-spike/README.md`:
- Around line 13-15: Declare the fenced code block language for the Avro
conversion output by adding the appropriate text fence marker around the
existing message, preserving the output unchanged.
In `@iceberg-editor/wasm/iceberg_wasm.cc`:
- Around line 35-50: Update escape_json to encode every JSON control character,
not only quotes, backslashes, and newlines; handle tabs, carriage returns, and
other bytes below 0x20 with valid JSON escape sequences or Unicode escapes while
preserving existing escaping behavior.
In `@src/css/iceberg-explorer.css`:
- Around line 9-21: Insert a blank line between the final custom property
declaration, --ice-cyan, and the font-size declaration in the affected CSS rule
to satisfy stylelint’s declaration-empty-line-before requirement.
In `@src/js/27-iceberg-explorer.js`:
- Around line 411-421: Update applyInitialConfig to detect legacy strings using
the section-format delimiter rather than rejecting any string containing a
colon: treat strings without a semicolon as legacy, so value_schema_latest
options still select schema_latest while section-based configs continue through
the section parser.
- Around line 22-67: Reformat the multi-line object literals in VAL_SCHEMA,
RECORD, and AVRO_VALUE_SCHEMA to satisfy object-curly-newline: place each nested
object’s multiple properties on separate lines with matching braces, including
VAL_SCHEMA.shipping_address, RECORD.value and headers entries, and the nested
AVRO_VALUE_SCHEMA shipping_address record. Preserve all values and schema
structure.
In `@src/ui.yml`:
- Around line 4-5: Update the src/ui.yml static_files entries so
iceberg-engine.wasm and iceberg-engine.js are only referenced after being
generated by the web WASM build and available in src/static; alternatively
remove the entries until those artifacts are produced. Ensure the selected
approach occurs before the gulp bundle/package bundle:pack tasks copy static
assets.
---
Nitpick comments:
In `@iceberg-editor/wasm-spike/shim/seastar/util/defer.hh`:
- Around line 32-35: Update the defer function to decay Func when constructing
and returning the deferred_action, so the scope guard owns the callable value
rather than preserving an lvalue reference. Use std::decay_t<Func> consistently
for the return type and deferred_action construction while preserving perfect
forwarding of f.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1a0a94e1-703d-4d11-b26f-dab2f81711c2
📒 Files selected for processing (34)
.github/workflows/build-iceberg-engine.yml.github/workflows/iceberg-dsl-conformance.ymliceberg-editor/README.mdiceberg-editor/wasm-spike/.gitignoreiceberg-editor/wasm-spike/README.mdiceberg-editor/wasm-spike/build-avro-cpp.shiceberg-editor/wasm-spike/env.shiceberg-editor/wasm-spike/fetch-sources.shiceberg-editor/wasm-spike/poc_main.cciceberg-editor/wasm-spike/shim/base/seastarx.hiceberg-editor/wasm-spike/shim/base/vassert.hiceberg-editor/wasm-spike/shim/iceberg/conversion/conversion_outcome.hiceberg-editor/wasm-spike/shim/seastar/core/chunked_vector.hhiceberg-editor/wasm-spike/shim/seastar/core/sstring.hhiceberg-editor/wasm-spike/shim/seastar/util/bool_class.hhiceberg-editor/wasm-spike/shim/seastar/util/defer.hhiceberg-editor/wasm-spike/shim/seastar/util/log.hhiceberg-editor/wasm-spike/shim/seastar/util/variant_utils.hhiceberg-editor/wasm-spike/shim/ssx/sformat.hiceberg-editor/wasm-spike/stage1-datatypes.shiceberg-editor/wasm-spike/stage2-schema-avro.shiceberg-editor/wasm-spike/third_party/fetch-deps.shiceberg-editor/wasm/.gitignoreiceberg-editor/wasm/build.shiceberg-editor/wasm/iceberg_wasm.ccpackage.jsonpreview-src/iceberg-explorer-test.adocpreview-src/ui-model.ymlsrc/css/iceberg-explorer.csssrc/css/site.csssrc/js/27-iceberg-explorer.jssrc/layouts/default.hbssrc/ui.ymltests/iceberg-dsl/conformance-test.js
Build was RED on ESLint. Fixes all nine unresolved review threads. src/js/27-iceberg-explorer.js - object-curly-newline (CRITICAL, broke `gulp lint:js`): expand the multi-line object literals in VAL_SCHEMA, RECORD and AVRO_VALUE_SCHEMA, getConfig() and the copy-button handler so each property gets its own line. `npx eslint` now reports 0 errors (24 pre-existing max-len warns). - no-undef on `createIcebergEngine`: qualify as `window.createIcebergEngine`. The bare identifier is an ESLint *error*, so this also broke the build. - Legacy DSL detection: use ';' rather than ':' as the discriminant. Legacy strings legitimately contain ':' (buildConfigString line 181 emits `value_schema_latest:subject=...`, matching the conformance vectors at tests/iceberg-dsl/conformance-test.js:42,45), so those strings fell through to the section parser where prefixFor['value_schema_latest'] is undefined and the val-mode radio was never set. The section format always joins three sections with ';' (line 202), so ';' is a sound discriminant. iceberg-editor/wasm/iceberg_wasm.cc - escape_json: escape \r \t \b \f and \u00XX for any byte < 0x20. Verified by compiling the function standalone: pre-fix output emitted a raw tab and JSON.parse failed with "Bad control character in string literal at position 13"; post-fix output parses. .github/workflows/build-iceberg-engine.yml - Template-injection: pass the event-controlled ref and the derived version through step `env` instead of interpolating them into `run`, and validate both before use. Add `permissions: contents: read` and checkout `persist-credentials: false`. Regexes verified to accept dev/v26.2.1/ release-25.3.x/SHAs and reject command, newline and $GITHUB_OUTPUT injection attempts. .github/workflows/iceberg-dsl-conformance.yml - Add `persist-credentials: false` and least-privilege permissions. Immutable engine-input pinning (env.sh, fetch-sources.sh, fetch-deps.sh) - Resolve REDPANDA_REF to a commit once, record it in vendor/REDPANDA_COMMIT, and fetch every later input at that same revision, so a build cannot straddle two revisions of `dev`. REDPANDA_REQUIRE_PINNED=1 rejects a mutable ref for release builds. - Stop hardcoding the Avro pin: read the commit, sha256 and patch list from bazel/repositories.bzl at the pinned revision, and verify the archive checksum. Verified against real upstream: v25.3.1 pins avro e54bf712 with 1 patch, dev pins 6821e2b4 with 3 (snappy, fmt-const, libcxx-includes) -- the old hardcoded list applied 2 patches in the wrong order and missed avro-libcxx-includes.patch entirely. - Apply patches with `git apply --unidiff-zero` and GIT_CEILING_DIRECTORIES. Apple's `patch 2.0` rejects avro-libcxx-includes.patch's zero-trailing- context hunks; and without the ceiling, git apply sees the surrounding docs-ui work tree, treats the patch paths as outside its prefix and SILENTLY skips them while exiting 0. A --numstat guard now fails loudly instead. Verified the resulting tree matches a reference patched outside any repo, byte for byte. src/css/iceberg-explorer.css + .stylelintrc - declaration-empty-line-before plus the other 20 stylelint errors in this PR's new stylesheet (rule-empty-line-before, at-rule-empty-line-before, declaration-block-single-line-max-declarations). Allowlist `accent-color` in property-no-unknown, matching the existing `contain`/`aspect-ratio` entries -- this stylelint version's known-properties list predates it. src/ui.yml - Drop the iceberg-engine.wasm/js static_files entries. Neither file is checked in nor produced by any task in `gulp bundle`, and `static_files` promotes a file to the *site root* (see src/partials/console-config-migrator.hbs:6) whereas the loader reads from <uiRoot> = /_/ (27-iceberg-explorer.js uiRoot()). Leaving src/static placement unlisted is what actually serves them from /_/. Verified: `npx gulp bundle` succeeds; `npx gulp lint` clean; `npm run test:iceberg-dsl` 7/7; markdownlint MD040 resolved on the spike README (remaining MD013 warnings pre-date this PR and are not CI-enforced).
Follow-up to the review on the engine-pinning work; three gaps that let a build straddle revisions or ship without provenance. - third_party/fetch-deps.sh: key the Avro cache on the resolved pin ($AVRO_EXTRACT) instead of the avro-cpp symlink. The symlink from an earlier pin still resolved, so a changed pin skipped the fetch, the sha256 verify and the new patch set while reporting the new sha, and wasm/build.sh (which reads third_party/avro-cpp) compiled the old revision. The symlink is now re-pointed on every run. - build-iceberg-engine.yml: set REDPANDA_REQUIRE_PINNED=1 and drop the 'dev' input default, so the only path that publishes engines cannot build from a mutable branch. - build-iceberg-engine.yml: emit dist/iceberg-engine-<V>.provenance.json with the resolved commit, which previously lived only in the git-ignored vendor/REDPANDA_COMMIT and never reached the artifact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Turns the standalone v26.2 "Iceberg Mode Explorer" TOI artifact into a docs-integrated, versioned interactive tool — and swaps its hand-written JavaScript rules for Redpanda's real C++ translation code compiled to WebAssembly.
An author drops the
[iceberg-explorer]block (companion PR below) on a page; this bundle hydrates it into the tool: config controls, the DSL config string, the resulting Iceberg table schema, and a sample translated record.Why WASM
The original re-typed the product's translation rules by hand, so it drifts the moment Redpanda changes. Running the real code in the browser (as the Bloblang playground already does) means the rule-bearing output can't drift, and a per-release build keeps every docs version correct with no manual upkeep.
Engine coverage (deliberate)
iceberg::type_to_iceberg)iceberg_mode_test.ccvectorsThe record path is Seastar-coroutine + serde-parser code with no synchronous entry point; a real-WASM port is a large, separate effort. Everything sits behind a
translate()seam, so any part can move to WASM later without UI changes.Changes
src/js/27-iceberg-explorer.js+src/css/iceberg-explorer.css— hydration module and theme-aware styling (follows the site's light/dark tokens).iceberg-editor/— Emscripten build of the real mapper.wasm-spike/is the reproducible feasibility proof (pinned toolchain: fmt 12.1.0, redpanda-data/avro fork + patches, thin Seastar shim);wasm/is the browser engine build (build.sh+ Embind). Fetched sources, third-party deps, and build outputs are git-ignored and reproducible from the scripts.iceberg-engine-<version>for the page's doc version → bundle asset → interim JS fallback.<body data-version>added todefault.hbs.tests/iceberg-dsl/— config-string conformance vectors copied from Redpanda'siceberg_mode_test.cc; caught and fixed two drift bugs (missing valuelayout, reversed option order).build-iceberg-engine.yml(build versioned engine per release tag) andiceberg-dsl-conformance.yml(run vectors + diff upstream).Companion PR
Needs the block macro: redpanda-data/docs-extensions-and-macros#219.
Test / preview
npm run test:iceberg-dsl→ 7/7 conformance vectors pass.gulp preview→ http://localhost:5252/iceberg-explorer-test.html (verified: real-engine schema, correct config strings, light/dark, two independent instances).iceberg-editor/wasm/build.sh {node|web}(needs emsdk).🤖 Generated with Claude Code