feat(kernel): SEA-via-kernel backend (opt-in, behind build tag)

[mani-mathur-arch]

Summary

Add a second execution backend that runs statements over the Statement Execution
API via the Rust databricks-sql-kernel, reached through a cgo C ABI. It is
opt-in behind WithUseKernel(true) + the databricks_kernel build tag, so the
default build stays pure-Go (CGO_ENABLED=0) and is unchanged — the only new
user-facing surface is WithUseKernel.

The user API is otherwise identical to Thrift: the kernel backend reads the same
config.Config the Thrift backend reads and routes those options to the kernel
internally (the kernel setters are never exposed to the user), mirroring how the
kernel's pyo3 / napi bindings work.

Testing & CI. The kernel path is verified locally — the cgo binding
links a kernel static library whose header and lib locations are supplied at
build time via the standard CGO_CFLAGS / CGO_LDFLAGS env vars (the #cgo
directives carry no hardcoded paths), and the test plan below was run against
that build. CI coverage for the kernel path (a tagged build + test job) is
not wired yet and will land in a follow-up PR — it will build the kernel from
source at a pinned revision for dev/test/CI, and at driver-release time link a
kernel production release. The exact distribution/linkage mechanism is still
being worked out and is out of scope here. The pure-Go default build
(CGO_ENABLED=0) and its CI are unaffected and green in this PR.

What's implemented

Verified with unit tests + live e2e against a staging warehouse, and a
Thrift-parity check asserting both backends render results identically:

• Auth: PAT
• Query execution: SQL read, Direct Results (inline), Cloud Fetch, ctx
  cancellation (watcher goroutine → the kernel's detached statement canceller),
  server statement timeout, query tags
• Data types: primitives, binary, timestamp/date (in the session time zone),
  and the complex types — array, map, struct, variant — all rendered to JSON
  byte-identical to Thrift; geometry (WKT). DECIMAL is rendered exactly, matching
  Thrift: a fixed-point string for a top-level column and an exact JSON number
  inside a nested value (never a lossy float64).
• Connectivity: TLS skip-verify, HTTP proxy (from the standard proxy env,
  same decision Thrift makes), SPOG org routing (via the http path ?o=)
• Observability: opt-in debug logging via DBSQL_KERNEL_DEBUG, which turns on
  the binding's step tracer and installs the kernel's own Rust log subscriber so
  both interleave on stderr; off by default (and during benchmarks). Kernel
  verbosity is controlled by RUST_LOG (target databricks::sql::kernel).
• Errors: kernel error → the driver's error surface with sqlstate; a
  session-unusable status on the session-lifecycle path maps to
  driver.ErrBadConn so the pool evicts the conn, while the execute/read path
  never returns ErrBadConn (so a statement is never silently re-run)
• Inherited from the kernel (verified, not re-implemented): retry, backoff,
  async HTTP, parallel Cloud Fetch

Deferred (rejected loudly at connect time where applicable)

Options that aren't wired yet return a clear error rather than behaving
differently than Thrift:

• Initial namespace (catalog/schema) — the kernel C ABI has no setter yet
  (needs a kernel change)
• Metric-view metadata — maps to a server session conf; deferred to route
  backend-neutrally rather than duplicate a Thrift-specific literal
• Richer TLS setup — only InsecureSkipVerify is honored today (mapped to
  both kernel relaxations: chain + hostname, matching crypto/tls semantics and
  the pyo3/napi bindings). The kernel's fuller TLS surface — a trusted-CA bundle,
  an independent hostname-only skip, and mTLS client cert/key — is not exposed
  yet. Doing so means adding new Go config params/setters (Go's native path
  honors only InsecureSkipVerify), so it's new public API rather than parity and
  is deferred to a follow-up.
• Bound query parameters — rejected with a clear error at execute time
  (they arrive per-query, not at connect); previously they were silently dropped.
• WithTimeout (server query timeout) and WithRetries used to disable
  retries — rejected at connect (the kernel C ABI has no timeout setter and
  retries internally). WithMaxRows / positive retry tuning are accepted but
  kernel-managed (documented, not applied).

OAuth (M2M/U2M) is not offered; a non-PAT authenticator is rejected with a clear
error at connect rather than reaching the kernel as an empty PAT. A PAT supplied
via WithAuthenticator(&pat.PATAuth{...}) is honored (token sourced from the
authenticator). Metadata is issued as ordinary SQL and runs like any other query.

Structure

• internal/backend/kernel/ (//go:build cgo && databricks_kernel): the cgo
  binding — cgo.go (FFI-safe call helper + error mapping + logging init),
  backend.go (session open + config), operation.go (blocking execute +
  watcher cancel), rows.go (zero-copy Arrow C Data Interface import; delegates
  cell rendering to internal/arrowscan).
• internal/arrowscan/: pure-Go Arrow cell → driver.Value rendering, including
  the nested List/Map/Struct → JSON grammar (native float32, exact decimals,
  time.Time formatting) that must stay byte-identical to Thrift. Untagged, so
  its rendering tests run in the default CGO_ENABLED=0 matrix rather than being
  dead behind the kernel build tag.
• internal/decimalfmt/: the exact fixed-point DECIMAL formatter, a dependency-
  free leaf package shared by the Thrift (arrowbased) and kernel result paths
  so a #274-class precision fix lands in both at once. Untagged, so its unit
  test runs in the default CGO_ENABLED=0 matrix.
• Pure-Go side: WithUseKernel / WithWarehouseID options, UseKernel /
  WarehouseID config + DSN parsing, a build-tagged stub that returns a clear
  error when the kernel backend isn't compiled in, and the factory branch in
  connector.Connect.

Test plan

• CGO_ENABLED=0 go build ./... + full pure-Go suite pass (default build unchanged)
• go build -tags databricks_kernel (CGO_ENABLED=1) compiles and links
• Tagged unit tests: error/status mapping, scalar + nested scanning, timezone
  rendering, proxy resolution, unsupported-option rejection
• Live e2e against a staging warehouse: select, per-type data types,
  CloudFetch (1M rows), ctx cancellation, query tags / statement timeout
  (read back via SET), timezone, TLS skip-verify
• Thrift-parity: same query (scalars, complex types, nested decimal) through
  both backends renders identical rows
• Debug logging: kernel Rust logs interleave with the binding's on stderr,
  verified live on both the success and connect-failure paths
• New default-build tests: internal/arrowscan cell/nested-JSON rendering
  (scalars, float32, exact decimals, list/map/struct, timezone), a
  cross-backend parity test feeding the same arrow.Record through both
  arrowscan and the Thrift arrowbased renderer (special-char struct keys,
  map keys, float32, decimal), internal/decimalfmt exact formatting, the
  proxy resolver seam (4 branches), auth/timeout/retry rejects + PAT-via-
  authenticator, useKernel/warehouseId DSN parse + DeepCopy, and the
  "kernel not compiled in" stub error — all run under CGO_ENABLED=0
• golangci-lint (v2.12.2, the CI version) clean repo-wide
• Reviewed with isaac review and a follow-up review pass (findings addressed
  — see below)

Code-review remediation

Four review passes surfaced correctness/contract defects (GA-readiness — the
backend has no production callers today, being behind the double opt-in). All
addressed and verified against both the default and kernel-tagged builds, plus
live e2e + Thrift-parity against a staging warehouse.

Round 1 — correctness & contract:

• AffectedRows for DML — cache the modified-row count at execute time.
  conn.ExecContext closes the op (nulling the executed handle) before reading
  AffectedRows, so the previous live read returned 0 for every
  INSERT/UPDATE/DELETE/MERGE.
• Fail-loud, per the doc contract — reject a non-PAT authenticator (OAuth /
  token-provider / external / federated) at connect, and reject bound parameters
  at execute, instead of silently sending an empty PAT / dropping the params.
• Nested FLOAT parity — marshal the native float32, not a widened
  float64, so ARRAY/MAP/STRUCT<FLOAT> match Thrift byte-for-byte (3.14, not
  3.140000104904175).
• Context cancellation — fail fast on an already-cancelled ctx in the read
  (nextBatch) and OpenSession paths before entering the blocking C calls.
• DECIMAL dedup — one shared internal/decimalfmt formatter (was duplicated
  verbatim across the two backends).
• Observability — OnChunkFetched reports a cumulative chunk count; kernel
  errors also log at the driver's default (Warn) level (no SQL/PII) so a failure
  is visible without DBSQL_KERNEL_DEBUG; klog logs sql.len, not raw SQL.

Round 2 — regressions from round 1, incomplete fixes, and coverage:

• Memory leak (High) — a round-1 change had memoized struct field-name JSON
  prefixes in a process-global map keyed by *arrow.StructType. The Arrow C Data
  import allocates a fresh *StructType per batch (no interning), so the map grew
  one never-evicted entry per batch — an unbounded leak on multi-batch struct
  results. Removed; the field name is marshaled inline.
• Top-level FLOAT parity — round 1 fixed only the nested path; a bare FLOAT
  column still widened to float64, so CAST(0.1 AS FLOAT) rendered
  0.10000000149011612 vs Thrift's 0.1. The scalar arm now returns native
  float32; the parity + e2e tests gain a non-exactly-representable FLOAT case.
• Close() contract — the bound-params error path returns a handle-less
  &kernelOp{}; Close() now reports closed=false for it (was true), so no
  phantom CLOSE_STATEMENT telemetry is recorded for a statement that never hit
  the server.
• Renderer extraction (internal/arrowscan) — the cell/nested-JSON grammar
  moved out of the cgo-tagged kernel package into an untagged leaf package, so its
  rendering tests (float32, exact decimals, nested grammar) now run in the default
  CGO_ENABLED=0 matrix instead of being dead behind the build tag.
• Honest docs — corrected a false comment claiming database/sql's watcher
  tears down an in-flight kernel fetch (it can't — Rows.closemu blocks close
  until the blocking Next returns; read-path cancellation is batch-boundary
  only); documented that StatementID() is "" because the kernel C ABI has no
  success-path id accessor, so per-statement telemetry does not fire (tracked as a
  kernel follow-up); dropped a stale "metadata commands error at connect" claim
  (metadata runs as ordinary SQL).
• Test hardening — the proxy test now injects a resolver seam to assert
  proxy-set / NO_PROXY / direct outcomes (was passable by a return "" stub);
  removed a duplicated test helper and rotting review/PR labels from comments.

Round 3 — grammar divergence, fail-loud holes, and CI reach:

• Struct-key JSON escaping (High) + a CI-visible parity guard — the nested-JSON
  grammar is rendered independently by arrowscan (kernel) and arrowbased
  (Thrift), and they had diverged: arrowscan json.Marshals struct field-name
  keys (escaped), while Thrift did a raw " + name + " concat that produces
  invalid JSON for a name with a quote/backslash/control char (a"b →
  {"a"b":1}). Fixed the Thrift path to escape. The only prior guard
  (TestKernelThriftParity) is build-tagged and needs a live warehouse, so it
  never ran in CI — added an untagged cross-backend parity test that feeds the
  same arrow.Record through both renderers and asserts byte-equality in the
  default matrix (special-char struct keys, string/special/int map keys, float32,
  decimal). Verified it fails on the pre-fix concat and passes after.
• Fail-loud holes (auth / timeout / retries) — WithAuthenticator(&pat.PATAuth{})
  passed the auth guard but shipped an empty token (the token was read from
  cfg.AccessToken, which only WithAccessToken sets) → opaque Unauthenticated
  for a valid, Thrift-supported config; now the token is sourced from the
  authenticator and an empty resolved token is rejected. WithTimeout (a server
  query timeout the kernel C ABI can't set) and WithRetries used to disable
  retries now error at connect instead of being silently dropped; WithMaxRows and
  positive retry tuning are documented as accepted-but-kernel-managed.
• queryId surfaced — the kernel error already parsed the server query_id;
  it is now included in the default-level Warn log and Error(), restoring the one
  correlation handle to server-side query history (StatementID() is still "").
• Proxy test reaches CI — the pure-Go proxy helper moved to an untagged file
  (kernel_proxy.go), so its four-branch test now runs under CGO_ENABLED=0
  instead of being trapped behind the kernel tag.

Round 4 — a data-correctness hazard, a self-inflicted regression, and CI reach:

• No ErrBadConn on the execute path (High) — a network/unavailable status
  surfaced after a statement was sent was classified as driver.ErrBadConn,
  which makes database/sql transparently re-run the statement — a silent
  duplicate write for a non-idempotent INSERT/UPDATE/MERGE. Split classification:
  toConnError (session lifecycle — nothing executed) keeps the bad-conn mapping
  so the pool evicts a dead session; toStatementError (execute + result read)
  never returns ErrBadConn. This restores the contract the rest of the stack
  already honors (the kernel classifies ExecuteStatement NonIdempotent; Thrift
  marks it non-retryable; Go's ErrBadConn docs forbid it when the server may have
  performed the op).
• Per-row struct-key regression fixed — round 3's escaping fix ran
  json.Marshal(fieldName) in structValueContainer.Value, the per-row hot path
  of the default Thrift backend, re-marshaling constant field names N_rows ×
  N_fields times. Precompute the escaped "name": prefixes once at container
  construction (the Thrift fieldNames slice is per-result-stable); byte-identical
  output, zero per-row marshal.
• Map-key parity for []byte — arrowscan's map-key path marshaled
  fmt.Sprintf("%v", key), so a MAP<BINARY,_> key rendered "[97 98 99]" vs
  Thrift's base64 "YWJj". Now marshals the key value directly (matching Thrift);
  parity test gains binary/date map-key cases + recursive-nesting, nested-timestamp,
  and null-leaf shapes.
• Config validation reaches CI + a dropped-field guard — the auth/timeout/retry
  guards lived in tag-gated code that CI never compiled. Extracted the pure-Go
  validation into an untagged validateKernelConfig (its full reject/auth tests
  now run under CGO_ENABLED=0), plus a reflective test that fails if any
  config.UserConfig field is unclassified — so a future option can't be silently
  dropped on the kernel path.
• Doc caveats — INTERVAL is not handled by the kernel scanner (errors);
  WithEnableMetricViewMetadata added to the connect-time reject list;
  QueryIdCallback fires with "" on the kernel backend (no per-statement id); a
  back-pointer on the Thrift struct renderer to its arrowscan mirror + the parity
  guard.

Deferred (documented, not silently dropped):

• CI — a CGO_ENABLED=1 -tags databricks_kernel test job needs the kernel lib
  linked in CI, which is the distribution follow-up above; go.yml documents the
  gap. The pure-Go renderers (arrowscan, decimalfmt) are already covered by the
  default matrix.
• Mid-fetch cancellation — real interruption of an in-flight CloudFetch batch
  needs the execute-path watcher applied to the read path (or a kernel stream-level
  abort); today it is batch-boundary only.
• Full renderer single-sourcing — the Thrift arrowbased path still has its
  own nested renderer (built on its columnValues container abstraction); having
  it delegate to arrowscan too is a separate, higher-risk refactor of the primary
  production path. The correctness + drift-guard gap is already closed by the
  round-3 cross-backend parity test; this is now purely a dedup follow-up.
• Statement-id telemetry — StatementID() is "" (no kernel C-ABI success-path
  accessor), so per-statement metrics don't fire. The server queryId from the
  error path is now logged for failures; full telemetry needs the kernel accessor.

---

This pull request and its description were written by Isaac.

[github-actions]

Go integration tests triggered (passthrough). View workflow runs.

[github-actions]

Go integration tests triggered (passthrough). View workflow runs.

[github-actions]

Integration test approval reset.

New commits were pushed to this PR. Label(s) integration-test-full were removed for security.

A maintainer must re-review and re-add a label to preview tests again. (The real gate runs in the merge queue.)

Latest commit: 5852480