SQL-free direct table API (TableService) + lossless Avro preservation

[jogrogan]

Decouple Proteus DDL

Hoptimator's validation and deployment logic is currently reachable only by issuing SQL DDL through a Calcite/JDBC connection. That couples every caller to Calcite and forces schema definitions through a lossy  Avro → RelDataType → Avro  round-trip. Hoptimator via a CLI or over REST/gRPC may want to create/validate tables by handing us a table name + a schema directly — not by constructing DDL.

This stacked series decouples the validators and deployers from Calcite so the same validation and deployment engine can be driven either by SQL or by a direct, connection-free call — while keeping validators/deployers as a clean SPI. The payoff: an API call to "create table from a schema" path with identical guarantees to DDL (including dry-run), lossless Avro fidelity, and no Calcite dependency for callers that don't want one. It also keeps the door open for follow-ups (e.g. materializations, source→sink field mappings) without re-plumbing.

Stacked series (review/merge in order):

1.  decouple-pr1-spi  — SPI foundation:  java.sql.Connection  →  DeploymentContext  (pure decoupling, no behavior change)
2.  decouple-pr2-api  — SQL-free  TableService  + lossless Avro preservation
3.  decouple-pr3-hardening  — typed transient failures + required  Deployer.exists() 
   (This is PR 2 of 3.)

PR 2/3 — SQL-free direct table API (TableService) + lossless Avro preservation

Base: jogrogan/decouple-pr1-spi · Branch: jogrogan/decouple-pr2-api
Stacked series: 2 of 3 (PR1 SPI foundation → this → PR3 hardening)

Review/merge after PR1. The diff below is against PR1.

Why

With the SPI decoupled from java.sql.Connection (PR1), we can now offer table validation
and creation without Calcite or DDL. This PR adds a connection-free entrypoint that
takes a table path + an Avro schema and runs the same validation and deployment that
CREATE TABLE does. This is the pathway a gRPC service (or any non-JDBC caller) needs:
"here is a table name and an Avro schema — validate it and create it," with a dry-run
option, mirroring what plan/specify do today.

What changes

• TableService.create(...) / TableService.delete(...) — a connection-free entrypoint
  that resolves the target from a path + Avro schema and runs the existing
  validation/deployment pipeline.
  • updateIfExists semantics: create-or-update vs. fail-if-exists.
  • dryRun semantics: render specs without mutating anything (the direct-API analogue of
    plan/specify).
• DirectDeploymentContext — carries the caller's Avro schema through the pipeline
  (connection-free).
• Registry-native DatabaseConfigResolver (SPI + K8s-native impl
  K8sDatabaseConfigResolver) — resolves Database config directly from the registry
  instead of via a Calcite catalog / JDBC connection.
• IdentityQuery — connection-free inter-tier pipeline rendering for logical tables
  (the direct-path equivalent of the identity INSERT INTO ... SELECT).
• Lossless Avro preservation — providedAvroSchema, valueSchemaOf, and the Avro
  key/payload split let deployers use the caller's Avro verbatim, avoiding the lossy
  Avro → RelDataType → Avro round-trip (e.g. nested record namespaces survive).

Integration tests (1:1 with the SQL path)

Adds per-connector *TableServiceIntegrationTests exercising the direct API against real
infra: K8s, Kafka, MySQL, Venice, plus Logical (online/offline tiers) — mirroring the
existing SQL-path integration coverage.

Venice test resolution note: the Venice integration test verifies store registration via
a direct jdbc:venice:// connection rather than the shared catalogs=k8s catalog. The
catalog resolves a table by enumerating the whole schema → the Venice router's bulk
/stores list, a periodically-refreshed cache that lags the controller (where create
writes). Because that refresh window can outlast a whole test run, a just-created store
could be missing for every store created that run (an intermittent "Table not resolvable
after create" flake). A direct connection builds a fresh ClusterSchema per connect()
and resolves the single store via the router's targeted discover_cluster endpoint,
which reflects the create immediately — deterministic, no polling.

Scope / non-goals

• Materializations are not prioritized here, but nothing is designed to preclude them.
• A future source→sink mapping API (source path + sink path + field mapping) is deliberately
  not implemented here, but not blocked either.

Testing

• New unit tests: TableServiceTest, DirectDeploymentContextTest, HoptimatorDriverTest,
  DatabaseConfigResolver(s)/provider tests, K8sDatabaseConfigResolverTest,
  IdentityQueryTest, VeniceDeployerTest additions.
• New integration tests per connector (see above).

Review notes

• ~30 files; additive (new TableService/context/resolver/tests). Existing SQL path
  untouched.
• HoptimatorDriver.rowType()/valueSchema() gain the direct-path (Avro) case alongside
  the Calcite case introduced in PR1 — this is the one dispatch point both paths share; PR3
  does not re-touch it.

[ryannedolan]

love it

[ryannedolan]

This split is the only real wart I've seen so far, but it isn't bad.

[jogrogan]

Yea agreed, I don't love this but this change is specifically for deployers and logical tables sort of cross the boundary between creating a deployer and creating a pipeline so had to do some weird things to get it to work.

[ryannedolan]

neat

[github-actions]

Code Coverage

Overall Project	84.66% -0.18%	🟢
Files changed	87.87%	🟢

File	Coverage
IdentityQuery.java	100%	🟢
DatabaseConfigResolverProvider.java	100%	🟢
DirectDeploymentContext.java	100%	🟢
TableService.java	100%	🟢
K8sDatabaseConfigResolverProvider.java	100%	🟢
K8sDatabaseConfigResolver.java	91.75% -8.25%	🟢
HoptimatorDriver.java	91.45%	🟢
DatabaseConfigResolvers.java	79.17% -20.83%	🟢
LogicalTableDeployer.java	74.19% -4.12%	❌
VeniceDeployer.java	56.42%	🟢