Skip to content

ClickHouse virtual catalog is not stripped from CREATE TABLE AS SELECT / INSERT source queries (physical layer) #5960

Description

@marsewe

Describe the bug

This is a sibling of #5938 / #5940 ("virtual catalog not stripped from CREATE VIEW source queries"). That fix covered create_view, but the same bug still exists on the physical-table path: _create_table (CTAS) and the INSERT paths strip the injected virtual catalog only from the target table name, not from the source query's FROM tables. As a result ClickHouse receives a 3-part identifier and fails with SYNTAX_ERROR (code 62) whenever a FULL (or incremental) model reads from another model.

Environment

  • sqlmesh 0.236.1
  • A ClickHouse gateway (catalog-unsupported) coexisting with a catalog-aware gateway (BigQuery) in the same project, so scheduler.get_default_catalog_per_gateway() injects a virtual catalog (__ch__) into the ClickHouse adapter.
  • default_gateway: ch

Reproduction

config.yaml:

model_defaults:
  start: 2025-01-01
  dialect: clickhouse

gateways:
  ch:
    connection:
      type: clickhouse
      host: <host>
      username: <user>
      password: <pw>
    state_connection: { type: postgres, host: ..., port: 5432, user: ..., password: ..., database: sqlmesh }
  bigquery:                 # <-- merely defining a catalog-aware gateway triggers virtual-catalog injection
    connection:
      type: bigquery
      method: oauth
      project: my-project
      location: EU
    state_connection: { type: postgres, ... }

default_gateway: ch

models/silver/test_table.sql:

MODEL (name silver.test_table, kind FULL);
SELECT 1 AS first_column;

models/gold/test_table.sql:

MODEL (name gold.test_table, kind FULL);
SELECT *, 2 AS second_column FROM silver.test_table;

Run:

sqlmesh plan

Actual behaviour

"__ch__"."gold"."test_table"
  DatabaseError:
    Code: 62. DB::Exception: Syntax error: failed at position 512 (AS):
    AS "test_table") AS "_subquery". ... (SYNTAX_ERROR)
Error: Plan application failed.

The statement sent to ClickHouse for the physical gold table is:

CREATE TABLE "sqlmesh__gold"."gold__test_table__..." AS
SELECT ...
FROM "__ch__"."sqlmesh__silver"."silver__test_table__..." AS "test_table"   -- <-- 3-part FQN, invalid in ClickHouse

(The nested source table still carries __ch__; ClickHouse only supports database.table.)

Impact

Gateway-managed virtual-layer promotion cannot complete for ClickHouse models
in affected multi-gateway projects. Physical objects may be created before the
promotion failure, leaving a partially applied plan to clean up.

Environment

SQLMesh: 0.236.1
ClickHouse: 26.7

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions