Skip to content

sql: add CREATE METRIC SINK as a durable catalog object (SQL-554) - #37958

Open
mtabebe wants to merge 1 commit into
MaterializeInc:mainfrom
mtabebe:ma/prom-metrics/sql-554-catalog-objects
Open

sql: add CREATE METRIC SINK as a durable catalog object (SQL-554)#37958
mtabebe wants to merge 1 commit into
MaterializeInc:mainfrom
mtabebe:ma/prom-metrics/sql-554-catalog-objects

Conversation

@mtabebe

@mtabebe mtabebe commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Problem:

To maintain cluster metrics we defined the MetricSink compute operator. However, there is no way to ask for one. We need a SQL surface that names a metric sink and a durable catalog representation that survives a restart.

Solution:

Add CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel> and DROP METRIC SINK, gated behind enable_metric_sink. Creating a sink writes a catalog item and nothing else: no dataflow is optimized or shipped, so a sink created today publishes no metrics.

Planning checks that the FROM relation exposes the five columns the operator reads (metric_name, metric_type, labels, value, help). Note: order is not enforced and extra columns are fine. Nullability is not checked.

Metric sinks need no new durable record. They persist as ordinary Items and item_type works the type out from create_sql.

However, the changes to audit and serialization does bump the catalog version.

There are some gaps in:

  • mz_comments has no MetricSink branch, but CommentObjectType has no variant either, so COMMENT ON METRIC SINK does not parse and no such record can exist
  • MZ_DEFAULT_PRIVILEGES has no CASE arm, but ON METRIC SINKS is rejected during planning

Note: enable_metric_sink is off by default

Testing:

  • New test/sqllogictest/metric_sink.slt: the column contract and each way of violating it, FROM targets with no rows to read, IF NOT EXISTS, the flag-off refusal, and the seven views above answering with
    a metric sink present.

Note to reviewers: The most important files to look at are:

  • ddl.rs which has the actual plan for create/drop metric sink
  • create_metric_sink.rs which writes the catalog item
  • objects.rs which defines the item_type
  • mz_catalog.rs small tweak here
  • rbac.rs/acl.rs for privilege handling
  • lib.rs/jsonb.rs for audit logging
  • v90_to_v91.rs the no-op migration
  • metric_sink.slt for what is expected behaviour

The rest is threading this all through

Problem:

To maintain cluster metrics we defined the MetricSink compute operator.
However, there is no way to ask for one. We need a SQL surface that
names a metric sink and a durable catalog representation that survives
a restart.

Solution:

Add `CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel>` and `DROP
METRIC SINK`, gated behind `enable_metric_sink`. Creating a sink writes a
catalog item and nothing else: no dataflow is optimized or shipped, so a
sink created today publishes no metrics.

Planning checks that the `FROM` relation exposes the five columns the
operator reads (`metric_name`, `metric_type`, `labels`, `value`, `help`).
Note: order is not enforced and extra columns are fine. Nullability is not
checked.

Metric sinks need no new durable record. They persist as ordinary `Item`s
and `item_type` works the type out from `create_sql`.

However, the changes to audit and serialization does bump the catalog
version.

There are some gaps in:
- `mz_comments` has no `MetricSink` branch, but `CommentObjectType`
has no variant either, so `COMMENT ON METRIC SINK` does not parse
and no such record can exist
- `MZ_DEFAULT_PRIVILEGES` has no CASE arm, but `ON METRIC SINKS`
is rejected during planning

Note: `enable_metric_sink` is off by default

Testing:

- New `test/sqllogictest/metric_sink.slt`: the column contract and each
 way of violating it, `FROM` targets with no rows to read, `IF NOT
EXISTS`, the flag-off refusal, and the seven views above answering with
 a metric sink present.

Co-Authored-By: Moritz Hoffmann <mh@materialize.com>
@mtabebe
mtabebe force-pushed the ma/prom-metrics/sql-554-catalog-objects branch from 0060131 to a469492 Compare July 30, 2026 00:11
@mtabebe
mtabebe requested review from SangJunBak and antiguru July 30, 2026 15:27
@mtabebe
mtabebe marked this pull request as ready for review July 30, 2026 15:28
@mtabebe
mtabebe requested review from a team as code owners July 30, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant