sql: add CREATE METRIC SINK as a durable catalog object (SQL-554) - #37958
Open
mtabebe wants to merge 1 commit into
Open
sql: add CREATE METRIC SINK as a durable catalog object (SQL-554)#37958mtabebe wants to merge 1 commit into
mtabebe wants to merge 1 commit into
Conversation
Problem: To maintain cluster metrics we defined the MetricSink compute operator. However, there is no way to ask for one. We need a SQL surface that names a metric sink and a durable catalog representation that survives a restart. Solution: Add `CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel>` and `DROP METRIC SINK`, gated behind `enable_metric_sink`. Creating a sink writes a catalog item and nothing else: no dataflow is optimized or shipped, so a sink created today publishes no metrics. Planning checks that the `FROM` relation exposes the five columns the operator reads (`metric_name`, `metric_type`, `labels`, `value`, `help`). Note: order is not enforced and extra columns are fine. Nullability is not checked. Metric sinks need no new durable record. They persist as ordinary `Item`s and `item_type` works the type out from `create_sql`. However, the changes to audit and serialization does bump the catalog version. There are some gaps in: - `mz_comments` has no `MetricSink` branch, but `CommentObjectType` has no variant either, so `COMMENT ON METRIC SINK` does not parse and no such record can exist - `MZ_DEFAULT_PRIVILEGES` has no CASE arm, but `ON METRIC SINKS` is rejected during planning Note: `enable_metric_sink` is off by default Testing: - New `test/sqllogictest/metric_sink.slt`: the column contract and each way of violating it, `FROM` targets with no rows to read, `IF NOT EXISTS`, the flag-off refusal, and the seven views above answering with a metric sink present. Co-Authored-By: Moritz Hoffmann <mh@materialize.com>
mtabebe
force-pushed
the
ma/prom-metrics/sql-554-catalog-objects
branch
from
July 30, 2026 00:11
0060131 to
a469492
Compare
mtabebe
marked this pull request as ready for review
July 30, 2026 15:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
To maintain cluster metrics we defined the MetricSink compute operator. However, there is no way to ask for one. We need a SQL surface that names a metric sink and a durable catalog representation that survives a restart.
Solution:
Add
CREATE METRIC SINK <name> IN CLUSTER <c> FROM <rel>andDROP METRIC SINK, gated behindenable_metric_sink. Creating a sink writes a catalog item and nothing else: no dataflow is optimized or shipped, so a sink created today publishes no metrics.Planning checks that the
FROMrelation exposes the five columns the operator reads (metric_name,metric_type,labels,value,help). Note: order is not enforced and extra columns are fine. Nullability is not checked.Metric sinks need no new durable record. They persist as ordinary
Items anditem_typeworks the type out fromcreate_sql.However, the changes to audit and serialization does bump the catalog version.
There are some gaps in:
mz_commentshas noMetricSinkbranch, butCommentObjectTypehas no variant either, soCOMMENT ON METRIC SINKdoes not parse and no such record can existMZ_DEFAULT_PRIVILEGEShas no CASE arm, butON METRIC SINKSis rejected during planningNote:
enable_metric_sinkis off by defaultTesting:
test/sqllogictest/metric_sink.slt: the column contract and each way of violating it,FROMtargets with no rows to read,IF NOT EXISTS, the flag-off refusal, and the seven views above answering witha metric sink present.
Note to reviewers: The most important files to look at are:
The rest is threading this all through