Per upstream server connection metrics via hidden and derived metrics

doc/admin-guide/files/records.yaml.en.rst

@@ -2007,6 +2007,35 @@ Origin Server Connect Attempts
    the connection. Useful when the origin supports keep-alive, removing the time needed to set up a
    new connection from the next request at the expense of added (inactive) connections.
 
+.. ts:cv:: CONFIG proxy.config.http.per_server.connection.metric_enabled INT 0
+   :reloadable:
+
+   Publish per upstream server connection metrics. These metrics are dynamically named, one set per
+   upstream server group or hostname, so the number of them scales with the number of distinct
+   upstream servers seen. See :ref:`per-server-connection-metrics`.
+
+   ===== ======================================================================================
+   Value Effect
+   ===== ======================================================================================
+   ``0`` No per server connection metrics.
+   ``1`` Publish only the per hostname aggregate metrics. The per group metrics from which the
+         aggregates are computed exist internally but are not published.
+   ``2`` Publish the per hostname aggregates and the per group metrics.
+   ===== ======================================================================================
+
+   Level ``2`` can produce a very large number of metrics when the
+   :ts:cv:`match type <proxy.config.http.per_server.connection.match>` includes the address or
+   port, since there is then one set per address and port rather than one per hostname.
+
+.. ts:cv:: CONFIG proxy.config.http.per_server.connection.metric_prefix STRING NULL
+   :reloadable:
+
+   An optional prefix inserted into the per server connection metric names, between the fixed
+   ``proxy.process.http.per_server.<counter>.`` portion of the name and the upstream server group
+   or hostname. Useful to distinguish metrics from separate
+   :ts:cv:`match <proxy.config.http.per_server.connection.match>` configurations sharing the same
+   upstream. See :ref:`per-server-connection-metrics`.
+
 .. ts:cv:: CONFIG proxy.config.http.connect_attempts_rr_retries INT 3
    :reloadable:
    :overridable:

doc/admin-guide/monitoring/statistics/core/http-connection.en.rst

@@ -201,6 +201,64 @@ HTTP Connection
    Current number of TCP connections for tunnels where the far end is the server,
    except for those counted by ``proxy.process.tunnel.current_server_connections_tls``
 
+.. _per-server-connection-metrics:
+
+Per Server Connection Metrics
+-----------------------------
+
+Unlike the metrics above these do not have fixed names. They are created dynamically, one set per
+upstream server group as defined by :ts:cv:`proxy.config.http.per_server.connection.match`, and,
+when the match type is ``both``, one aggregate set per hostname. Whether any of them are published,
+and at what granularity, is controlled by
+:ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. An optional
+:ts:cv:`proxy.config.http.per_server.connection.metric_prefix` can be inserted into the names.
+
+Per group names are ``proxy.process.http.per_server.<counter>.<group>``, where ``<group>`` depends on
+the match type: an IP address, an ``address:port`` pair, a hostname, or, for ``both``,
+``<hostname>.<address:port>``. Per hostname names are
+``proxy.process.http.per_server.<counter>.<hostname>``. Aggregates exist only for match type
+``both``, because that is the only match type with more than one group per hostname; for match type
+``host`` the group name is already the bare hostname, so an aggregate would carry the same name as
+the single group it summarises.
+
+For a group, ``<counter>`` is one of:
+
+current_connection
+   Gauge. The number of connections currently open to the group.
+
+total_connection
+   Counter. The total number of connections ever opened to the group. Never decreases.
+
+blocked_connection
+   Counter. The total number of connection attempts to the group blocked by
+   :ts:cv:`proxy.config.http.per_server.connection.max`. Never decreases.
+
+For a hostname aggregate, ``<counter>`` is one of those three, each summed across the groups of that
+hostname, plus:
+
+current_connection_max
+   Gauge. The largest ``current_connection`` value among the groups of that hostname at the moment
+   of sampling, so the maximum rather than the sum of the groups' current counts. This is useful
+   because :ts:cv:`proxy.config.http.per_server.connection.max` is enforced per group rather than
+   per hostname, so the busiest group is what determines whether connections are about to be
+   blocked. Like ``current_connection`` it rises and falls with traffic and is not a high-water
+   mark. There is no per group ``current_connection_max``; it exists only as a hostname aggregate.
+
+Every published per server metric is recomputed periodically, currently every 5 seconds, rather than
+on every connection event, so a reader sees a value up to that interval old. This is true of the
+hostname aggregates and, at level ``2``, of the published per group metrics as well: those are
+mirrored from the internal ones by the same periodic mechanism, not written as connections open and
+close. It applies to ``current_connection_max`` too, which reports the maximum across groups as of
+the last sample rather than a running peak. To obtain the peak over a longer window, compute a
+maximum over time from this gauge in the monitoring system.
+
+At :ts:cv:`metric_enabled <proxy.config.http.per_server.connection.metric_enabled>` level ``1`` the
+per group metrics still exist internally, since the aggregates are computed from them, but are not
+published. They can be listed with ``traffic_ctl metric match per_server --include-hidden``, which
+reads them directly and so is not subject to the sampling delay above. That visibility is intended
+for debugging and is not a stable interface: the existence, granularity and naming of the per group
+metrics may change independently of the published aggregates.
+
 HTTP/2
 ------
 

doc/appendices/command-line/traffic_ctl.en.rst

@@ -949,13 +949,20 @@ traffic_ctl metric
    Display the current value of the specified statistics.
 
 .. program:: traffic_ctl metric
-.. option:: match REGEX [REGEX...]
+.. option:: match [--include-hidden] REGEX [REGEX...]
 
    :ref:`admin_lookup_records`
 
    Display the current values of all statistics whose names match
    the given regular expression.
 
+.. option:: --include-hidden
+
+   Also match hidden metrics. Hidden metrics are internal metrics that are stored but never
+   published through the normal metrics registry; they are not part of the stable metric
+   contract and may be added, changed, or removed between releases without notice. This
+   option is intended for debugging.
+
 .. program:: traffic_ctl metric
 .. option:: describe RECORD [RECORD...]
 

doc/developer-guide/internal-libraries/Metrics.en.rst

@@ -0,0 +1,198 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+   or more contributor license agreements.  See the NOTICE file
+   distributed with this work for additional information
+   regarding copyright ownership.  The ASF licenses this file
+   to you under the Apache License, Version 2.0 (the
+   "License"); you may not use this file except in compliance
+   with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing,
+   software distributed under the License is distributed on an
+   "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+   KIND, either express or implied.  See the License for the
+   specific language governing permissions and limitations
+   under the License.
+
+.. include:: ../../common.defs
+
+Metrics
+*******
+
+Synopsis
+========
+
+.. code-block:: cpp
+
+    #include "tsutil/Metrics.h"
+
+``ts::Metrics`` is the metrics registry. A metric is a named ``int64_t`` counter or gauge,
+reached either by an integer id or by a pointer to its underlying atomic. This page covers two
+facilities layered on top of it: a separate store for metrics that should not be published, and
+derived metrics that aggregate other metrics.
+
+Metric types
+============
+
+Every metric has a ``ts::Metrics::MetricType``, either ``COUNTER`` (monotonically increasing)
+or ``GAUGE`` (rises and falls). The type is chosen by the facade used to create the metric,
+``ts::Metrics::Counter`` or ``ts::Metrics::Gauge``, and is encoded into the metric id.
+
+.. code-block:: cpp
+
+    auto *hits = ts::Metrics::Counter::createPtr("proxy.process.example.hits");
+    auto *live = ts::Metrics::Gauge::createPtr("proxy.process.example.live");
+
+    ts::Metrics::Counter::increment(hits);
+    ts::Metrics::Gauge::store(live, 5);
+
+The two stores
+==============
+
+There are two entirely separate stores:
+
+``ts::Metrics::instance()``
+   The published store. Everything here is visible to :program:`traffic_ctl`, the JSONRPC API and
+   ``stats_over_http``.
+
+``ts::Metrics::hidden_instance()``
+   The hidden store. Metrics here are recorded normally but are never published.
+
+Hidden metrics exist for high cardinality intermediate values, where the individual values are not
+useful to publish but an aggregate over them is. A separate store is used rather than a
+"hidden" flag on each metric so that hidden metrics are *structurally* unreachable from the
+published store: no consumer can expose one by forgetting to check a flag.
+
+Create a hidden metric with ``createHiddenPtr`` on either facade:
+
+.. code-block:: cpp
+
+    auto *g = ts::Metrics::Gauge::createHiddenPtr("proxy.process.example.per_thing.", thing_name);
+
+    // The ordinary typed mutators work unchanged on a hidden metric.
+    ts::Metrics::Gauge::increment(g);
+    ts::Metrics::Gauge::decrement(g);
+
+``createHiddenPtr`` returns the same correctly typed pointer as ``createPtr``, so a hidden metric is
+read and written with the normal mutators and no cast is needed at the call site. There are two
+overloads on each facade, one taking a name and one taking a prefix and a name.
+
+.. important::
+
+   An id from one store is meaningless in the other. Both stores number their metrics from zero, so
+   passing a hidden id to the published store silently reads a different metric, with no error and
+   no crash. Prefer ``createHiddenPtr``, which returns a pointer and never hands out an id.
+
+Inspecting hidden metrics
+-------------------------
+
+Because hidden metrics are invisible to normal queries, they can be listed explicitly with
+``traffic_ctl metric match --include-hidden``. This sets an additional record type bit which
+is deliberately outside ``RECT_ALL``, so hidden metrics are returned only when asked for by name and
+never as a side effect of a broad query.
+
+.. note::
+
+   Hidden metrics are internal. They are not part of the stable metric contract and may be added,
+   renamed or removed between releases without notice. Do not build monitoring on them; use the
+   published aggregate instead.
+
+Derived metrics
+===============
+
+A derived metric is a published metric whose value is computed from other metrics, its *sources*. A
+source may live in either store, which is the point of the facility: high cardinality sources stay
+hidden while only the aggregate is published.
+
+Sources are combined with one of three operations, ``ts::Metrics::Derived::Op``:
+
+``SUM``
+   Add the sources together. This is the default.
+
+``MAX``
+   The largest source value.
+
+``MIN``
+   The smallest source value.
+
+Declaring aggregates up front
+-----------------------------
+
+``ts::Metrics::Derived::derive()`` takes a list of specifications and is meant for aggregates whose
+sources are all known at startup. Each source may be given as a pointer, an id or a name:
+
+.. code-block:: cpp
+
+    ts::Metrics::Derived::derive({
+      {"proxy.process.example.total", ts::Metrics::MetricType::COUNTER, {a, b, c}},
+      {"proxy.process.example.peak",  ts::Metrics::MetricType::GAUGE,   {a, b, c},
+        ts::Metrics::Derived::Op::MAX},
+    });
+
+A source that does not resolve, because the name or id is unknown, is skipped.
+
+Building aggregates at runtime
+------------------------------
+
+``ts::Metrics::Derived::add_source()`` adds a single source to a derived metric, creating the
+derived metric if it does not exist yet. Use it when sources are discovered as the process runs, for
+example one per upstream server as traffic arrives:
+
+.. code-block:: cpp
+
+    ts::Metrics::Derived::add_source("proxy.process.example.total", ts::Metrics::MetricType::COUNTER,
+                                     per_thing_metric);
+
+Repeatedly calling ``ts::Metrics::Derived::derive()`` for the same derived name does **not** work
+for this: each call appends a separate entry targeting the same metric, so every update overwrites
+the others with its own subset of sources and the last one to run silently wins.
+``ts::Metrics::Derived::add_source()`` accumulates into a single entry instead.
+
+Adding a source that is already registered for that derived metric is a no-op, so a caller which may
+re-register the same source, such as one recreating an object for the same key, need not track that
+itself. The ``type`` and ``op`` arguments are ignored if the derived metric already exists.
+
+A hidden source can feed a published aggregate:
+
+.. code-block:: cpp
+
+    auto *hidden = ts::Metrics::Gauge::createHiddenPtr("per_thing.", name);
+
+    ts::Metrics::Derived::add_source("proxy.process.example.live", ts::Metrics::MetricType::GAUGE,
+                                     hidden, ts::Metrics::Derived::Op::SUM);
+
+When derived values update
+--------------------------
+
+Derived metrics are not recomputed when a source changes. They are recalculated by
+``ts::Metrics::Derived::update_derived()``, which runs on an ``ET_TASK`` thread every
+``REC_RAW_STAT_SYNC_INTERVAL_MS``, currently 5000 ms. Consequences:
+
+* A derived value lags its sources by up to one interval.
+* Reading a derived metric immediately after changing a source returns the previous value. Unit
+  tests must call ``ts::Metrics::Derived::update_derived()`` directly.
+* The cost of the pass is proportional to the total number of registered sources, and it runs
+  single threaded while holding a lock. Registering very large numbers of sources is therefore not
+  free, even though registration itself is rare.
+
+Because the pass samples its sources, a derived ``MAX`` reports the largest value *observed at a
+sampling point*, not the true peak. There are two ways to arrange this, with different tradeoffs:
+
+* A ``MAX`` over instantaneous gauges is sampled, so a brief spike occurring between two samples is
+  not observed. The value rises and falls with the sources, so a monitoring system that scrapes it
+  can compute a maximum over any time window.
+* A ``MAX`` over monotonically increasing sources, such as each source's own all-time peak, is exact
+  and never misses a spike. It also never decreases, so the time dimension is lost: the value
+  reports only that a peak occurred at some point, not when.
+
+Which is appropriate depends on whether the consumer needs to aggregate over time downstream.
+
+Storage limits
+==============
+
+Metrics are allocated from fixed size blobs, ``MAX_BLOBS`` of ``MAX_SIZE`` entries each, for a
+maximum of about 8M metrics per store. Creating a metric when the store is full returns the reserved
+``bad_id`` rather than growing past the end, so an exhausted store degrades to writing into a
+throwaway slot instead of corrupting memory. Reaching this limit means the naming scheme is
+unbounded, and hidden metrics with per-connection or per-URL names are the likely cause.

doc/developer-guide/internal-libraries/index.en.rst

@@ -31,6 +31,7 @@ development team.
    ArgParser.en
    MemArena.en
    MemSpan.en
+   Metrics.en
    TextView.en
    buffer-writer.en
    scalar.en