Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,6 @@
"reference/core/architecture/adrs/core/0102-evolith-agent-runtime.md",
"reference/core/architecture/adrs/core/0103-architecture-planning-gate-intake.es.md",
"reference/core/architecture/adrs/core/0103-architecture-planning-gate-intake.md",
"reference/core/architecture/adrs/core/0106-master-tenant-context-projections.es.md",
"reference/core/architecture/adrs/core/0106-master-tenant-context-projections.md",
"reference/core/architecture/adrs/core/0107-single-cluster-kubernetes-deployment-topology.es.md",
"reference/core/architecture/adrs/core/0107-single-cluster-kubernetes-deployment-topology.md",
"reference/core/architecture/adrs/core/0108-masstransit-owned-message-topology.es.md",
Expand Down
24 changes: 13 additions & 11 deletions product/operations/alerts/prometheus-alerts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ groups:
summary: "Error budget burning too fast (6h window)"
description: "6h burn rate {{ $value | humanizePercentage }} exceeds 6x threshold"

# Master-data projection messaging (ADR-0108 / deployment strategy §5.3, GT-463).
# Tenant-snapshot projection messaging (ADR-0108 / ADR-0129 / deployment strategy §5.3, GT-463).
# MassTransit moves a message that exhausts its retries into `<queue>_error` — it never
# nacks, so the broker DLX never fires. ANY message in an `_error` queue is a poison
# message that needs a human: alert on depth > 0 (not a backlog threshold).
Expand All @@ -166,7 +166,7 @@ groups:
# on a series nobody emits never fires, and a rule that cannot fire is worse than no rule
# because it reads as coverage.
#
# The bus going away is LOSSLESS, not incorrect: MMS's transactional outbox commits and
# The bus going away is LOSSLESS, not incorrect: UMS's transactional outbox commits and
# drains on reconnect, consumers idle and catch up, and readiness never gates on the broker
# (deployment strategy §5.4). So these are freshness alerts, and their severity says so.
- alert: MessageBusDisconnected
Expand All @@ -181,39 +181,41 @@ groups:
service: messaging
annotations:
summary: "The message bus is unreachable"
description: "No `rabbitmq_queue_messages` series is being reported: the broker or its exporter is unreachable. Master-data projection is FROZEN, not lost — the producer's outbox commits and drains on reconnect. Correctness is unaffected; freshness is not."
description: "No `rabbitmq_queue_messages` series is being reported: the broker or its exporter is unreachable. Tenant-snapshot projection is FROZEN, not lost — the producer's outbox commits and drains on reconnect. Correctness is unaffected; freshness is not."
runbook_url: "https://github.com/beyondnetcode/evolith/blob/main/product/operations/incident-response-poison-message-reprocess.md"

- alert: TenantProjectionLag
# A projection queue in steady state is near-empty: the consumer drains it as fast as MMS
# A projection queue in steady state is near-empty: the consumer drains it as fast as UMS
# publishes. Sustained depth means the consumer is not keeping up, which is the freshness
# symptom this row exists for.
#
# The threshold is SEMANTIC, not measured: `> 0 for 10m` means "not draining", which is a
# property of the queue's purpose rather than a percentile nobody has taken. The generic
# `RabbitMQQueueDepth` above only fires at 1000, and on a tenant-projection queue that
# `RabbitMQQueueDepth` above only fires at 1000, and on a tenant-snapshot queue that
# number means freshness has been broken for a long time already. Re-tune this from real
# traffic once there is any — and until then, an honest semantic bound beats a fabricated
# numeric one.
expr: rabbitmq_queue_messages{queue=~".*\\.tenant-projection"} > 0
# One consumer, one queue: UMS publishes the snapshot and consumes nothing of it
# (ADR-0129), so this names `tracker.tenant-snapshot` rather than matching a family.
expr: rabbitmq_queue_messages{queue="tracker.tenant-snapshot"} > 0
for: 10m
labels:
severity: warning
service: messaging
annotations:
summary: "Tenant projection is lagging on {{ $labels.queue }}"
description: "{{ $labels.queue }} has held {{ $value }} undelivered message(s) for 10 minutes. A projection queue in steady state is near-empty, so this means the consumer is not draining. Tenant master data served by this product is STALE, not wrong."
summary: "The tenant snapshot is lagging on {{ $labels.queue }}"
description: "{{ $labels.queue }} has held {{ $value }} undelivered message(s) for 10 minutes. A projection queue in steady state is near-empty, so this means the consumer is not draining. The tenant data this product serves is STALE, not wrong."
runbook_url: "https://github.com/beyondnetcode/evolith/blob/main/product/operations/incident-response-poison-message-reprocess.md"

# A projection consumer endpoint that never binds its queue means the pod is Ready but
# the consumer is dead (the 406 redeclare failure ADR-0108 retired the CRDs to prevent).
- alert: TenantProjectionQueueMissing
expr: absent(rabbitmq_queue_messages{queue="ums.tenant-projection"}) or absent(rabbitmq_queue_messages{queue="tracker.tenant-projection"})
expr: absent(rabbitmq_queue_messages{queue="tracker.tenant-snapshot"})
for: 5m
labels:
severity: critical
service: messaging
annotations:
summary: "A tenant-projection consumer endpoint queue is absent"
description: "ums.tenant-projection and/or tracker.tenant-projection is not declared on the broker — a consumer endpoint failed to start (dead consumer behind a Ready pod)."
summary: "The tenant-snapshot consumer endpoint queue is absent"
description: "tracker.tenant-snapshot is not declared on the broker — the consumer endpoint failed to start (dead consumer behind a Ready pod)."
runbook_url: "https://github.com/beyondnetcode/evolith/blob/main/product/operations/incident-response-poison-message-reprocess.md"
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,22 @@

> **Navegación bilingüe:** [View English version](./incident-response-poison-message-reprocess.md)

Playbook operativo para mensajes fallidos de la proyección de datos maestros. Se dispara desde la
Playbook operativo para mensajes fallidos de la proyección del retrato de tenant. Se dispara desde la
alerta `MassTransitPoisonMessages` (estrategia de despliegue §5.3 · ADR-0108 · GT-463).

## Qué lo disparó

MassTransit reintenta un consumo fallido (exponencial, 5 intentos). Agotados los reintentos,
**mueve** el mensaje a `<cola-endpoint>_error` — nunca hace `nack`, así que el dead-letter exchange
del broker jamás se dispara. **Cualquier** mensaje en una cola `_error` es un mensaje veneno que un
consumidor no pudo procesar. Las colas afectadas para la proyección de tenant son:
consumidor no pudo procesar. El retrato de tenant tiene exactamente un consumidor, así que solo una
cola puede guardar veneno suyo:

- `ums.tenant-projection_error`
- `tracker.tenant-projection_error`
- `tracker.tenant-snapshot_error`

UMS publica el retrato y no consume nada de él (ADR-0129 · ADR-UMS-107), así que no hay cola `ums.*`
de proyección que shovelear. Las `ums.tenant-projection_error` y `tracker.tenant-projection_error`
que este playbook nombró hasta el 2026-08-22 eran del consumidor de la era MMS que T-059 retiró.

## Severidad

Expand Down Expand Up @@ -41,31 +45,34 @@ Cuando la causa sea transitoria o esté corregida, mueve los mensajes de vuelta

```bash
kubectl exec -n evolith-messaging evolith-rabbitmq-server-0 -c rabbitmq -- \
rabbitmqctl set_parameter shovel reprocess-ums-projection \
'{"src-protocol":"amqp091","src-uri":"amqp:///","src-queue":"ums.tenant-projection_error",
"dest-protocol":"amqp091","dest-uri":"amqp:///","dest-queue":"ums.tenant-projection",
rabbitmqctl set_parameter shovel reprocess-tenant-snapshot \
'{"src-protocol":"amqp091","src-uri":"amqp:///","src-queue":"tracker.tenant-snapshot_error",
"dest-protocol":"amqp091","dest-uri":"amqp:///","dest-queue":"tracker.tenant-snapshot",
"src-delete-after":"queue-length","ack-mode":"on-confirm"}'
```

- `src-delete-after: queue-length` drena solo los mensajes presentes al iniciar el shovel (no hace
bucle infinito con los que fallen después).
- Repite con `tracker.tenant-projection_error` → `tracker.tenant-projection` para Tracker.
- Borra el shovel al terminar: `rabbitmqctl clear_parameter shovel reprocess-ums-projection`.
- Borra el shovel al terminar: `rabbitmqctl clear_parameter shovel reprocess-tenant-snapshot`.

## Verificar

1. La profundidad de la cola `_error` vuelve a 0 y la alerta `MassTransitPoisonMessages` se limpia.
2. La proyección se pone al día: `masterdata_projection_applied_total` sube; sin un alza equivalente
en `masterdata_projection_discarded_total` más allá de los descartes esperados por la guarda de stale.
3. Verifica puntualmente el/los tenant(s) afectados en `masterdata.tenant_projection` (UMS + Tracker)
contra MMS.
2. La proyección se pone al día: `tracker.tenant-snapshot` drena y se queda casi vacía. No hay
contador de aplicados/descartados que leer — las métricas `masterdata_projection_*` de la era MMS
que aquí se nombraban hasta el 2026-08-22 nunca tuvieron emisor, y T-059 no introdujo
sustitutas. La señal es la profundidad de la cola.
3. Verifica puntualmente el/los tenant(s) afectados: la fila de `tenants` del Tracker lleva el
`ums_projection_version` que UMS publicó por última vez, y `code`/`name`/`status` coinciden con
UMS, que es el maestro (ADR-0129).

## Prevenir recurrencia

- Veneno-por-contenido persistente → el productor emitió un evento fuera de contrato: reconcilia
contra `Evolith.Messaging.Contracts` (DS-12) y añade un contract test del productor.
- Faults transitorios repetidos → revisa la salud de la BD del consumidor (`MasterDataDb`) y la
postura de readiness (readiness nunca debe condicionarse a AMQP — §5.4).
- Faults transitorios repetidos → revisa la base de datos del Tracker donde escribe el consumidor
(T-059 retiró la cadena de conexión aparte `MasterDataDb`; la proyección aterriza en el esquema
propio del Tracker) y la postura de readiness (readiness nunca debe condicionarse a AMQP — §5.4).

## Referencias
- Estrategia de despliegue §5.3 (manejo de veneno) · ADR-0108 (topología propiedad de MassTransit) · GT-463.
Expand Down
39 changes: 24 additions & 15 deletions product/operations/incident-response-poison-message-reprocess.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,23 @@

> **Bilingual Navigation:** [Versión en Español](./incident-response-poison-message-reprocess.es.md)

Operational playbook for handling faulted master-data projection messages. Fires from the
Operational playbook for handling faulted tenant-snapshot projection messages. Fires from the
`MassTransitPoisonMessages` alert (deployment strategy §5.3 · ADR-0108 · GT-463).

## What triggered this

MassTransit retries a failing consume (exponential, 5 attempts). When retries are exhausted it
**moves** the message to `<endpoint-queue>_error` — it never `nack`s, so the broker's dead-letter
exchange never fires. **Any** message sitting in an `_error` queue is a poison message a consumer
could not process. The affected queues for the tenant projection are:
could not process. The tenant snapshot has exactly one consumer, so exactly one queue can hold
poison for it:

- `ums.tenant-projection_error`
- `tracker.tenant-projection_error`
- `tracker.tenant-snapshot_error`

UMS publishes the snapshot and consumes nothing of it (ADR-0129 · ADR-UMS-107), so there is no
`ums.*` projection queue to shovel. The `ums.tenant-projection_error` and
`tracker.tenant-projection_error` this playbook named until 2026-08-22 belonged to the MMS-era
consumer that T-059 removed.

## Severity

Expand All @@ -27,7 +32,7 @@ could not process. The affected queues for the tenant projection are:
1. Inspect a faulted message (RabbitMQ management UI → the `_error` queue → *Get messages*, requeue=true):
```bash
kubectl port-forward -n evolith-messaging svc/evolith-rabbitmq 15672:15672
# open http://localhost:15672 → Queues → ums.tenant-projection_error
# open http://localhost:15672 → Queues → tracker.tenant-snapshot_error
```
2. Read the MassTransit fault headers (`MT-Fault-Message`, `MT-Fault-StackTrace`) on the message.
3. Classify the root cause **before** reprocessing:
Expand All @@ -41,30 +46,34 @@ Once the root cause is transient or fixed, move the messages back with a **dynam

```bash
kubectl exec -n evolith-messaging evolith-rabbitmq-server-0 -c rabbitmq -- \
rabbitmqctl set_parameter shovel reprocess-ums-projection \
'{"src-protocol":"amqp091","src-uri":"amqp:///","src-queue":"ums.tenant-projection_error",
"dest-protocol":"amqp091","dest-uri":"amqp:///","dest-queue":"ums.tenant-projection",
rabbitmqctl set_parameter shovel reprocess-tenant-snapshot \
'{"src-protocol":"amqp091","src-uri":"amqp:///","src-queue":"tracker.tenant-snapshot_error",
"dest-protocol":"amqp091","dest-uri":"amqp:///","dest-queue":"tracker.tenant-snapshot",
"src-delete-after":"queue-length","ack-mode":"on-confirm"}'
```

- `src-delete-after: queue-length` drains only the messages present when the shovel starts (does not
loop forever on newly-faulted ones).
- Repeat with `tracker.tenant-projection_error` → `tracker.tenant-projection` for Tracker.
- Delete the shovel when done: `rabbitmqctl clear_parameter shovel reprocess-ums-projection`.
- Delete the shovel when done: `rabbitmqctl clear_parameter shovel reprocess-tenant-snapshot`.

## Verify

1. `_error` queue depth returns to 0 and the `MassTransitPoisonMessages` alert clears.
2. The projection catches up: `masterdata_projection_applied_total` increases; no matching rise in
`masterdata_projection_discarded_total` beyond the expected stale-guard discards.
3. Spot-check the affected tenant(s) in `masterdata.tenant_projection` (UMS + Tracker) match MMS.
2. The projection catches up: `tracker.tenant-snapshot` drains and stays near-empty. There is no
applied/discarded counter to read — the MMS-era `masterdata_projection_*` metrics named here
until 2026-08-22 never had an emitter, and T-059 did not introduce replacements. Queue depth is
the signal.
3. Spot-check the affected tenant(s): the Tracker's `tenants` row carries the
`ums_projection_version` UMS last published, and `code`/`name`/`status` match UMS, which is the
master (ADR-0129).

## Prevent recurrence

- Persistent poison-by-content → the producer emitted an out-of-contract event: reconcile against
`Evolith.Messaging.Contracts` (DS-12) and add a producer contract test.
- Repeated transient faults → check the consumer's DB (`MasterDataDb`) health and the readiness
posture (readiness must never gate on AMQP — §5.4).
- Repeated transient faults → check the Tracker database the consumer writes to (T-059 removed the
separate `MasterDataDb` connection string; the projection lands in the Tracker's own schema) and
the readiness posture (readiness must never gate on AMQP — §5.4).

## References
- Deployment strategy §5.3 (poison handling) · ADR-0108 (MassTransit-owned topology) · GT-463.
Expand Down
Loading
Loading