You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: src/pages/docs/observe/features/alerts.mdx
+22-20Lines changed: 22 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,17 +3,19 @@ title: "Alerts and monitors"
3
3
description: "Define monitors on Observe project metrics (system or evaluation) and get notified by email or Slack when values cross a threshold."
4
4
---
5
5
6
-
## What it is
6
+
## About
7
7
8
-
**Alerts and monitors**are Observe’s way to get notified when a project metric crosses a threshold—so regressions in error rate, latency, cost, or evaluation quality can trigger email or Slack instead of someone watching the dashboard. Monitors cover system metrics (errors, response time, token usage) and evaluation metrics (e.g. toxicity, bias). Each monitor evaluates on a schedule and, when the threshold is breached, creates a critical or warning alert and sends notifications. Alert history is stored so past triggers can be reviewed and marked resolved; monitors can be muted without being deleted.
8
+
**Alerts and monitors**notify you when a metric goes above or below a value you set. Pick a metric (error rate, latency, cost, or an eval score), define a threshold, and choose where to get notified: email, Slack, or both. Monitors check the metric on a schedule. If the threshold is breached, you get an alert. You can review past alerts, mark them resolved, or mute a monitor without deleting it.
9
9
10
-
## Use cases
10
+
---
11
+
12
+
## When to use
11
13
12
-
-**Error and reliability** — Alert when error rate, LLM API failure rate, or error-free session rate crosses a threshold so you catch outages or degradation early.
13
-
-**Latency and performance** — Monitor span or LLM response time and get notified when p95 or average exceeds a limit.
14
-
-**Cost and usage** — Track token usage or daily/monthly tokens spent and alert when spend crosses a budget threshold.
15
-
-**Evaluation quality** — Monitor an eval (e.g. fail rate for a pass/fail eval, or a numeric score) and alert when quality drops below or goes above a value.
16
-
-**Notifications** — Send alerts to up to five email addresses and/or a Slack webhook so the right people are informed without checking the UI.
14
+
-**Catch errors early**: Get notified when error rate or API failure rate spikes after a deployment.
15
+
-**Stay within latency limits**: Alert when response time goes above your target.
16
+
-**Control costs**: Track token usage and get a warning before you hit your budget.
17
+
-**Monitor eval quality**: Know when a pass/fail eval like toxicity starts failing more often.
18
+
-**Stay informed without watching dashboards**: Send alerts to email, Slack, or both.
17
19
18
20
---
19
21
@@ -24,8 +26,8 @@ description: "Define monitors on Observe project metrics (system or evaluation)
24
26
Create a monitor for an Observe project and select the **metric type**:
25
27
26
28
27
-
-**System metrics** — e.g. count of errors, error-free session rates, LLM API failure rates, span response time, LLM response time, token usage, daily/monthly tokens spent.
28
-
-**Evaluation metrics** — Attach a CustomEvalConfig (eval) for that project. For pass/fail or choice evals you can set **threshold_metric_value** to the specific value to monitor (e.g. fail rate or a choice label).
29
+
-**System metrics**: count of errors, error-free session rates, LLM API failure rates, span response time, LLM response time, token usage, daily/monthly tokens spent.
30
+
-**Evaluation metrics**: attach an eval config for that project. For pass/fail or choice evals you can set **threshold_metric_value** to the specific value to monitor (e.g. fail rate or a choice label).
29
31
30
32
The monitor is scoped to one project (Observe projects only).
31
33
</Step>
@@ -34,21 +36,21 @@ description: "Define monitors on Observe project metrics (system or evaluation)
34
36
Set how the alert is triggered:
35
37
36
38
37
-
-**threshold_operator** —**Greater than** or **Less than** (the current metric value is compared to the threshold).
38
-
-**threshold_type** — How the threshold is determined:
39
-
-**Static** — You set fixed **critical_threshold_value** and optionally **warning_threshold_value**. Alert fires when the metric is greater than (or less than) these values.
40
-
-**Percentage change** — Threshold is based on percentage change from a baseline (e.g. historical mean over a time window). You set **critical_threshold_value** and optionally **warning_threshold_value** as percentage values. **auto_threshold_time_window** (default one week, in minutes) defines the window used to compute the baseline.
39
+
-**threshold_operator**:**Greater than** or **Less than** (the current metric value is compared to the threshold).
40
+
-**threshold_type**: how the threshold is determined:
41
+
-**Static**: you set fixed **critical_threshold_value** and optionally **warning_threshold_value**. Alert fires when the metric is greater than (or less than) these values.
42
+
-**Percentage change**: threshold is based on percentage change from a baseline (e.g. historical mean over a time window). You set **critical_threshold_value** and optionally **warning_threshold_value** as percentage values. **auto_threshold_time_window** (default one week, in minutes) defines the window used to compute the baseline.
41
43
42
44
When the condition is met, the system creates an alert log (critical or warning) and triggers notifications.
43
45
</Step>
44
46
45
47
<Steptitle="Set alert frequency">
46
-
**alert_frequency** is how often the monitor is evaluated, in minutes (minimum 5, default 60). The monitor runs on this schedule and checks the metric over the relevant time window; if the threshold is breached, an alert is created and notifications are sent.
48
+
**alert_frequency** is how often the monitor is evaluated, in minutes (minimum 5, default 60). The monitor runs on this schedule and checks the metric over the relevant time window. If the threshold is breached, an alert is created and notifications are sent.
47
49
</Step>
48
50
49
51
<Steptitle="Configure notifications">
50
-
-**Email** — Add up to five addresses in **notification_emails**. They receive an email when an alert is triggered (subject and body include alert name, message, and type).
51
-
-**Slack** — Set**slack_webhook_url** to your Slack incoming webhook. Optional **slack_notes** are included in the message.
52
+
-**Email**: add up to five addresses in **notification_emails**. They receive an email when an alert is triggered (subject and body include alert name, message, and type).
53
+
-**Slack**: set**slack_webhook_url** to your Slack incoming webhook. Optional **slack_notes** are included in the message.
0 commit comments