|
1 | 1 | --- |
2 | | -title: "Evaluations" |
3 | | -description: "Create and run eval tasks on Observe project data: filter spans, choose historic or continuous runs, set sampling and limits, and attach preset or custom evaluations." |
| 2 | +title: "Run Evals on Traces" |
| 3 | +description: "Run automated quality checks on your traced spans in Observe: filter spans, choose historic or continuous runs, set sampling and limits, and attach preset or custom evaluations." |
4 | 4 | --- |
5 | 5 |
|
6 | 6 | ## About |
7 | 7 |
|
8 | | -**Evaluations** in Observe are automated quality checks run on your traced spans, e.g. hallucination, bias, context adherence, toxicity. The feature scores LLM (or other) outputs so you can see pass/fail and numeric results per span in the dashboard, track quality over time, and trigger alerts when scores cross a threshold. Evals can run on existing data (historic) or on new spans as they arrive (continuous), and results are stored on the span and available for filtering, export, and monitors. |
| 8 | +Evals run automated quality checks on your production traces, scoring every LLM response for hallucination, tone, bias, toxicity, and more. You configure which checks to run, filter which spans they apply to, and choose whether to evaluate historical data or new spans as they arrive. Results appear per span in the Observe dashboard and can trigger alerts when quality drops. |
| 9 | + |
9 | 10 | {/* ARCADE EMBED START */} |
10 | 11 | <script>{` function onArcadeIframeMessage(e) { if (e.origin !== 'https://demo.arcade.software' || !e.isTrusted) return; const arcadeIframe = document.querySelector(\`iframe[src*=\${e.data.id}]\`); if (!arcadeIframe || !arcadeIframe.contentWindow) return; if (e.data.event === 'arcade-init') { arcadeIframe.contentWindow.postMessage({event: 'register-popout-handler'}, '*'); } if (e.data.event === 'arcade-popout-open') { arcadeIframe.style['position'] = 'fixed'; arcadeIframe.style['z-index'] = '9999999'; } if (e.data.event === 'arcade-popout-close') { arcadeIframe.style['position'] = 'absolute'; arcadeIframe.style['z-index'] = 'auto'; } } window.addEventListener('message', onArcadeIframeMessage); `}</script> |
11 | 12 | <div style={{position: 'relative', paddingBottom: 'calc(57.1875% + 100px)', height: 0, minWidth: '600px', width: '100%'}}><iframe src="https://demo.arcade.software/Yu4mABONU00uVaeC2NKP?embed&embed_mobile=inline&embed_desktop=inline&show_copy_link=true" title="Datasets Evaluations" frameBorder="0" loading="lazy" allowFullScreen allow="clipboard-write" style={{position: 'absolute', top: 0, left: 0, width: '100%', height: '100%', colorScheme: 'light'}} ></iframe></div> |
12 | 13 | {/* ARCADE EMBED END */} |
| 14 | + |
| 15 | +--- |
| 16 | + |
13 | 17 | ## When to use |
14 | 18 |
|
15 | | -- **Historic batch**: Run evals on a time range of existing spans to score quality (e.g. hallucination, bias, context adherence) after the fact. |
16 | | -- **Continuous monitoring**: Run evals automatically on new spans as they arrive so you catch regressions in production. |
17 | | -- **Cost and volume control**: Use sampling rate (e.g. 10%) and max spans per run so you don’t evaluate every span and can control cost. |
18 | | -- **Targeted evaluation**: Filter by observation type (e.g. LLM only), session, or span attributes so only relevant spans are evaluated. |
19 | | -- **Multiple evals per task**: Attach several eval configs to one task so each span gets multiple scores in a single run. |
| 19 | +- **Scoring production output quality**: Run historic evals after a release to check for hallucinations, bias, or unsafe content across real traffic. |
| 20 | +- **Catching regressions in production**: Set up a continuous eval task so new spans are scored automatically and you see quality drops before users report them. |
| 21 | +- **Spot-checking a specific time window**: Filter by date range or session to evaluate only the spans from an incident or a specific user flow. |
| 22 | +- **Controlling eval cost**: Use sampling rate and span limits to evaluate a representative subset instead of every span. |
| 23 | +- **Running multiple quality checks at once**: Attach several evals to one task so each span gets scored for tone, safety, and accuracy in a single run. |
20 | 24 |
|
21 | 25 | --- |
22 | 26 |
|
23 | 27 | ## How to |
24 | 28 |
|
25 | 29 | <Steps> |
26 | 30 | <Step title="Set filters"> |
27 | | - Define filters so the task runs only on the spans you care about. Supported filter keys: |
| 31 | + Define filters so the task runs only on the spans you care about. |
28 | 32 |
|
29 | 33 |  |
30 | 34 |
|
31 | | - - **observation_type**: Node/span type (e.g. `llm`, `chain`, `agent`). Pass a string or list of types. |
32 | | - - **date_range**: Time range: a two-element list `[start_date, end_date]` (applied to `created_at`). |
33 | | - - **created_at**: Minimum creation time (spans created at or after this value). |
34 | | - - **project_id**: Restrict to a specific Observe project. |
35 | | - - **session_id**: Restrict to traces in a given session. |
36 | | - - **span_attributes_filters**: List of span-attribute conditions (same structure as in the Observe UI filters). |
| 35 | + | Filter | Description | |
| 36 | + |--------|-------------| |
| 37 | + | `observation_type` | Node/span type (e.g. `llm`, `chain`, `agent`). | |
| 38 | + | `date_range` | Time range: `[start_date, end_date]` applied to `created_at`. | |
| 39 | + | `created_at` | Minimum creation time (spans at or after this value). | |
| 40 | + | `project_id` | Restrict to a specific Observe project. | |
| 41 | + | `session_id` | Restrict to traces in a given session. | |
| 42 | + | `span_attributes_filters` | List of span-attribute conditions. | |
37 | 43 |
|
38 | | - Filters are stored in the task’s `filters` JSON and applied when the task runs. |
| 44 | + Filters are stored in the task's `filters` field and applied when the task runs. |
39 | 45 | </Step> |
40 | 46 |
|
41 | 47 | <Step title="Choose run type"> |
42 | | - Set **run_type**: |
| 48 | + Set the **run type**: |
43 | 49 |
|
44 | 50 |  |
45 | 51 |
|
46 | | - - **Historical**: Run on existing spans that match the filters (optionally within a time range). The task processes up to the sampling cap and span limit, then can complete. |
47 | | - - **Continuous**: Run on new spans as they arrive. Each run only considers spans created after the last run; the task stays active for ongoing evaluation. |
| 52 | + - **Historical**: Run on existing spans matching the filters, up to the sampling cap and span limit. The task completes after processing. |
| 53 | + - **Continuous**: Run on new spans as they arrive. Each run only processes spans created after the last run; the task stays active for ongoing evaluation. |
48 | 54 | </Step> |
49 | 55 |
|
50 | 56 | <Step title="Set sampling rate and span limit"> |
51 | 57 |  |
52 | 58 |
|
53 | | - - **sampling_rate**: Percentage of matching spans to evaluate (0–100). Example: `50` means 50% of filtered spans are sampled per run. Helps control cost and volume. |
54 | | - - **spans_limit**: Maximum number of spans to process per run (default is 1000). For historical runs, the task stops when either the sampled count or this limit is reached. |
| 59 | + - **sampling_rate**: Percentage of matching spans to evaluate (0-100). For example, `50` evaluates 50% of filtered spans per run. |
| 60 | + - **spans_limit**: Maximum number of spans to process per run (default 1000). The task stops when either the sampled count or this limit is reached. |
55 | 61 | </Step> |
56 | 62 |
|
57 | 63 | <Step title="Select evals to run"> |
58 | | - |
59 | | - Attach one or more **CustomEvalConfig** IDs to the task (the evals you’ve already created for the project). The task runs each selected eval on every span it processes. For evals that need an input (e.g. Bias Detection), configure the **input key** to a span attribute path (e.g. `llm.output_messages.0.message.content`) so the eval reads the right field from each span. See [built-in evals](/docs/evaluation/builtin) for supported evaluations and their inputs. |
| 64 | + Attach one or more eval configs to the task. The task runs each selected eval on every span it processes. For evals that need an input (e.g. Bias Detection), set the **input key** to a span attribute path (e.g. `gen_ai.output.messages.0.message.content`) so the eval reads the right field from each span. See [built-in evals](/docs/evaluation/builtin) for supported evaluations and their required inputs. |
60 | 65 | </Step> |
61 | 66 |
|
62 | 67 | <Step title="Run the task"> |
63 | 68 |  |
64 | | - Create or update the eval task via the API or UI, then run it. You can test the configuration (filters and evals) before saving. Task status values: `pending`, `running`, `completed`, `failed`, `paused`, `deleted`. Results appear on the spans in the Observe dashboard and can be used for alerts. |
| 69 | + |
| 70 | + Create or update the eval task via the API or UI, then run it. You can test the configuration before saving. Task status values: `pending`, `running`, `completed`, `failed`, `paused`, `deleted`. Results appear on the spans in the Observe dashboard and can be used for alerts. |
65 | 71 | </Step> |
66 | 72 | </Steps> |
67 | 73 |
|
68 | 74 | <Note> |
69 | | - Eval tasks are processed asynchronously (e.g. by a cron). Status and results update as runs complete. For continuous tasks, new spans are picked up on subsequent runs. |
| 75 | + Eval tasks are processed asynchronously. Status and results update as runs complete. For continuous tasks, new spans are picked up on subsequent runs. |
70 | 76 | </Note> |
71 | 77 |
|
72 | | -## Key concepts |
73 | | - |
74 | | -- **Span attributes**: Spans store data in key-value form (e.g. `llm.output_messages.0.message.content`). When an eval needs an input, you point it to one of these attribute paths. See [spans](/docs/tracing/concepts/spans) and [span attributes](/docs/tracing/concepts/spans#span-attributes) for the schema. |
75 | | -- **Bias Detection example**: Set the eval’s input key to a span attribute that holds the text to check (e.g. `llm.output_messages.0.message.content`). The eval returns Passed (neutral) or Failed (bias detected). |
76 | | - |
77 | 78 | --- |
78 | 79 |
|
79 | 80 | ## Next Steps |
|
0 commit comments