Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/ollama-instrumentation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"braintrust": minor
---

feat: Add `ollama` instrumentation
2 changes: 2 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ pnpm run test:e2e:record # Re-record provider cassettes and update snap

When adding or modifying e2e tests, run the relevant e2e verification twice before stopping so flakes are caught proactively. After running `pnpm run test:e2e:update` or `pnpm run test:e2e:record`, always run the normal e2e tests afterward to verify there is no snapshot drift or unstable output.

New instrumentation e2e coverage must test both a pinned SDK dependency and a separately named latest dependency alias for every supported version line. Only the latest alias should participate in `test:e2e:bump`; add pinned and latest variants to the CI e2e summary.

Span-tree snapshots are paired: `*.span-tree.json` is the structural contract, and `*.span-tree.txt` is the human-readable ASCII tree generated from the same normalized spans. Both files are asserted and should be updated together through `pnpm run test:e2e:update` or `pnpm run test:e2e:record`; do not hand-edit only one side of the pair.

**From repo root:**
Expand Down
7 changes: 4 additions & 3 deletions e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ Provider credentials are only required when recording or explicitly running live
- `CURSOR_API_KEY`
- `OPENROUTER_API_KEY`
- `MISTRAL_API_KEY`
- `OLLAMA_API_KEY`
- `HUGGINGFACE_API_KEY`
- `COHERE_API_KEY`
- `GROQ_API_KEY`
Expand Down Expand Up @@ -165,15 +166,15 @@ pnpm --filter=@braintrust/js-e2e-tests run test:e2e:record -- <name>
ANTHROPIC_API_KEY=... AWS_BEARER_TOKEN_BEDROCK=... \
OPENAI_API_KEY=... GEMINI_API_KEY=... \
COHERE_API_KEY=... GROQ_API_KEY=... HUGGINGFACE_API_KEY=... \
MISTRAL_API_KEY=... OPENROUTER_API_KEY=... \
MISTRAL_API_KEY=... OLLAMA_API_KEY=... OPENROUTER_API_KEY=... \
CURSOR_API_KEY=... \
pnpm --filter=@braintrust/js-e2e-tests run test:e2e:record
```

After recording, run again **without any provider keys** to confirm the cassette is sufficient:

```bash
unset ANTHROPIC_API_KEY AWS_BEARER_TOKEN_BEDROCK OPENAI_API_KEY GEMINI_API_KEY GOOGLE_API_KEY GOOGLE_GENAI_API_KEY COHERE_API_KEY GROQ_API_KEY HUGGINGFACE_API_KEY MISTRAL_API_KEY OPENROUTER_API_KEY CURSOR_API_KEY
unset ANTHROPIC_API_KEY AWS_BEARER_TOKEN_BEDROCK OPENAI_API_KEY GEMINI_API_KEY GOOGLE_API_KEY GOOGLE_GENAI_API_KEY COHERE_API_KEY GROQ_API_KEY HUGGINGFACE_API_KEY MISTRAL_API_KEY OLLAMA_API_KEY OPENROUTER_API_KEY CURSOR_API_KEY
pnpm --filter=@braintrust/js-e2e-tests run test:e2e
```

Expand All @@ -185,7 +186,7 @@ After any successful record run, stale cassette variants are cleaned only inside

These scenarios have cassette wiring in place and will use cassettes once they're recorded:

`anthropic-bedrock-instrumentation`, `anthropic-instrumentation`, `openai-instrumentation`, `openai-codex-instrumentation`, `ai-sdk-instrumentation`, `ai-sdk-otel-export`, `claude-agent-sdk-instrumentation`, `cohere-instrumentation`, `cursor-sdk-instrumentation`, `github-copilot-instrumentation`, `google-adk-instrumentation`, `google-genai-instrumentation`, `groq-instrumentation`, `huggingface-instrumentation`, `mistral-instrumentation`, `openrouter-agent-instrumentation`, `openrouter-instrumentation`, `wrap-langchain-js-traces`.
`anthropic-bedrock-instrumentation`, `anthropic-instrumentation`, `openai-instrumentation`, `openai-codex-instrumentation`, `ai-sdk-instrumentation`, `ai-sdk-otel-export`, `claude-agent-sdk-instrumentation`, `cohere-instrumentation`, `cursor-sdk-instrumentation`, `github-copilot-instrumentation`, `google-adk-instrumentation`, `google-genai-instrumentation`, `groq-instrumentation`, `huggingface-instrumentation`, `mistral-instrumentation`, `ollama-instrumentation`, `openrouter-agent-instrumentation`, `openrouter-instrumentation`, `wrap-langchain-js-traces`.

### Cassette format

Expand Down
23 changes: 23 additions & 0 deletions e2e/config/pr-comment-scenarios.json
Original file line number Diff line number Diff line change
Expand Up @@ -397,6 +397,29 @@
}
]
},
{
"scenarioDirName": "ollama-instrumentation",
"label": "Ollama Instrumentation",
"metadataScenario": "ollama-instrumentation",
"variants": [
{
"variantKey": "ollama-v0.6-wrapped",
"label": "v0.6 pinned wrapped"
},
{
"variantKey": "ollama-v0.6-auto",
"label": "v0.6 pinned auto-hook"
},
{
"variantKey": "ollama-v0.6-latest-wrapped",
"label": "v0.6 latest wrapped"
},
{
"variantKey": "ollama-v0.6-latest-auto",
"label": "v0.6 latest auto-hook"
}
]
},
{
"scenarioDirName": "cohere-instrumentation",
"label": "Cohere Instrumentation",
Expand Down
3 changes: 3 additions & 0 deletions e2e/helpers/scenario-harness.ts
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,7 @@ function getCassetteServerRoutes(): CassetteServerRoute[] {
upstreamOrigin: "https://router.huggingface.co",
},
{ prefix: "/mistral", upstreamOrigin: "https://api.mistral.ai" },
{ prefix: "/ollama", upstreamOrigin: "https://ollama.com" },
{ prefix: "/openai", upstreamOrigin: "https://api.openai.com" },
{ prefix: "/openrouter", upstreamOrigin: "https://openrouter.ai" },
];
Expand Down Expand Up @@ -349,6 +350,7 @@ function getCassetteEnv(wiring: ActiveCassetteWiring): Record<string, string> {
HUGGINGFACE_ROUTER_BASE_URL: `${serverUrl}/huggingface-router`,
MISTRAL_API_URL: `${serverUrl}/mistral`,
MISTRAL_BASE_URL: `${serverUrl}/mistral`,
OLLAMA_HOST: `${serverUrl}/ollama`,
OPENAI_BASE_URL: `${serverUrl}/openai/v1`,
OPENROUTER_BASE_URL: `${serverUrl}/openrouter/api/v1`,
};
Expand Down Expand Up @@ -398,6 +400,7 @@ const CASSETTE_PROVIDER_KEYS: Array<{
{ envVars: ["GROQ_API_KEY"], placeholder: "gsk_cassette-placeholder" },
{ envVars: ["HUGGINGFACE_API_KEY"], placeholder: "hf_cassette-placeholder" },
{ envVars: ["MISTRAL_API_KEY"], placeholder: "cassette-placeholder" },
{ envVars: ["OLLAMA_API_KEY"], placeholder: "cassette-placeholder" },
{ envVars: ["OPENAI_API_KEY"], placeholder: "sk-cassette-placeholder" },
{
envVars: ["OPENROUTER_API_KEY"],
Expand Down
Loading
Loading