Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 3 additions & 5 deletions workshops/build_workshop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ feature branch, open a PR to protected `dev-build-workshop-v1`, verify
## Architecture

The app edge runs on the participant's laptop. ClickHouse, Postgres, ClickPipes, and
ClickStack/HyperDX live in ClickHouse Cloud; OpenAI, Langfuse, and optional LibreChat are
separate hosted services. No database or product UI is deployed locally.
ClickStack/HyperDX live in ClickHouse Cloud; OpenAI and Langfuse are separate hosted
services. No database or product UI is deployed locally.

```mermaid
flowchart LR
Expand All @@ -40,7 +40,6 @@ flowchart LR
subgraph THIRD["Third-party"]
OAI["OpenAI API<br/>chat completions"]
LF["Langfuse Cloud<br/>chat traces"]
LC["Hosted LibreChat<br/>optional SRE chat"]
TLC["NYC TLC public dataset<br/>download source only"]
end

Expand All @@ -58,7 +57,6 @@ flowchart LR
TLC -.->|url seed, module 01| CH
HDX -->|reads otel db| CH
AG -->|RBAC-governed SQL| CH
LC -.->|remote MCP| MCP
```

The published diagrams (the ClickHouse Cloud platform stack, the workshop architecture, the
Expand Down Expand Up @@ -100,7 +98,7 @@ sequenceDiagram
G->>P: INSERT trips (throttled, TLS)
Note over G,P: first run also creates the table<br/>and publication pub_taxi
P->>CP: WAL changes via publication + slot
CP->>CH: rows land in realtime_trips (_peerdb_* cols), about 60s
CP->>CH: rows land in default.realtime_trips, about 60s
CH->>CH: materialized view fans rows into nyc_tlc_data.taxi_trips
UI->>CH: parameterized SQL / guarded NL-to-SQL / RBAC-governed BI
CH-->>UI: live Ops dashboard + 3M-row Historical seed
Expand Down
4 changes: 2 additions & 2 deletions workshops/build_workshop/app/OBSERVABILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,12 +107,12 @@ cp .env.workshop.example .env.workshop

# 2) Bring up the stack with the overlay:
docker compose --env-file .env.workshop \
-f docker-compose.workshop.yml -f docker-compose.otel.yml up -d
-f docker-compose.workshop.yml -f docker-compose.otel.yml up -d --build

# Optional: also scrape raw container stdout (see VERIFY-LIVE):
docker compose --env-file .env.workshop \
-f docker-compose.workshop.yml -f docker-compose.otel.yml \
--profile container-logs up -d
--profile container-logs up -d --build
```

## Environment variables
Expand Down
2 changes: 1 addition & 1 deletion workshops/build_workshop/app/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ docker compose --env-file .env.workshop -f docker-compose.workshop.yml up -d

# with the ClickStack observability overlay (module 05 onward):
docker compose --env-file .env.workshop \
-f docker-compose.workshop.yml -f docker-compose.otel.yml up -d
-f docker-compose.workshop.yml -f docker-compose.otel.yml up -d --build
```

Frontend: http://localhost:8080 - Backend API docs (FastAPI Swagger):
Expand Down
9 changes: 4 additions & 5 deletions workshops/build_workshop/app/WORKSHOP_CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,19 +19,18 @@ The workshop intentionally separates application code from managed services.
- remote ClickHouse and ClickStack MCP endpoints
- ClickHouse Agents
- Langfuse Cloud and OpenAI
- the optional instructor-provided HTTPS LibreChat instance

No PostgreSQL, ClickHouse, MongoDB, LibreChat, HyperDX, Langfuse, or MCP server is
started on a learner machine.
No PostgreSQL, ClickHouse, MongoDB, HyperDX, Langfuse, or MCP server is started on a
learner machine.

## Runtime sequence

1. Module 00 creates ClickHouse Cloud and starts only the local frontend/backend.
2. Module 01 creates and seeds the Cloud schema.
3. Module 03 creates managed Postgres, validates it with
`./preflight.sh --require-postgres`, then explicitly enables the `cdc` trip writer.
4. Module 05 enables Managed ClickStack and starts the stateless collector overlay.
5. Module 06b uses hosted LibreChat; Module 08 sends chat traces to Langfuse Cloud.
4. Module 05 starts the stateless collector overlay, then enables Managed ClickStack.
5. Module 08 sends chat traces to Langfuse Cloud.

`.env.workshop.example` leaves managed Postgres credentials blank until Module 03 and
requires TLS. `preflight.sh` rejects loopback/local database hosts. CI policy checks in
Expand Down
2 changes: 1 addition & 1 deletion workshops/build_workshop/app/docker-compose.otel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# "ClickStack observability" section of .env.workshop.example):
#
# docker compose --env-file .env.workshop \
# -f docker-compose.workshop.yml -f docker-compose.otel.yml up -d
# -f docker-compose.workshop.yml -f docker-compose.otel.yml up -d --build
#
# Add raw container-log scraping (optional, see VERIFY-LIVE notes in OBSERVABILITY.md):
# ... -f docker-compose.otel.yml --profile container-logs up -d
Expand Down
239 changes: 84 additions & 155 deletions workshops/build_workshop/docs/diagrams/gen_diagrams.py

Large diffs are not rendered by default.

157 changes: 52 additions & 105 deletions workshops/build_workshop/docs/diagrams/workshop-architecture.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
161 changes: 58 additions & 103 deletions workshops/build_workshop/docs/diagrams/workshop-data-flow.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
37 changes: 15 additions & 22 deletions workshops/build_workshop/docs/diagrams/workshop-module-flow.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
23 changes: 11 additions & 12 deletions workshops/build_workshop/playbook/content/docs/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -89,33 +89,33 @@ stateful data or product service is cloud-hosted from the start.
ClickHouse Cloud is a single platform that spans the whole stack — from ingesting data at
the bottom, to storing and analyzing it in ClickHouse, to observing it and layering AI on
top. This workshop is a guided tour through exactly these pieces: you ingest with
ClickPipes, analyze in ClickHouse, and observe with Managed ClickStack/HyperDX. Langfuse,
OpenAI, and optional LibreChat are separate hosted services.
ClickPipes, analyze in ClickHouse, and observe with Managed ClickStack/HyperDX. Langfuse
and OpenAI are separate hosted services.

![The ClickHouse Cloud platform: layered stack from sources and ClickPipes ingestion, through ClickHouse and Postgres, up to HyperDX, Langfuse and agentic AI](/clickhouse-platform.svg)

### Architecture (target end state)

Here is how those pieces come together: the app edge on your laptop; ClickHouse-managed
state in your trial; and separate hosted OpenAI, Langfuse, and optional LibreChat services.
First, group the pieces by where they run: the stateless app and tools on your laptop,
stateful services in ClickHouse Cloud, and the separate hosted AI services.

![Workshop architecture: participant laptop, your ClickHouse Cloud trial (service, ClickPipes, managed Postgres), and third-party services](/workshop-architecture.svg)
![Workshop components grouped into participant laptop, ClickHouse Cloud, and external hosted services](/workshop-architecture.svg)

### How data flows

The same picture, followed as data: seeded once from object storage, streamed continuously
via CDC, then read by the dashboards, chat, and agents, and observed through ClickStack.
Now follow one live trip from the local load generator through managed Postgres and
ClickPipes, into `default.realtime_trips`, and finally into the application dashboards.

![Data flow: four lanes — one-time seed, live CDC, read paths, and observability — all through your ClickHouse service](/workshop-data-flow.svg)
![A live trip flows from the load generator to managed Postgres, through ClickPipes into default.realtime_trips, then through a materialized view to the app](/workshop-data-flow.svg)

The diagrams above are generated from `workshops/build_workshop/docs/diagrams/gen_diagrams.py`
(edit the script and re-run it to regenerate the SVGs) — see the module flow below.

![Module flow: ten core modules plus optional hosted LibreChat 06b](/workshop-module-flow.svg)
![Module flow: ten core workshop modules](/workshop-module-flow.svg)

## Modules

Ten core modules plus optional 06b, in order. The app is complete on `build-workshop-v1`, so every module except
Ten core modules, in order. The app is complete on `build-workshop-v1`, so every module except
07 needs no per-module checkout — you configure and connect services rather than change
app code. The only branch switches are the fault branches in module 07.

Expand All @@ -128,8 +128,7 @@ app code. The only branch switches are the fault branches in module 07.
| 04 | 10 min | [ClickHouse Agents](/docs/learner/04-clickhouse-agents) | [notes](/docs/instructor/04-clickhouse-agents) | `build-workshop-v1` | Conversational BI: create an agent over your taxi data and explore it in natural language |
| 05 | 15 min | [ClickStack](/docs/learner/05-clickstack) | [notes](/docs/instructor/05-clickstack) | `build-workshop-v1` | Enable ClickStack and send app traces and logs to HyperDX |
| 06 | 15 min | [AI SRE](/docs/learner/06-ai-sre) | [notes](/docs/instructor/06-ai-sre) | `build-workshop-v1` | Use the ClickStack MCP connection to build an SRE dashboard and alert |
| 06b | Optional, 15 min | [Hosted LibreChat AI SRE](/docs/learner/06b-ai-sre-librechat) | [notes](/docs/instructor/06b-ai-sre-librechat) | `build-workshop-v1` | Use a hosted chat agent with remote ClickStack and GitHub MCP connections |
| 07 | 20 min | [Test, fail, and fix](/docs/learner/07-break-and-fix) | [notes](/docs/instructor/07-break-and-fix) | `fault/*` | Continue from AI SRE or optional 06b, inject a fault, diagnose it, fix it, and prove recovery |
| 07 | 20 min | [Test, fail, and fix](/docs/learner/07-break-and-fix) | [notes](/docs/instructor/07-break-and-fix) | `fault/*` | Continue from AI SRE, inject a fault, diagnose it, fix it, and prove recovery |
| 08 | 15 min | [Chat and Langfuse](/docs/learner/08-chat-langfuse) | [notes](/docs/instructor/08-chat-langfuse) | `build-workshop-v1` | Use the in-app AI chat and follow its traces, generations, and costs in Langfuse |
| 09 | 10 min | [Wrap-up](/docs/learner/09-wrap-up) | [notes](/docs/instructor/09-wrap-up) | `build-workshop-v1` | Review what you built, take it home, and extend it to your own data |

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ collector as early as this module allows and keep it running.
- Managed ClickStack stores telemetry in ClickHouse Cloud; HyperDX is the hosted UI.
The local collector only forwards data. Show both an end-to-end request trace and a
fresh `DEBUG ... ClickHouse query ok` record in the Log source.
- Have learners start the collector in Step 1 before they launch Managed ClickStack in
Step 2. This gives telemetry time to arrive and makes the first hosted-UI view useful.
- Show one end-to-end request trace on the projector.

## Common failures
Expand Down Expand Up @@ -42,6 +44,6 @@ collector as early as this module allows and keep it running.
## Reset steps

- Restart with both compose files:
`docker compose --env-file .env.workshop -f docker-compose.workshop.yml -f docker-compose.otel.yml up -d`.
`docker compose --env-file .env.workshop -f docker-compose.workshop.yml -f docker-compose.otel.yml up -d --build`.
- In the Cloud console, open the service, choose **ClickStack**, and relaunch the hosted
UI if the SSO session expired.
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: 06a AI SRE
title: 06 AI SRE
description: Instructor notes for module 06 — timing, talk track, common failures, and reset steps.
---

Expand Down
Original file line number Diff line number Diff line change
@@ -1,17 +1,24 @@
---
title: 06b AI SRE with hosted LibreChat
description: Instructor prework and facilitation notes for the optional cloud-hosted LibreChat module.
title: "Archived: 06b AI SRE with hosted LibreChat"
description: Archived instructor reference for the former optional hosted LibreChat module.
---

<Callout type="warn" title="Archived module">
Module 06b is no longer part of the run of show. Do not provision LibreChat, learner
accounts, or its MCP connections for the current workshop. Continue directly from
Module 06 to [Module 07](/docs/instructor/07-break-and-fix). The notes below remain
only as a historical reference.
</Callout>

Facilitator companion to
[06b AI SRE with hosted LibreChat](/docs/learner/06b-ai-sre-librechat).

## Timing
## Historical timing

Optional, about 15 minutes. Run it immediately before Module 07. Skip it if the room is
tight; Module 06a already covers the remote ClickStack MCP workflow.
This module previously took about 15 minutes immediately before Module 07. Module 06
already covers the remote ClickStack MCP workflow.

## Required cloud prework
## Historical cloud prework

Learners must not deploy LibreChat or MongoDB locally.

Expand Down Expand Up @@ -41,14 +48,14 @@ LibreChat documents remote MCP OAuth, per-user credentials, and agent sharing in

## Talk track

- Module 06a is an agent in the editor; 06b is the same evidence chain in a hosted chat UI.
- Module 06 is an agent in the editor; 06b was the same evidence chain in a hosted chat UI.
- LibreChat is a client surface. ClickStack/HyperDX and the telemetry remain managed in
ClickHouse Cloud; GitHub remains the source-of-truth service.
- Each learner owns their ClickHouse OAuth session and GitHub PAT.

## Common failures

- **Hosted URL is unavailable**: skip 06b and continue with Module 06a. Do not improvise a
- **Hosted URL is unavailable**: skip this archived module and continue with Module 06. Do not improvise a
local Docker deployment during the workshop.
- **OAuth returns to the wrong host**: correct LibreChat's public `DOMAIN_SERVER` and
`DOMAIN_CLIENT` to the HTTPS deployment URL, then reconnect.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@ so the wrap-up is not squeezed.
## Talk track

- This is the payoff: use everything built so far to run a real incident.
- If the room completed 06b, fault 01 is already diagnosed — this module becomes
confirm-fix-verify, so keep the momentum and move quickly to the fix.
- Describe the symptom, not the cause, and let the agents converge from telemetry.
- Consider showing several attendees' diagnoses side by side on the projector.
- The lab runs fault 01. If time permits an extra round, add fault 02 (and fault 03 only
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,6 @@ About 10 minutes. Reserve time for two or three volunteer demos on the projector

## Reset steps

- Reset any shared state used for demos (managed Postgres fallback pool, hosted LibreChat,
projector channel).
- Reset any shared state used for demos (managed Postgres fallback pool and projector
channel).
- TODO: confirm the shared-state teardown checklist.
Loading
Loading