Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
5a0a5cf
feat: add constructive-infra pgpm package (standalone, no RLS/grants)
pyramation Jun 8, 2026
68a782f
feat: add proper pgpm module scaffolding (package.json, Makefile, .np…
pyramation Jun 8, 2026
45c8d90
pkgs
pyramation Jun 8, 2026
63d3dbe
fix: update constructive-infra with dependency resolution and metasch…
pyramation Jun 8, 2026
39d0522
feat: add constructive-infra-services package for MetaSchema registra…
pyramation Jun 8, 2026
7784149
feat: add pgpm integration test workflow + fix type revert SQL
pyramation Jun 8, 2026
0d22dac
feat: add compute-worker, compute-service, 3-tier dev setup, and skills
pyramation Jun 8, 2026
4fb2c9d
fix: add pgpm install step before deploy (resolves missing extension …
pyramation Jun 8, 2026
80ff516
feat: add make status and make verify-platform commands
pyramation Jun 8, 2026
b3eded2
updates
pyramation Jun 8, 2026
b750aeb
chore: remove pgpm install steps (extensions now committed to repo)
pyramation Jun 8, 2026
d58708d
feat: move function seeds into pgpm fixture (no manual seed step)
pyramation Jun 8, 2026
e9f376d
refactor: extract seed fixture into constructive-infra-seed package
pyramation Jun 8, 2026
5e8e3e9
fix: remove generated function deps from compute-service
pyramation Jun 8, 2026
dc97639
feat: add secret definitions seed + env checker + required_secrets/co…
pyramation Jun 8, 2026
9119928
feat: add make up/down lifecycle + make up:email-job/down:email-job
pyramation Jun 8, 2026
d469381
fix: filter non-Node functions, add port summary, respect env var ove…
pyramation Jun 8, 2026
56a6a34
feat: add www/ platform UI + default namespace seed + make up:www
pyramation Jun 8, 2026
5f6d473
fix: add missing tsconfig.app.json and tsconfig.node.json for www/
pyramation Jun 8, 2026
e902cb1
feat: add commander + kubernetesjs with K8s tab for browser-based clu…
pyramation Jun 8, 2026
7664730
fix: parse ANSI colors to HTML + remove output height cap in Commands…
pyramation Jun 8, 2026
3c03a12
feat: add secret/config coverage check to email-job-up startup
pyramation Jun 8, 2026
f4b0a28
feat: add .env loader + DB-driven secret/config coverage to dev-compute
pyramation Jun 8, 2026
b984d0b
fix: add default partitions for invocations + namespace_events
pyramation Jun 8, 2026
72f94dc
feat: add secrets editor UI with .env read/write
pyramation Jun 8, 2026
9743294
feat: add inline Trigger button on Functions tab
pyramation Jun 8, 2026
19b6320
fix: make shell scripts bash 3 compatible (macOS)
pyramation Jun 8, 2026
ac2dd57
fix: use default database_id when job doesn't provide one
pyramation Jun 8, 2026
b73f70f
feat: add FBP integration, Flow Graph UI, and inline trigger enhancem…
pyramation Jun 8, 2026
ef13963
feat: bidirectional DB <-> .env secrets sync + per-function env injec…
pyramation Jun 8, 2026
42c6889
fix: update default payloads to match actual handler signatures
pyramation Jun 8, 2026
1cf0713
Merge pull request #55 from constructive-io/feat/secrets-sync
pyramation Jun 8, 2026
d8a6cc5
Merge pull request #53 from constructive-io/feat/fbp-flow-graph
pyramation Jun 8, 2026
ab91579
feat: add standalone pgpm modules (store, objects, fbp, storage)
pyramation Jun 8, 2026
ed3638e
feat: deploy all 5 base modules + add MinIO for storage
pyramation Jun 8, 2026
8ebee9f
fix: show actual pgpm deploy errors instead of swallowing them
pyramation Jun 8, 2026
8f85710
fix: add missing uuid-ossp/pgcrypto to module .control files
pyramation Jun 8, 2026
7c3475a
feat: replace React Flow with @fbp/graph-editor
pyramation Jun 8, 2026
764d623
fix: reorder pgpm.plan so procedures deploy after table columns
pyramation Jun 8, 2026
dc558b5
fix: add pgpm-database-jobs to constructive-infra requires
pyramation Jun 8, 2026
ece9b18
fix: create default partitions for range-partitioned tables
pyramation Jun 8, 2026
e7ad628
feat: restyle graph editor nodes to match React Flow visual
pyramation Jun 8, 2026
317cbd9
feat: auto-registration from handler.json + example templates
pyramation Jun 8, 2026
6986fea
feat: unified node palette — merge inner/outer sidebars into single s…
pyramation Jun 8, 2026
65b7295
fix: Jobs panel dynamically loads registered functions
pyramation Jun 8, 2026
fc860f5
feat: standardize node sizes and improve icon/title alignment
pyramation Jun 8, 2026
0ba29c2
feat: schematize functions with payloadSchema → FBP NodeDefinition
pyramation Jun 8, 2026
e83b82b
unify node sizes: rich function nodes now use same 240px width as com…
pyramation Jun 8, 2026
9b5661d
feat: auto-seed secrets/configs from handler.json + load .env values …
pyramation Jun 8, 2026
0ccf757
uniform sidebar items + fix icon alignment on canvas nodes
pyramation Jun 8, 2026
d6ea86e
refactor: use real upstream tables for secrets/configs, remove vibe-c…
pyramation Jun 8, 2026
c0b5054
Merge pull request #56 from constructive-io/feat/fbp-graph-editor
pyramation Jun 8, 2026
1c88750
Merge pull request #57 from constructive-io/feat/fbp-flow-graph
pyramation Jun 8, 2026
66bb873
Merge pull request #52 from constructive-io/feat/compute-worker
pyramation Jun 8, 2026
23809dc
refactor: use platform_secrets_set/get for production PGP encryption
pyramation Jun 8, 2026
6849b48
Merge pull request #58 from constructive-io/feat/secrets-encryption
pyramation Jun 8, 2026
ef52e1d
fix: set all JWT claims before calling upstream secret functions
pyramation Jun 8, 2026
06c93f7
Merge pull request #60 from constructive-io/feat/jwt-claims-standalone
pyramation Jun 8, 2026
a573041
fix: standalone compat for platform_secrets ON CONFLICT
pyramation Jun 8, 2026
6353e5a
refactor: move standalone compat patch to its own pgpm package
pyramation Jun 8, 2026
0d5f095
refactor: combine standalone compat patch into constructive-infra-seed
pyramation Jun 8, 2026
c7df5da
Merge pull request #61 from constructive-io/feat/standalone-secrets-c…
pyramation Jun 8, 2026
ce4b5ba
fix: add argument signatures to all function revert SQL
pyramation Jun 8, 2026
07638e2
Merge pull request #64 from constructive-io/fix/slicer-revert-signatures
pyramation Jun 8, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
The diff you're trying to view is too large. We only load the first 3000 changed files.
157 changes: 157 additions & 0 deletions .agents/skills/compute-worker/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
---
name: compute-worker
description: Platform-aware compute worker and service for constructive-functions. Discovers functions from the database, tracks invocations, dispatches via HTTP. Use when working on the compute-worker, compute-service, function discovery, or invocation tracking.
---

# Compute Worker & Service

The compute-worker is a platform-aware replacement for the legacy knative-job-worker. Instead of discovering functions from a static manifest or env vars, it queries `constructive_infra_public.platform_function_definitions` and tracks every invocation in `platform_function_invocations`.

## Architecture

```
┌─────────────────────────────────────────────────────────┐
│ compute-service (orchestrator) │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ HTTP callback │ │ ComputeWorker│ │ Scheduler │ │
│ │ server (:8080)│ │ (polls jobs) │ │ (cron jobs) │ │
│ └──────────────┘ └──────┬───────┘ └──────────────┘ │
│ │ │
│ ┌──────────────┼──────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ FunctionDiscovery InvocationTracker compute_request │
│ (TTL-cached DB (INSERT/UPDATE (HTTP POST to │
│ lookups) invocations) function URL) │
└─────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
platform_function platform_function Function HTTP
_definitions _invocations endpoint
(read) (write) (send-email:8081)
```

## Packages

### job/compute-worker (`@constructive-io/compute-worker`)

Core worker class and supporting modules:

| File | Purpose |
|------|---------|
| `src/index.ts` | `ComputeWorker` class — lifecycle, job polling, dispatch |
| `src/discovery.ts` | `FunctionDiscovery` — lazy TTL-cached DB lookups |
| `src/invocation.ts` | `InvocationTracker` — create/complete/fail invocation records |
| `src/req.ts` | `compute_request()` — HTTP POST dispatch with X-* headers |
| `src/cache.ts` | `TtlCache<T>` — generic TTL cache |
| `src/types.ts` | TypeScript interfaces |

### job/compute-service (`@constructive-io/compute-service`)

Orchestrator that starts the callback server, ComputeWorker, and Scheduler:

| File | Purpose |
|------|---------|
| `src/index.ts` | `ComputeService` class + `bootCompute()` entry point |
| `src/run.ts` | CLI entry point (`node dist/run.js`) |
| `src/registry.ts` | Function registry loader (for optional in-process function servers) |
| `src/types.ts` | TypeScript interfaces |

## Key differences from legacy worker

| Feature | knative-job-worker | compute-worker |
|---------|-------------------|----------------|
| Function discovery | Static manifest / `JOBS_SUPPORTED` env | DB query (TTL-cached) |
| Invocation tracking | None | `platform_function_invocations` table |
| Task filtering | `JOBS_SUPPORTED` allowlist | Accepts any registered task |
| URL resolution | Gateway URL + dev map | `service_url` from DB → dev map → gateway fallback |
| Infra requirement | Only needs `app_jobs` schema | Needs `app_jobs` + `constructive_infra_public` |

## Function discovery flow

```
Job arrives (task_identifier = "send-email")
FunctionDiscovery.resolve("send-email")
├─ Cache hit? → return cached definition
└─ Cache miss? → SQL query:
SELECT * FROM constructive_infra_public.platform_function_definitions
WHERE task_identifier = 'send-email'
└─ Cache result (TTL default: 60s)
PlatformFunctionDefinition {
id, name, task_identifier, service_url,
is_invocable, max_attempts, priority, ...
}
```

## Invocation lifecycle

```
1. create() → INSERT INTO platform_function_invocations
(status='running', started_at=now())
→ returns invocation_id

2a. complete() → UPDATE SET status='completed',
completed_at=now(), duration_ms=X

2b. fail() → UPDATE SET status='failed',
completed_at=now(), duration_ms=X, error='...'
```

## Running locally

```bash
# Tier 1: pgpm-local
pgpm docker start --image docker.io/constructiveio/postgres-plus:18
eval "$(pgpm env)"
make setup-platform # deploy infra + seed functions
make dev-compute # start compute-service + functions

# Tier 2: compose-local
make dev # docker compose up
make dev-compute # start compute-service + functions
```

## Testing a job manually

```bash
# Insert a test job
eval "$(pgpm env)"
psql -d constructive-functions-db1 -c "
INSERT INTO app_jobs.jobs (task_identifier, payload)
VALUES ('send-email', '{\"to\":\"test@example.com\",\"subject\":\"test\",\"html\":\"<p>hi</p>\"}'::json)
"

# Check invocation records
psql -d constructive-functions-db1 -c "
SELECT id, task_identifier, status, duration_ms, error
FROM constructive_infra_public.platform_function_invocations
ORDER BY started_at DESC LIMIT 5
"
```

## Environment variables

| Variable | Default | Description |
|----------|---------|-------------|
| `COMPUTE_JOBS_ENABLED` | `true` | Enable/disable the compute worker |
| `COMPUTE_CALLBACK_URL` | — | URL functions POST to on completion |
| `COMPUTE_GATEWAY_URL` | — | Fallback gateway URL for functions without `service_url` |
| `JOBS_SCHEMA` | `app_jobs` | PostgreSQL schema for the jobs table |
| `INTERNAL_JOBS_CALLBACK_PORT` | `8080` | Port for the HTTP callback server |
| `INTERNAL_GATEWAY_DEVELOPMENT_MAP` | — | JSON map of task→URL for local dev |

## Database requirements

The compute-service checks two things at boot:
1. `app_jobs.jobs` table exists (deployed by `@pgpm/database-jobs`, a dependency of `constructive-infra`)
2. `constructive_infra_public.platform_function_definitions` table exists (deployed by `constructive-infra`)

Both are deployed together via `make setup-platform` or the `platform-setup` Docker Compose service.
171 changes: 171 additions & 0 deletions .agents/skills/dev-tiers/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
---
name: dev-tiers
description: Three-tier local development environment for constructive-functions. Covers pgpm-local (Tier 1), compose-local (Tier 2), and k8s-local (Tier 3). Use when setting up the dev environment, starting services, or debugging infrastructure.
---

# Development Tiers

constructive-functions supports three tiers of local development, each adding more infrastructure while keeping the same function code. Choose based on what you need.

## Tier 1 — pgpm-local

**What it is:** Postgres only via `pgpm docker`. Functions + services run as bare Node.js processes on the host. Fastest edit-run cycle.

**When to use:** Day-to-day function development, quick iteration, debugging a single function.

**Setup:**
```bash
# Start Postgres
pgpm docker start --image docker.io/constructiveio/postgres-plus:18
eval "$(pgpm env)"

# Deploy infra schema + seed function definitions
make setup-platform

# Generate function packages (if not done)
pnpm generate && pnpm install && pnpm build

# Start functions + compute-service (platform-aware)
make dev-compute

# Or start functions + legacy job-service
make dev-fn
```

**What runs where:**
| Component | Where |
|-----------|-------|
| PostgreSQL | Docker container (via pgpm) |
| MinIO (optional) | Docker container (via pgpm) |
| Functions (send-email, etc.) | Local Node.js process |
| compute-service / job-service | Local Node.js process |
| GraphQL server | Not running (unless started manually) |

**Database:** `constructive-functions-db1` (configurable via `DB_NAME` env var)

**Ports:**
| Service | Port |
|---------|------|
| PostgreSQL | 5432 |
| compute-service / job-service | 8080 |
| send-email | 8081 |
| send-verification-link | 8082 |

---

## Tier 2 — compose-local

**What it is:** Docker Compose runs infrastructure (Postgres, db-setup, GraphQL server, mailpit, platform-setup). Functions still run as local Node.js processes.

**When to use:** Testing with full infrastructure, working with email, needing GraphQL API, integration testing.

**Setup:**
```bash
# Create .env from example
cp .env.example .env

# Start infrastructure
make dev # docker compose up -d

# Wait for db-setup + platform-setup to complete
docker compose logs -f db-setup platform-setup

# Start functions + compute-service
make dev-compute

# Or start functions + legacy job-service
make dev-fn
```

**What runs where:**
| Component | Where |
|-----------|-------|
| PostgreSQL | Docker container |
| db-setup (deploys constructive, metaschema) | Docker container (runs once) |
| platform-setup (deploys constructive-infra, seeds functions) | Docker container (runs once) |
| GraphQL server | Docker container (port 3002) |
| Mailpit (email testing) | Docker container (SMTP 1025, UI 8025) |
| Functions | Local Node.js process |
| compute-service / job-service | Local Node.js process |

**Database:** `constructive` (full constructive stack)

**Ports:**
| Service | Port |
|---------|------|
| PostgreSQL | 5432 |
| GraphQL server | 3002 |
| Mailpit SMTP | 1025 |
| Mailpit UI | 8025 |
| compute-service / job-service | 8080 |
| send-email | 8081 |
| send-verification-link | 8082 |

---

## Tier 3 — k8s-local

**What it is:** Everything runs in a local Kubernetes cluster via Skaffold. Two sub-profiles: `local-simple` (plain Deployments) and `local` (Knative Serving).

**When to use:** Testing K8s manifests, verifying production-like behavior, testing Knative scaling, pre-deployment validation.

**Setup (local-simple — no Knative):**
```bash
# One-time: install K8s tooling
make setup-dev

# Start everything
make skaffold-dev
```

**Setup (local — Knative):**
```bash
# One-time: install Knative operators
cd k8s && make operators-knative-only

# Start everything
make skaffold-dev-knative
```

**What runs where:**
| Component | Where |
|-----------|-------|
| Everything | Kubernetes pods |

**Key manifests (local-simple overlay):**
- `k8s/overlays/local-simple/postgres-local.yaml` — PostgreSQL StatefulSet
- `k8s/overlays/local-simple/constructive-db-job.yaml` — DB setup Job
- `k8s/overlays/local-simple/job-service.yaml` — Legacy job service Deployment
- `k8s/overlays/local-simple/compute-service.yaml` — Platform-aware compute service Deployment
- `k8s/overlays/local-simple/constructive-server.yaml` — GraphQL server

---

## Switching between tiers

The tiers are independent. Tear down one before starting another to avoid port conflicts:

```bash
# Stop Tier 2
make dev-down

# Stop Tier 3
# Ctrl+C in the skaffold terminal

# Stop Tier 1
pgpm docker stop
```

## Environment variables

Common env vars across all tiers:

| Variable | Tier 1 Default | Tier 2 Default | Description |
|----------|---------------|---------------|-------------|
| `PGHOST` | localhost | localhost | PostgreSQL host |
| `PGPORT` | 5432 | 5432 | PostgreSQL port |
| `PGUSER` | postgres | postgres | PostgreSQL user |
| `PGPASSWORD` | (from pgpm env) | (from .env) | PostgreSQL password |
| `PGDATABASE` | constructive-functions-db1 | constructive | Database name |
| `JOBS_SCHEMA` | app_jobs | app_jobs | Jobs table schema |
| `COMPUTE_JOBS_ENABLED` | true | true | Enable compute worker |
Loading
Loading