feat: add standalone constructive-infra pgpm package#51
Open
pyramation wants to merge 58 commits into
Open
Conversation
Sliced from constructive-db's monolithic constructive module using: pnpm run slice:constructive -- --renumber-alterations --strip-cross-package-deps Package includes: - constructive_infra_public schema, types, and all tables - constructive_infra_private schema and trigger functions - 254 changes, 764 files Standalone: requires only plpgsql, pgpm-inflection, pgpm-stamps No RLS policies, no grants, no job triggers
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
…mignore) The slicer now generates a complete pgpm module with: - package.json with @pgpm/* dependencies mapped from .control requires - Makefile with PGXS include - .npmignore excluding test files This fixes 'No package.json found' error when running pgpm install.
|
All alerts resolved. Learn more about Socket for GitHub. This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored. |
…ema stripping - Add missing deploy dependencies (type, constraint→column, FK→pkey) - Plan is now topologically sorted for correct deploy order - Remove metaschema partition registrations (INSERT INTO metaschema_public.*) that require the closed-source MetaSchema system - Successfully tested: pgpm deploy --yes --createdb --database testinfra
…tion Separate pgpm module that registers the constructive-infra function tables as a MetaSchema function_module. This allows querying MetaSchema to discover the infra function tables (definitions, invocations, execution_logs, secret_definitions). Dependencies: - constructive-infra (the raw DDL) - @pgpm/metaschema-modules (function_module table) - @pgpm/services (service registration layer) The registration INSERT provides explicit schema/table names for the platform-scoped infra tables, letting the MetaSchema trigger system handle ID resolution and API routing.
Adds a CI workflow that runs 'pgpm test-packages --full-cycle' on every PR that touches pgpm/ files. This validates the full deploy/verify/revert/deploy cycle for constructive-infra against a real postgres-plus:18 instance. constructive-infra-services is excluded since it requires the full MetaSchema system (closed-source triggers) to deploy. Also fixes the function_requirement type revert SQL which was missing the TYPE keyword (DROP → DROP TYPE).
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub. |
- job/compute-worker: platform-aware worker that discovers functions from constructive_infra_public.platform_function_definitions (TTL-cached), tracks invocations in platform_function_invocations, dispatches via HTTP - job/compute-service: orchestrator (callback server + ComputeWorker + Scheduler), mirrors job/service patterns - scripts/setup-platform-db.sh: Tier 1 setup (pgpm deploy + seed) - scripts/dev-compute.ts: dev launcher for compute-service + functions - scripts/seed-functions.sql: registers send-email + send-verification-link - docker-compose.yml: adds platform-setup service (Tier 2) - k8s/overlays/local-simple/compute-service.yaml: K8s manifest (Tier 3) - Makefile: adds setup-platform, dev-compute targets + tier documentation - .agents/skills/dev-tiers: skill for the 3-tier dev model - .agents/skills/compute-worker: skill for the compute-worker system - pgpm/constructive-infra: adds @pgpm/database-jobs dependency
…error) pgpm deploy requires module dependencies to be pre-installed via 'pgpm install' before deployment. Without this, packages like @pgpm/database-jobs fail with 'extension not available'. Adds pgpm install to: - scripts/setup-platform-db.sh (Tier 1) - docker-compose.yml platform-setup service (Tier 2) - .github/workflows/pgpm-test.yaml (CI)
- make status: shows Docker containers, PG connection, databases with infra schema, Node/pnpm/pgpm versions, and build state - make verify-platform: checks DB exists, infra + jobs schemas deployed, tables present, functions seeded — exits non-zero with fix instructions if anything is wrong
Dan committed all extension deps (database-jobs, metaschema-modules, services, etc.) directly into extensions/. pgpm install is no longer needed before deploy. Also restores pgpm volume to :ro in Docker Compose and mounts extensions/ for the platform-setup service.
Built-in function definitions (send-email, send-verification-link) are now deployed as a pgpm fixture inside constructive-infra. The fixture uses the standard deploy/revert/verify pattern following the inflection module's fixtures convention. - deploy: INSERT ON CONFLICT DO NOTHING - revert: DELETE WHERE is_built_in AND scope='platform' - verify: SELECT 1 from seed row Removes manual psql seed step from setup script, docker-compose, and verify-platform fix instructions. pgpm deploy now handles everything.
Moves the built-in function seed data (send-email, send-verification-link) from constructive-infra into its own pgpm package: constructive-infra-seed. This keeps constructive-infra as pure DDL (schemas, tables, triggers) and the seed data as a separate deployable unit. The seed package depends on constructive-infra via the .control file requires. Both packages pass pgpm test-packages --full-cycle.
compute-service discovers functions from the database, so it doesn't need direct workspace deps on send-email-fn or send-verification-link-fn (which are generated packages in generated/ and not in the workspace).
…nfigs - New fixture: seed_built_in_secrets seeds MAILGUN_* and SMTP_* into platform_secret_definitions (with well-known default database_id) - Function definitions now include required_secrets and required_configs arrays (function_requirement[] type) linking functions to their deps - All SQL files now use proper pgpm format: BEGIN/COMMIT wrappers, '-- Deploy:' header style with '-- made with <3 @ constructive.io' - New script: load-platform-env.sh reads .env, cross-references against DB function requirements, reports satisfied vs missing keys - .env.example updated with SMTP/Mailpit + dry-run toggle sections - Makefile: added 'make check-env' target
Procedural lifecycle targets: make up # prereqs → docker → bootstrap → deploy → seed → verify make up DB_NAME=mydb # same with custom DB make down # stop docker compose + pgpm docker stop DROP=1 make down DB_NAME=mydb # also drop the DB make up:email-job # start mailpit + compute-service (SMTP mode) make down:email-job # stop mailpit + compute-service email-job-up verifies platform is up first, starts mailpit, loads .env (with sane SMTP defaults), then launches compute-service. status now shows mailpit container state.
…rrides - Skip python-example and other non-node-graphql functions from startup - Print a clear port/service summary table before launching processes - Respect SEND_EMAIL_DRY_RUN, SEND_VERIFICATION_LINK_DRY_RUN, EMAIL_SEND_USE_SMTP, and SMTP_FROM from environment (was hardcoded)
- www/: Vite + React + Express app with 6 tabs (Functions, Secrets, Jobs, Invocations, Commands, Terminal) - Express backend: REST API for DB queries + WebSocket terminal via child_process - Secrets & Namespaces tab: shows seeded secret definitions and default namespace - Seed: add default platform namespace, link functions to namespace_id - Function API: parse composite-type arrays into proper JSON - Makefile: add make up:www target - scripts/www-up.sh: checks platform, installs deps, starts Vite + Express
…ster management - Install commander and kubernetesjs packages in www/ - Add K8s proxy endpoint (/api/k8s/*) in Express server - Add K8s tab with Pods/Deployments/Services views per namespace - Namespace selector defaults to constructive-functions - Graceful error when kubectl proxy not running
… tab - ansiToHtml() converts ANSI escape sequences into colored <span> elements - Remove max-h-60 so output expands naturally with content - Exit code shown in a separate footer bar - Strip remaining escape sequences that don't match known codes
Step 4b cross-references loaded env vars against platform_function_definitions required_secrets/required_configs. Warns about missing secrets but doesn't block (defaults still work for Mailpit/SMTP mode). Suggests 'make check-env' for details.
- Reads .env file and merges with process.env + dev defaults - Queries platform_function_definitions for required_secrets/required_configs - Reports per-function coverage at startup (✓ all set / ● N missing) - Priority: .env > process.env > hardcoded defaults - Graceful fallback if DB query fails (schema not deployed)
Both tables are PARTITION BY RANGE(created_at) but had no partitions, causing 'no partition of relation found for row' on INSERT. Adds DEFAULT partition for each so rows land somewhere until time-based partitions are created.
Secrets tab now shows: - Function coverage badges (e.g. send-email 5/8) - All required secrets/configs from DB + .env merged - Editable input fields with eye/reveal toggle for sensitive keys - 'Save to .env' button writes grouped .env file to project root - Shows which functions require each secret - File status indicator (.env exists / will be created) Backend adds: - GET /api/env — reads .env and returns parsed vars - POST /api/env — merges values + writes grouped .env file
Each function card now has a 'Trigger' button that opens an inline payload editor with sensible defaults (send-email gets to/subject/html, send-verification-link gets to/type/link). Submitting creates a job in app_jobs.jobs and shows success/error inline — no need to switch tabs.
…ents - Add .agents/skills/fbp/SKILL.md mapping FBP NodeDefinitions to platform functions - Add docs/spec/fbp-integration.md research doc with full mapping spec - Add Flows tab with React Flow: drag-and-drop function nodes, edge creation, localStorage persistence, sidebar palette, and minimap - Enhance FunctionsPanel trigger success message with Invocations tab link - Install @xyflow/react for the flow graph canvas
…tion - Add platform_secret_values table (pgpm migration: deploy/revert/verify) Columns: id, secret_name, configured_value, database_id, created_at, updated_at Unique constraint on (secret_name, database_id) - Backend API (www/server/index.ts): GET /api/secret-values — read configured values from DB POST /api/secret-values — write configured values to DB POST /api/secrets/sync-from-db — DB values -> .env POST /api/secrets/sync-to-db — .env values -> DB POST /api/env now also syncs to DB on save (best-effort) - Frontend (SecretsPanel.tsx): Add 'Sync from DB' and 'Sync to DB' buttons DB status indicator in header - Refactor scripts/dev-compute.ts: Secrets pipeline: .env > DB > hardcoded defaults Per-function env injection (only needed secrets/configs) Fail fast on missing required secrets, warn on optional - Add make secrets:sync (bidirectional sync script) - Add scripts/secrets-sync.sh for CLI-based sync
send-email: to, subject, html (was correct) send-verification-link: email_type, email, email_id, verification_token (was using wrong field names)
feat: bidirectional DB ↔ .env secrets sync + per-function env injection
feat: FBP integration, Flow Graph UI, and inline trigger enhancements
Sliced from constructive-db without RLS/grants — standalone publishable: - constructive-store: encrypted secrets (platform/org/user), config, user state - constructive-objects: content-addressable merkle store (store, object, commit, ref) - constructive-fbp: flow-based graph engine with graph-specific merkle store - constructive-storage: file uploads, buckets, versioning, GC - constructive-infra: updated to latest slice output
- Updated up.sh to deploy all pgpm modules in dependency order: infra → store → objects → fbp → storage - Added MinIO (S3-compatible) to docker-compose.yml and up.sh for constructive-storage file upload support - Updated down.sh to stop MinIO container - Updated verify-platform.sh to check all module schemas - Updated docker-compose platform-setup to deploy all modules - Updated .env.example with MinIO/S3 config vars - Refreshed storage module: force_current_user_actor_id triggers moved to security, data-integrity triggers kept in base - Added constructive-infra to constructive-store .control requires
The grep | true pattern was hiding deploy failures. Now captures output and shows the full error when deploy doesn't report SUCCESS.
Root cause: standalone module .control files only had plpgsql, pgpm-inflection,pgpm-stamps but objects needs uuid-ossp (for uuid_generate_v5/uuid_ns_url) and fbp needs uuid-ossp + pgcrypto (for digest()). Without these, pgpm deploy fails on function creation. Also improved deploy output detection: write to temp file instead of shell variable to avoid macOS bash 3 issues with large output.
- Copy @FBP packages (types, spec, evaluator, graph-editor) into packages/ as workspace dependencies consumed as source TypeScript by Vite - Rewrite FlowsPanel to use <GraphEditor> with full feature set: subnets, collapse-to-subnet (Shift+C), selections, hotkeys, properties panel, node palette, status bar, edge creation, box select - Wire @fbp/evaluator for local flow execution (evaluate button in sidebar) - Load built-in node definitions (math, core, ui, net, graphql) + platform functions from API as NodeDefinitions - Add onGraphChange callback to GraphProvider/GraphEditor for localStorage persistence - Add Vite aliases for @fbp/* workspace packages - Add Tailwind @source directive to scan graph-editor components - Remove @xyflow/react dependency - Add docs/spec/fbp-deep-plan.md (comprehensive feature audit + migration plan)
The slicer's topological sort placed procedures before column additions because procedures only declared schema/table deps, not column deps. Functions referencing composite types (e.g. IN obj schema.table) need all columns to exist first. Reordered all 5 module plans: schemas → types → tables → columns → alterations → constraints → indexes → trigger_fns → procedures → triggers.
The job queue (app_jobs schema) is provided by pgpm-database-jobs. Adding it as a dependency ensures it gets deployed automatically when constructive-infra is deployed.
In the monolith, partitions are created at runtime by metaschema triggers during database provisioning. In standalone mode we don't have metaschema, so we create DEFAULT partitions in the seed step. Covers all 5 partitioned tables: - constructive_infra_public.platform_function_invocations - constructive_infra_public.platform_function_execution_logs - constructive_infra_public.platform_namespace_events - constructive_fbp_private.function_graph_executions - constructive_fbp_private.function_graph_execution_outputs
- Rich function cards (320px) with ⚡ icon, description, task ID, secrets/configs counts, scope badge, blue/green ports - Compact nodes (180px) for built-in types (math, json, flow, etc) - Zinc-950 dark theme across all editor components (was slate) - Variable-width box-select hit testing in GraphContext - External graph sync in GraphProvider (fixes sidebar add-to-canvas) - FlowsPanel sidebar: FLOWS + FUNCTIONS sections with mock data fallback - Top bar: flow name input + Save + Evaluate buttons
- scripts/register-functions.ts: reads functions/*/handler.json and generates the seed SQL (deploy/revert/verify) that registers them in platform_function_definitions. Run: make register - handler.json now carries registration metadata: scope, requiredSecrets, requiredConfigs (+ reserved payloadSchema for FBP typed ports) - Node example (functions/example): copy-ready template with documented handler signature and context destructuring - Python example (functions/python-example): copy-ready template with FastAPI runtime Adding a new function is now: 1. cp -r functions/example functions/my-function 2. Edit handler.json + handler.ts 3. make register && pgpm kill && make up
…ystem - Hide GraphEditor's built-in NodePalette (showNodePalette=false) - Hide GraphEditor's header bar (showHeader=false) — FlowsPanel has its own - Merge all node types into outer sidebar: Functions (3), Graph I/O (3), Constants (1), Math (2), JSON (2), Flow Control (1), String (2), Layout (1), Form (3), Content (1), GraphQL (1) - Functions use rich card styling (⚡ icon, description, monospace name) - Built-in nodes use compact styling with category icons - All items are both clickable (click-to-add) and draggable (drag-to-canvas) - Collapsible category sections with chevron toggles and item counts - Remove duplicate PALETTE_DEFINITIONS (graph/input etc) — evaluator already provides them - Add useMemo for definitions and groupedDefinitions for performance
The + button now fetches all invocable functions from /api/functions instead of hardcoding send-email and send-verification-link. Uses a generic JSON payload editor so any function can be triggered.
- Compact nodes: 180px → 240px wide, header 28px → 32px - Icon + title left-aligned as a group (icon then title, not icon at edge + centered title) - Add shadow to compact nodes (consistent with rich function cards) - Rounded corners rx=8 → rx=10 for compact nodes - Auto-layout spacing increased (250 → 300) for wider nodes - GraphContext node dimensions synced with GraphNode
- Add payloadSchema (JSON Schema) to all 4 handler.json manifests: send-email, send-verification-link, node-example, python-example - Add payload_schema JSONB column to platform_function_definitions - Wire register-functions.ts to emit payload_schema in seed SQL - Add /api/definitions endpoint returning FBP NodeDefinition[] with typed inputs[].schema, outputs[], and props[] (secrets/configs) - FunctionsPanel now renders payload schema properties inline - JobsPanel generates default payloads from schema when available - Types: NodeDefinition, PortDef, PropDef aligned with @fbp/types
…into DB - Add platform_secret_values table to constructive-infra (secret_name, configured_value, database_id) - Extend register-functions.ts to auto-generate seed_built_in_secrets.sql from all handler.json requiredSecrets/requiredConfigs (11 unique defs across 4 functions) - Update load-platform-env.sh to UPSERT .env values into platform_secret_values after coverage check - Update verify-platform.sh to check platform_secret_values table + seeded counts - up.sh step 8 now syncs .env into DB instead of just checking coverage
…oded platform_secret_values - Remove platform_secret_values table from constructive-infra - Remove seed_built_in_secrets.sql generation from register-functions.ts - Remove seed_built_in_secrets from infra-seed pgpm.plan - Update load-platform-env.sh to UPSERT into real tables: - secrets → constructive_store_private.platform_secrets (bytea) - configs → constructive_store_public.platform_config (text) - Update secrets-sync.sh for bidirectional sync with real tables - Update dev-compute.ts to load from real tables - Update www/server to query real platform_secrets + platform_config - Update verify-platform.sh to check real table counts - Derive secret definitions from inlined arrays on platform_function_definitions - All tables use namespace_id from default namespace (database_id 0000...)
feat: replace React Flow with @fbp/graph-editor + unified node palette
Feat/fbp flow graph
feat: platform compute-worker with all infrastructure modules + MinIO
Replace all raw INSERT/convert_to/convert_from with the real upstream functions: - Writes: platform_secrets_set(name, value, 'pgp', 'default') → trigger encrypts via pgp_sym_encrypt with per-secret key_id - Reads: platform_secrets_get(name, NULL, 'default') → decrypts via pgp_sym_decrypt using stored key_id Updated files: - load-platform-env.sh: .env → DB sync uses platform_secrets_set - secrets-sync.sh: bidirectional sync uses set/get functions - www/server/index.ts: all 5 secret endpoints use set/get - dev-compute.ts: loads decrypted secrets via platform_secrets_get
refactor: use platform_secrets_set/get for production PGP encryption
The upstream platform_secrets_set/get/del functions read database_id
from current_setting('jwt.claims.database_id'). In standalone mode
there's no PostGraphile middleware to set these, so we need to set
them explicitly in each transaction.
All 7 JWT claims are faked for standalone mode:
database_id = 00000000-0000-0000-0000-000000000000
user_id = 00000000-0000-0000-0000-000000000001
token_id = 00000000-0000-0000-0000-000000000002
session_id = 00000000-0000-0000-0000-000000000003
ip_address = 127.0.0.1
user_agent = constructive-functions/standalone
origin = http://localhost:3000
Each calling path wraps secret operations in BEGIN/SET LOCAL/COMMIT:
- load-platform-env.sh: per-secret psql calls
- secrets-sync.sh: write + read paths
- www/server/index.ts: withJwtClaims() helper for all 5 endpoints
- dev-compute.ts: runtime secret loading
fix: set all JWT claims before calling upstream secret functions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds standalone pgpm packages for the constructive-infra schema, sliced from constructive-db's monolithic export, plus a CI workflow to continuously validate they deploy correctly.
Three additions:
pgpm/constructive-infra— Standalone DDL (schemas, types, tables, columns, constraints, indexes, triggers) for the infra function system:platform_function_definitions,platform_function_invocations,platform_function_execution_logsplatform_secret_definitions,platform_namespaces,platform_namespace_eventspgpm/constructive-infra-services— MetaSchema registration layer:constructive-infra+@pgpm/metaschema-modules+@pgpm/servicesfunction_moduleINSERT to register infra tables as a MetaSchema module.github/workflows/pgpm-test.yaml— pgpm integration test workflow:pgpm test-packages --full-cycleagainstpostgres-plus:18on every PR touchingpgpm/pgpm-modulesrepoAlso adds
pgpm.jsonworkspace config at root.Companion PR: https://github.com/constructive-io/constructive-db/pull/1575 (slicer improvements that make this reproducible)
Link to Devin session: https://app.devin.ai/sessions/e4d315e507cf4d95ac42056ad22b8925
Requested by: @pyramation