release: merge v0.8.0-rc3 integration branch to main

.agents/skills/obol-stack-dev/SKILL.md

@@ -79,10 +79,10 @@ All 4 paths use the same OpenClaw config pattern:
 
 ### Paid Routing Notes
 
-- The paid path uses **vanilla LiteLLM**. Do not fork LiteLLM for `paid/*`.
+- The paid path uses the **Obol LiteLLM fork** because paid-model lifecycle relies on the config-only model management API.
 - `litellm-config` carries one static route: `paid/* -> openai/* -> http://127.0.0.1:8402`.
 - `x402-buyer` runs as a **sidecar in the LiteLLM pod**, not as a separate Service.
-- `buy.py buy` updates only buyer ConfigMaps; it must not patch LiteLLM `model_list` per purchase.
+- `buy.py buy` signs auths locally and creates a `PurchaseRequest`; the controller writes per-upstream buyer files and keeps LiteLLM model entries in sync.
 - The currently validated local OSS model is `qwen3.5:9b`. Prefer that exact model in live commerce tests.
 
 ## Essential Commands
@@ -187,6 +187,7 @@ obol kubectl exec -i -n openclaw-<id> deploy/openclaw -c openclaw -- python3 - <
 
 ### MUST DO
 - Always route through `obol` CLI verbs in tests (covers CLI + helmfile + helm chart)
+- Preserve failing exit codes when logging or filtering command output. Use `set -o pipefail` or capture `PIPESTATUS` for any pipeline such as `flow.sh | tee log`, `obol stack up 2>&1 | tail`, or `helmfile ... | tee`; otherwise Helm/obol failures can be masked by the final command in the pipe.
 - Use `obol openclaw token <id>` to get Bearer token before API calls
 - Set `Authorization: Bearer <token>` on all `/v1/chat/completions` requests
 - Use `obol model setup --provider <name> --api-key <key>` for cloud provider config
@@ -195,6 +196,7 @@ obol kubectl exec -i -n openclaw-<id> deploy/openclaw -c openclaw -- python3 - <
 - Set env vars for dev mode: `OBOL_DEVELOPMENT=true`, `OBOL_CONFIG_DIR`, `OBOL_BIN_DIR`, `OBOL_DATA_DIR`
 - Prefer `qwen3.5:9b` when validating the current local paid-inference route
 - Use unique buy-side names in reused-cluster commerce tests so the sidecar cannot inherit stale in-memory spend counters
+- Use narrow review/delegation scopes for x402 changes. Name the exact files and invariants to verify, such as "controller never signs or reads remote-signer", "agent write RBAC is namespace-scoped", "paid route uses real obol CLI/human flow", and "tests support x402 v2 amount fields".
 
 ### MUST NOT DO
 - Call internal Go functions directly when testing the deployment path
@@ -203,6 +205,7 @@ obol kubectl exec -i -n openclaw-<id> deploy/openclaw -c openclaw -- python3 - <
 - Assume TCP connectivity means HTTP is ready (port-forward warmup race)
 - Use `app.kubernetes.io/instance=openclaw-<id>` for pod labels (Helm uses `openclaw`)
 - Run multiple integration tests without cleaning up between them (pod sandbox errors)
+- Delegate or accept broad "review the architecture" findings without converting them into concrete file-level checks and reproducible tests.
 
 ## Sell-Side Monetize Lifecycle
 
@@ -261,3 +264,4 @@ go test -tags integration -v -run TestIntegration_Tunnel_SellDiscoverBuySidecar_
 - **ConfigMap propagation**: File watcher takes 60-120s. Force restart verifier for immediate effect.
 - **Projected ConfigMap refresh**: the LiteLLM pod can take ~60s to reflect updated buyer ConfigMaps in the sidecar.
 - **eRPC balance lag**: `buy.py balance` uses `eth_call` through eRPC, and the default unfinalized cache TTL is 10s. After a paid request, poll until the reported balance catches up with the on-chain delta.
+- **kubectl exec shell quoting**: NEVER use `sh -c` with `fmt.Sprintf` to embed JSON or secrets in shell commands passed via `kubectl exec`. JSON body or auth tokens containing single quotes will break the shell. Instead, pass args directly: `kubectl exec ... -- wget -qO- --post-data=<json> --header=Authorization:\ Bearer\ <key> <url>`. Each argument goes as a separate argv element, bypassing shell interpretation entirely.

.agents/skills/obol-stack-dev/references/integration-testing.md

@@ -91,7 +91,7 @@ This means the test suite always passes in CI without infrastructure. Only tests
 1. Sell and reconcile a `ServiceOffer` through the LiteLLM gateway
 2. Register the offer on ERC-8004 and discover it again from the agent
 3. Probe pricing and buy it with `buy.py`
-4. Serve the model as `paid/qwen3.5:9b` through vanilla LiteLLM and the in-pod `x402-buyer` sidecar
+4. Serve the model as `paid/qwen3.5:9b` through the Obol LiteLLM fork and the in-pod `x402-buyer` sidecar
 5. Confirm the sidecar quota moves `remaining 3 -> 2` and `spent 0 -> 1`
 6. Confirm USDC moves buyer -> seller and that `buy.py balance` eventually matches the on-chain balance
 

CLAUDE.md

@@ -11,6 +11,7 @@ Obol Stack: framework for AI agents to run decentralised infrastructure locally.
 - **Commits**: Conventional commits — `feat:`, `fix:`, `docs:`, `test:`, `chore:`, `security:` with optional scope
 - **Branches**: `feat/`, `fix/`, `research/`, `codex/` prefixes
 - **Detailed architecture reference**: `@.claude/skills/obol-stack-dev/SKILL.md` (invoke with `/obol-stack-dev`)
+- **Review scope**: Avoid broad, vague review/delegation boundaries. State the exact files, invariants, and expected evidence before reviewing or spawning agents. Prefer concrete checks such as "controller cannot access signer/Secrets", "agent write RBAC is namespace-scoped", and "flow uses real obol CLI path" over generic "review architecture".
 
 ## Build, Test, Run
 
@@ -113,7 +114,7 @@ k3d: 1 server, ports 80:80 + 8080:80 + 443:443 + 8443:443, `rancher/k3s:v1.35.1-
 
 ## LLM Routing
 
-**LiteLLM gateway** (`llm` ns, port 4000): OpenAI-compatible proxy routing to Ollama/Anthropic/OpenAI. ConfigMap `litellm-config` (YAML config.yaml with model_list), Secret `litellm-secrets` (master key + API keys). Auto-configured with Ollama models during `obol stack up` (no manual `obol model setup` needed). `ConfigureLiteLLM()` patches config + Secret + restarts. Custom endpoints: `obol model setup custom --name --endpoint --model` (validates before adding). Paid remote inference stays on vanilla LiteLLM with a static route `paid/* -> openai/* -> http://127.0.0.1:8402`; no LiteLLM fork is required. OpenClaw always routes through LiteLLM (openai provider slot), never native providers; `dangerouslyDisableDeviceAuth` is enabled for Traefik-proxied access.
+**LiteLLM gateway** (`llm` ns, port 4000): OpenAI-compatible proxy routing to Ollama/Anthropic/OpenAI. ConfigMap `litellm-config` (YAML config.yaml with model_list), Secret `litellm-secrets` (master key + API keys). Auto-configured with Ollama models during `obol stack up` (no manual `obol model setup` needed). `ConfigureLiteLLM()` patches config + Secret + restarts or hot-adds via the LiteLLM model API. Paid remote inference uses the Obol LiteLLM fork plus the `x402-buyer` sidecar, with a static `paid/* -> openai/* -> http://127.0.0.1:8402` route and explicit paid-model entries when needed. OpenClaw always routes through LiteLLM (openai provider slot), never native providers; `dangerouslyDisableDeviceAuth` is enabled for Traefik-proxied access.
 
 **Auto-configuration**: During `obol stack up`, `autoConfigureLLM()` detects host Ollama models and patches LiteLLM config so agent chat works immediately without manual `obol model setup`. During install, `obolup.sh` `check_agent_model_api_key()` reads `~/.openclaw/openclaw.json` agent model, resolves API key from environment (`ANTHROPIC_API_KEY`, `CLAUDE_CODE_OAUTH_TOKEN` for Anthropic; `OPENAI_API_KEY` for OpenAI), and exports it for downstream tools.
 
@@ -133,7 +134,7 @@ Skills = SKILL.md + optional scripts/references, embedded in `obol` binary (`int
 
 ## Buyer Sidecar
 
-`x402-buyer` — lean Go sidecar for buy-side x402 payments using pre-signed ERC-3009 authorizations. It runs as a second container in the `litellm` Deployment, not as a separate Service. Agent `buy.py` commands mutate only buyer ConfigMaps; LiteLLM keeps one static public namespace `paid/<remote-model>`. The sidecar exposes `/status`, `/healthz`, and `/metrics`; metrics are scraped via `PodMonitor`. Zero signer access, bounded spending (max loss = N × price). Key code: `cmd/x402-buyer/` and `internal/x402/buyer/`.
+`x402-buyer` — lean Go sidecar for buy-side x402 payments using pre-signed ERC-3009 authorizations. It runs as a second container in the `litellm` Deployment, not as a separate Service. Agent `buy.py` signs auths locally and creates a `PurchaseRequest`; the controller writes per-upstream buyer config/auth files into the buyer ConfigMaps and keeps LiteLLM routes in sync. The sidecar exposes `/status`, `/healthz`, `/metrics`, and `/admin/reload`; metrics are scraped via `PodMonitor`. Zero signer access, bounded spending (max loss = N × price). Key code: `cmd/x402-buyer/` and `internal/x402/buyer/`.
 
 ## Development Constraints
 
@@ -142,6 +143,7 @@ Skills = SKILL.md + optional scripts/references, embedded in `obol` binary (`int
 3. **Unique namespaces** — each deployment must have unique namespace
 4. **`OBOL_DEVELOPMENT=true`** — required for `obol stack up` to auto-build local images (x402-verifier, serviceoffer-controller, x402-buyer)
 5. **Root-owned PVCs** — `-f` flag required to remove in `obol stack purge`
+6. **Narrow review boundaries** — for controller/RBAC/payment changes, spell out exact security and user-journey invariants before editing or delegating; broad review prompts have previously produced noisy findings and missed test drift
 
 ### OpenClaw Version Management
 
@@ -203,7 +205,7 @@ The Cloudflare tunnel exposes the cluster to the public internet. Only x402-gate
 
 **Docs**: `docs/guides/monetize-inference.md` (E2E monetize walkthrough), `README.md`.
 
-**Deps**: Docker 20.10.0+, Go 1.25+. Installed by obolup.sh: kubectl 1.35.3, helm 3.20.1, k3d 5.8.3, helmfile 1.4.3, k9s 0.50.18, helm-diff 3.15.4, ollama 0.20.2. Key Go: `urfave/cli/v3`, `dustinkirkland/golang-petname`, `mark3labs/x402-go`.
+**Deps**: Docker 20.10.0+, Go 1.25+. Installed by obolup.sh: kubectl 1.35.3, helm 3.20.1, k3d 5.8.3, helmfile 1.4.3, k9s 0.50.18, helm-diff 3.15.4, ollama 0.20.2. Key Go: `urfave/cli/v3`, `dustinkirkland/golang-petname`, `coinbase/x402/go` (v2 SDK, v1 wire format).
 
 ## Related Codebases
 

cmd/obol/sell.go

@@ -35,7 +35,6 @@ import (
 	"github.com/ObolNetwork/obol-stack/internal/validate"
 	x402verifier "github.com/ObolNetwork/obol-stack/internal/x402"
 	"github.com/ethereum/go-ethereum/crypto"
-	"github.com/mark3labs/x402-go"
 	"github.com/urfave/cli/v3"
 )
 
@@ -105,7 +104,7 @@ Examples:
 			&cli.StringFlag{
 				Name:  "facilitator",
 				Usage: "x402 facilitator URL",
-				Value: "https://facilitator.x402.rs",
+				Value: x402verifier.DefaultFacilitatorURL,
 			},
 			&cli.StringFlag{
 				Name:    "listen",
@@ -259,7 +258,7 @@ Examples:
 				}
 			}
 
-			chain, err := resolveX402Chain(cmd.String("chain"))
+			chain, err := x402verifier.ResolveChainInfo(cmd.String("chain"))
 			if err != nil {
 				return err
 			}
@@ -1375,15 +1374,19 @@ Examples:
 			}
 			agentURI := endpoint + "/.well-known/agent-registration.json"
 
-			// Determine signing method: remote-signer (preferred) or private key file (fallback).
+			// Determine signing method: private key file (if explicitly provided)
+			// or remote-signer (default when OpenClaw agent is deployed).
 			useRemoteSigner := false
 			var signerNS string
 
-			if _, err := openclaw.ResolveWalletAddress(cfg); err == nil {
-				ns, nsErr := openclaw.ResolveInstanceNamespace(cfg)
-				if nsErr == nil {
-					useRemoteSigner = true
-					signerNS = ns
+			// If --private-key-file is explicitly provided, honour user intent.
+			if !cmd.IsSet("private-key-file") {
+				if _, err := openclaw.ResolveWalletAddress(cfg); err == nil {
+					ns, nsErr := openclaw.ResolveInstanceNamespace(cfg)
+					if nsErr == nil {
+						useRemoteSigner = true
+						signerNS = ns
+					}
 				}
 			}
 
@@ -1430,7 +1433,7 @@ Examples:
 					}
 				} else {
 					// Fallback: direct on-chain with private key file.
-					if err := registerDirectWithKey(ctx, u, net, agentURI, fallbackKey); err != nil {
+					if err := registerDirectWithKey(ctx, cfg, u, net, agentURI, fallbackKey); err != nil {
 						u.Warnf("registration failed: %v", err)
 						continue
 					}
@@ -1494,7 +1497,8 @@ func registerDirectViaSigner(ctx context.Context, cfg *config.Config, u *ui.UI,
 	u.Printf("    Wallet:   %s", addr.Hex())
 
 	// Connect to eRPC for this network.
-	client, err := erc8004.NewClientForNetwork(ctx, "http://localhost/rpc", net)
+	rpcBaseURL := stack.LocalIngressURL(cfg) + "/rpc"
+	client, err := erc8004.NewClientForNetwork(ctx, rpcBaseURL, net)
 	if err != nil {
 		return fmt.Errorf("connect to %s via eRPC: %w", net.Name, err)
 	}
@@ -1520,7 +1524,7 @@ func registerDirectViaSigner(ctx context.Context, cfg *config.Config, u *ui.UI,
 }
 
 // registerDirectWithKey performs a direct on-chain registration using a raw private key.
-func registerDirectWithKey(ctx context.Context, u *ui.UI, net erc8004.NetworkConfig, agentURI, keyHex string) error {
+func registerDirectWithKey(ctx context.Context, cfg *config.Config, u *ui.UI, net erc8004.NetworkConfig, agentURI, keyHex string) error {
 	u.Printf("    Using direct on-chain registration with private key...")
 
 	keyHex = strings.TrimPrefix(keyHex, "0x")
@@ -1529,7 +1533,8 @@ func registerDirectWithKey(ctx context.Context, u *ui.UI, net erc8004.NetworkCon
 		return fmt.Errorf("invalid private key: %w", err)
 	}
 
-	client, err := erc8004.NewClientForNetwork(ctx, "http://localhost/rpc", net)
+	rpcBaseURL := stack.LocalIngressURL(cfg) + "/rpc"
+	client, err := erc8004.NewClientForNetwork(ctx, rpcBaseURL, net)
 	if err != nil {
 		return fmt.Errorf("connect to %s via eRPC: %w", net.Name, err)
 	}
@@ -1556,7 +1561,7 @@ func registerDirectWithKey(ctx context.Context, u *ui.UI, net erc8004.NetworkCon
 // ---------------------------------------------------------------------------
 
 // runInferenceGateway starts the x402 inference gateway and blocks until shutdown.
-func runInferenceGateway(u *ui.UI, d *inference.Deployment, chain x402.ChainConfig) error {
+func runInferenceGateway(u *ui.UI, d *inference.Deployment, chain x402verifier.ChainInfo) error {
 	gw, err := inference.NewGateway(inference.GatewayConfig{
 		ListenAddr:      d.ListenAddr,
 		UpstreamURL:     d.UpstreamURL,
@@ -1593,27 +1598,6 @@ func runInferenceGateway(u *ui.UI, d *inference.Deployment, chain x402.ChainConf
 	return gw.Start()
 }
 
-// resolveX402Chain maps a chain name to an x402 ChainConfig.
-func resolveX402Chain(name string) (x402.ChainConfig, error) {
-	switch name {
-	case "base", "base-mainnet":
-		return x402.BaseMainnet, nil
-	case "base-sepolia":
-		return x402.BaseSepolia, nil
-	case "ethereum", "ethereum-mainnet", "mainnet":
-		// Ethereum mainnet USDC: verified 2025-10-28
-		return x402.ChainConfig{
-			NetworkID:      "ethereum",
-			USDCAddress:    "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48",
-			Decimals:       6,
-			EIP3009Name:    "USD Coin",
-			EIP3009Version: "2",
-		}, nil
-	default:
-		return x402.ChainConfig{}, fmt.Errorf("unsupported chain: %s (supported: base-sepolia, base, ethereum)", name)
-	}
-}
-
 // startSignerPortForward launches a temporary port-forward to the remote-signer
 // service in the given namespace. Caller must call pf.Stop() when done.
 func startSignerPortForward(cfg *config.Config, namespace string) (*signerPortForwarder, error) {

cmd/obol/sell_test.go

@@ -5,6 +5,7 @@ import (
 	"testing"
 
 	"github.com/ObolNetwork/obol-stack/internal/config"
+	x402verifier "github.com/ObolNetwork/obol-stack/internal/x402"
 	"github.com/urfave/cli/v3"
 )
 
@@ -186,7 +187,7 @@ func TestSellInference_Flags(t *testing.T) {
 	assertStringDefault(t, flags, "chain", "base-sepolia")
 	assertStringDefault(t, flags, "listen", ":8402")
 	assertStringDefault(t, flags, "upstream", "http://localhost:11434")
-	assertStringDefault(t, flags, "facilitator", "https://facilitator.x402.rs")
+	assertStringDefault(t, flags, "facilitator", "https://x402.gcp.obol.tech")
 	assertStringDefault(t, flags, "vm-image", "ollama/ollama:latest")
 	assertIntDefault(t, flags, "vm-cpus", 4)
 	assertIntDefault(t, flags, "vm-memory", 8192)
@@ -310,9 +311,9 @@ func TestResolveX402Chain(t *testing.T) {
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			_, err := resolveX402Chain(tt.name)
+			_, err := x402verifier.ResolveChainInfo(tt.name)
 			if (err != nil) != tt.wantErr {
-				t.Errorf("resolveX402Chain(%q) error = %v, wantErr %v", tt.name, err, tt.wantErr)
+				t.Errorf("ResolveChainInfo(%q) error = %v, wantErr %v", tt.name, err, tt.wantErr)
 			}
 		})
 	}

cmd/x402-buyer/main.go

@@ -28,8 +28,10 @@ import (
 
 func main() {
 	var (
-		configPath     = flag.String("config", "/config/config.json", "path to upstream config JSON")
-		authsPath      = flag.String("auths", "/config/auths.json", "path to pre-signed auths JSON")
+		configDir      = flag.String("config-dir", "", "directory of per-upstream config files (SSA mode)")
+		authsDir       = flag.String("auths-dir", "", "directory of per-upstream auth files (SSA mode)")
+		configPath     = flag.String("config", "/config/config.json", "single config JSON file (legacy)")
+		authsPath      = flag.String("auths", "/config/auths.json", "single auths JSON file (legacy)")
 		statePath      = flag.String("state", "/state/consumed.json", "path to persisted consumed-auth state")
 		listen         = flag.String("listen", ":8402", "listen address")
 		reloadInterval = flag.Duration("reload-interval", 5*time.Second, "config/auth reload interval")
@@ -42,14 +44,9 @@ func main() {
 		log.Fatalf("load state: %v", err)
 	}
 
-	cfg, err := buyer.LoadConfig(*configPath)
+	cfg, auths, err := loadConfigAndAuths(*configDir, *authsDir, *configPath, *authsPath)
 	if err != nil {
-		log.Fatalf("load config: %v", err)
-	}
-
-	auths, err := buyer.LoadAuths(*authsPath)
-	if err != nil {
-		log.Fatalf("load auths: %v", err)
+		log.Fatalf("load config/auths: %v", err)
 	}
 
 	proxy, err := buyer.NewProxy(cfg, auths, state)
@@ -89,26 +86,25 @@ func main() {
 		defer ticker.Stop()
 
 		go func() {
+			reload := func(reason string) {
+				newCfg, newAuths, err := loadConfigAndAuths(*configDir, *authsDir, *configPath, *authsPath)
+				if err != nil {
+					log.Printf("reload (%s): %v", reason, err)
+					return
+				}
+				if err := proxy.Reload(newCfg, newAuths); err != nil {
+					log.Printf("reload proxy (%s): %v", reason, err)
+				}
+			}
+
 			for {
 				select {
 				case <-ctx.Done():
 					return
 				case <-ticker.C:
-					cfg, err := buyer.LoadConfig(*configPath)
-					if err != nil {
-						log.Printf("reload config: %v", err)
-						continue
-					}
-
-					auths, err := buyer.LoadAuths(*authsPath)
-					if err != nil {
-						log.Printf("reload auths: %v", err)
-						continue
-					}
-
-					if err := proxy.Reload(cfg, auths); err != nil {
-						log.Printf("reload proxy: %v", err)
-					}
+					reload("ticker")
+				case <-proxy.ReloadCh():
+					reload("admin")
 				}
 			}
 		}()
@@ -124,3 +120,33 @@ func main() {
 		fmt.Fprintf(os.Stderr, "shutdown: %v\n", err)
 	}
 }
+
+// loadConfigAndAuths loads config and auths from either directory mode (SSA,
+// one file per upstream) or single-file mode (legacy, all upstreams in one JSON).
+func loadConfigAndAuths(configDir, authsDir, configPath, authsPath string) (*buyer.Config, buyer.AuthsFile, error) {
+	var (
+		cfg   *buyer.Config
+		auths buyer.AuthsFile
+		err   error
+	)
+
+	if configDir != "" {
+		cfg, err = buyer.LoadConfigDir(configDir)
+	} else {
+		cfg, err = buyer.LoadConfig(configPath)
+	}
+	if err != nil {
+		return nil, nil, fmt.Errorf("config: %w", err)
+	}
+
+	if authsDir != "" {
+		auths, err = buyer.LoadAuthsDir(authsDir)
+	} else {
+		auths, err = buyer.LoadAuths(authsPath)
+	}
+	if err != nil {
+		return nil, nil, fmt.Errorf("auths: %w", err)
+	}
+
+	return cfg, auths, nil
+}