Skip to content

docs(guides): add building custom agents guide#1179

Open
ascerra wants to merge 1 commit into
fullsend-ai:mainfrom
ascerra:custom-agent-user-guide
Open

docs(guides): add building custom agents guide#1179
ascerra wants to merge 1 commit into
fullsend-ai:mainfrom
ascerra:custom-agent-user-guide

Conversation

@ascerra
Copy link
Copy Markdown
Contributor

@ascerra ascerra commented May 19, 2026

Summary

  • Adds docs/guides/user/building-custom-agents.md — an end-to-end guide for creating new agents from scratch on per-repo fullsend installations
  • Covers all 10 steps: agent registration, prompt authoring, harness config, sandbox policies, output schemas, pre/post scripts, GitHub Actions workflows, slash-command dispatch, agent chaining, and debugging
  • Updates docs/guides/README.md index and cross-links from docs/guides/user/customizing-agents.md

Review and security fixes applied

  • Nested code blocks: Replaced \``` escaping with 4-backtick outer fences for correct GitHub rendering
  • Post-script security: Added JSON validation, status allowlisting, and a "Post-script security considerations" section documenting that agent output must be treated as untrusted input
  • Workflow permissions: Added a table explaining each permission with least-privilege guidance (remove unused permissions)
  • Glossary links: Cross-referenced sandbox, skill, and harness terms to docs/glossary.md
  • Reference section: Added links to related guides, architecture docs, security threat model, and ADR 0035

Test plan

  • Verify all internal links resolve (checked locally: all 8 links resolve)
  • Verify markdown renders correctly on GitHub, especially the 4-backtick nested code fence in Step 2
  • make lint-md-links passes once lychee is available (not installed in local env)
  • Review guide against ADR 0023 writing rules

Made with Cursor

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 19, 2026

Site preview

Preview: https://dd2ae088-site.fullsend-ai.workers.dev

Commit: fd0dc7ffc0985d8e4a0c64d69d377dfdba8bc254

@fullsend-ai-review
Copy link
Copy Markdown

fullsend-ai-review Bot commented May 19, 2026

Review

Findings

No findings.

Well-structured end-to-end guide for building custom agents on per-repo installations. All 8 internal links verified (files exist, anchors resolve). The guide accurately reflects the layered configuration architecture (ADR 0035), the sandbox/trusted-runner security model, and harness YAML structure documented in customizing-agents.md. Post-script security section correctly treats agent output as untrusted input. Bidirectional cross-links added between this guide and customizing-agents.md, and the guides index is updated.

The prior medium finding (config.yaml role validation mismatch in Step 1) is resolved — the guide no longer references config.yaml or custom role registration.

Previous run

Review

Findings

Medium

  • [correctness] docs/guides/user/building-custom-agents.md:54 — Step 1 instructs users to add a custom role name (my-agent) to .fullsend/config.yaml, but PerRepoConfig.Validate() in internal/config/config.go:248-264 restricts roles to a hardcoded allowlist returned by ValidRoles(): fullsend, triage, coder, review, fix, retro, prioritize. Adding my-agent would fail validation if the CLI ever re-parses the config (e.g., during fullsend admin install). The standalone workflow pattern in Step 8 bypasses this check at runtime (since fullsend run does not validate config.yaml roles), so the guide works in practice — but Step 1 is misleading because the config edit either fails validation or is unnecessary.
    Remediation: Either (a) note that custom role names are not yet supported in config.yaml and remove Step 1 (the standalone workflow does not need it), or (b) add a callout that custom roles require extending ValidRoles() upstream, or (c) if custom role support is planned, mark Step 1 with a > **Planned** callout per ADR 0023 writing rule 5.

Review

Findings

No findings.

Well-structured documentation guide that accurately reflects the codebase. All internal links resolve, the config.yaml format is correct for per-repo mode (verified against PerRepoConfig in internal/config/config.go), referenced scaffold scripts exist, and the security model section correctly documents the sandbox/trusted-runner separation. Follows ADR 0023 writing rules. Cross-links added in both directions between the new guide and customizing-agents.md.

Previous run

Review

Findings

Medium

  • [correctness] docs/guides/user/building-custom-agents.md:54 — Step 1 instructs users to add a custom role name (my-agent) to .fullsend/config.yaml, but PerRepoConfig.Validate() in internal/config/config.go:248-264 restricts roles to a hardcoded allowlist returned by ValidRoles(): fullsend, triage, coder, review, fix, retro, prioritize. Adding my-agent would fail validation if the CLI ever re-parses the config (e.g., during fullsend admin install). The standalone workflow pattern in Step 8 bypasses this check at runtime (since fullsend run does not validate config.yaml roles), so the guide works in practice — but Step 1 is misleading because the config edit either fails validation or is unnecessary.
    Remediation: Either (a) note that custom role names are not yet supported in config.yaml and remove Step 1 (the standalone workflow does not need it), or (b) add a callout that custom roles require extending ValidRoles() upstream, or (c) if custom role support is planned, mark Step 1 with a > **Planned** callout per ADR 0023 writing rule 5.
Previous run (2)

Review

Findings

No findings.

Well-structured documentation guide that accurately reflects the codebase. All internal links resolve, the config.yaml format is correct for per-repo mode (verified against PerRepoConfig in internal/config/config.go), referenced scaffold scripts exist, and the security model section correctly documents the sandbox/trusted-runner separation. Follows ADR 0023 writing rules. Cross-links added in both directions between the new guide and customizing-agents.md.

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label May 19, 2026
@ascerra ascerra force-pushed the custom-agent-user-guide branch from c1aba1a to 7d8c761 Compare May 19, 2026 14:53
@fullsend-ai-review fullsend-ai-review Bot added requires-manual-review Review requires human judgment and removed ready-for-merge All reviewers approved — ready to merge labels May 19, 2026
Copy link
Copy Markdown

@deboer-tim deboer-tim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a 'documenting what works today' perspective, this is a huge step to explaining how the system works and creating a working agent. Prior to this it would be complete guesswork and when things fail you'd have no idea where the problem was or if you were doing something 'unsupported'. I'm currently blocked on install but hope to give this a try soon.

I feel like I need to add that from an end-user perspective this seems extremely cumbersome for minor agents, exposes lots of internal implementation details (overly complex, brittle), and would be hard to share between teams. However, we have to start somewhere so 👍🏼 to documenting what should be possible today.

Comment thread docs/guides/user/building-custom-agents.md Outdated
Comment thread docs/guides/user/building-custom-agents.md Outdated
Comment thread docs/guides/user/building-custom-agents.md Outdated
Comment thread docs/guides/user/building-custom-agents.md Outdated
Comment thread docs/guides/user/building-custom-agents.md Outdated
Comment thread docs/guides/user/building-custom-agents.md
Comment thread docs/guides/user/building-custom-agents.md
Comment thread docs/guides/user/building-custom-agents.md Outdated
Comment thread docs/guides/user/building-custom-agents.md Outdated
Comment thread docs/guides/user/building-custom-agents.md Outdated
Comment thread docs/guides/user/building-custom-agents.md Outdated
Comment thread docs/guides/user/building-custom-agents.md Outdated
End-to-end guide for creating new custom agents from scratch on
per-repo fullsend installations. Covers agent prompts, harness config,
sandbox policies, output schemas, pre/post scripts, GitHub Actions
workflows, slash-command dispatch, and troubleshooting.

Improvements over the source material:
- Fixed nested code block rendering (4-backtick outer fences)
- Added post-script security section (treat agent output as untrusted)
- Added glossary cross-references and Reference section
- Deduplicated with customizing-agents.md via cross-links

Addressed review feedback (PR fullsend-ai#1179):
- Removed explore/refine agent references (not upstream)
- Fixed install command (fullsend admin install <owner/repo>)
- Removed WIF secrets from prerequisites (part of installation)
- Replaced security model diagram with inline summary + link
- Removed Step 1 (config.yaml roles not yet extensible)
- Added target-repo checkout step and --target-repo flag
- Added MY_VAR to workflow env block
- Reduced permissions to least-privilege (contents:read, id-token:write)
- Added trigger instructions (UI and CLI) before dispatch workflow
- Extracted agent chaining to future guide
- Condensed debugging tips into troubleshooting table

Co-authored-by: Cursor <cursoragent@cursor.com>
@fullsend-ai-review fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed requires-manual-review Review requires human judgment labels May 20, 2026
Comment on lines +171 to +208
version: 1

filesystem_policy:
include_workdir: true
read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log]
read_write: [/sandbox, /tmp, /dev/null]
landlock:
compatibility: best_effort
process:
run_as_user: sandbox
run_as_group: sandbox

network_policies:
# Required: Vertex AI for model access
vertex_ai:
name: vertex-ai
endpoints:
- host: "*.googleapis.com"
port: 443
protocol: tcp
enforcement: enforce
access: allow
binaries:
- path: "**/claude"
- path: "**/node"

# Optional: GitHub API access (if agent needs it)
github_api:
name: github-api
endpoints:
- host: "api.github.com"
port: 443
protocol: tcp
enforcement: enforce
access: allow
binaries:
- path: "**/gh"
- path: "**/curl"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
version: 1
filesystem_policy:
include_workdir: true
read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log]
read_write: [/sandbox, /tmp, /dev/null]
landlock:
compatibility: best_effort
process:
run_as_user: sandbox
run_as_group: sandbox
network_policies:
# Required: Vertex AI for model access
vertex_ai:
name: vertex-ai
endpoints:
- host: "*.googleapis.com"
port: 443
protocol: tcp
enforcement: enforce
access: allow
binaries:
- path: "**/claude"
- path: "**/node"
# Optional: GitHub API access (if agent needs it)
github_api:
name: github-api
endpoints:
- host: "api.github.com"
port: 443
protocol: tcp
enforcement: enforce
access: allow
binaries:
- path: "**/gh"
- path: "**/curl"
version: 1
filesystem_policy:
include_workdir: true
read_only: [/usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log]
read_write: [/sandbox, /tmp, /dev/null]
landlock:
compatibility: best_effort
process:
run_as_user: sandbox
run_as_group: sandbox
network_policies:
# Required: Vertex AI for model access
vertex_ai:
name: vertex-ai
endpoints:
- host: "*.googleapis.com"
port: 443
protocol: rest
enforcement: enforce
access: read-write
- host: "api.anthropic.com"
port: 443
protocol: rest
enforcement: enforce
access: read-write
binaries:
- path: "**/claude"
- path: "**/node"
# Optional: GitHub API access (if agent needs it)
github_api:
name: github-api
endpoints:
- host: "api.github.com"
port: 443
protocol: rest
enforcement: enforce
access: read-only
binaries:
- path: "**/gh"
- path: "**/curl"

With OpenShell 0.0.38 landed, it needs to be this way.

env:
ISSUE_KEY: ${{ inputs.issue_key }}
ISSUE_SOURCE: ${{ inputs.issue_source || 'github' }}
MY_VAR: ${{ vars.MY_VAR }}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This requires them to add it to the variables in their repository, for an example I think it is going too far. Can't we add a simple ABC here as value? Or you want them to add this to their variables (in that case you should mention it somewhere).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-merge All reviewers approved — ready to merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants