Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
209 lines (146 loc) · 6.4 KB

DEVELOPMENT.md

File metadata and controls

209 lines (146 loc) · 6.4 KB

Development Guide

This guide covers setup and workflows for the 508.dev monorepo (discord bot + api + worker + shared package).

Prerequisites

  • Python 3.12+
  • uv
  • Docker (optional, for Compose-based local runs)

Monorepo Layout

apps/discord_bot/src/five08/discord_bot/  # Discord bot package
apps/api/src/five08/backend/              # Backend API package
apps/worker/src/five08/worker/            # Worker consumer package
packages/shared/src/five08/     # Shared package

Setup

  1. Install dependencies:
uv sync
  1. Configure environment:
cp .env.example .env

The backend API process runs Alembic migrations on startup (apps/worker/src/five08/worker/db_migrations.py) so the jobs table is created or upgraded before requests are accepted.

  1. Start local infrastructure:
./scripts/dev.sh infra
./scripts/dev.sh api
./scripts/dev.sh worker
./scripts/dev.sh discord-bot

infra brings up only the Docker infra. Use the host-service subcommands to run the app processes with per-worktree ports and derived localhost URLs.

To launch infra plus all host-run services together with prefixed logs:

./scripts/dev.sh all

Show, export, or stop the local dev environment:

./scripts/dev.sh ports
./scripts/dev.sh env
./scripts/dev.sh down

./scripts/dev.sh env emits shell-safe exports for the current worktree and avoids printing the resolved Postgres password directly.

  1. Run services on the host:
# bot
uv run --package discord_bot discord-bot

# webhook ingest API
uv run --package api backend-api

# job consumer
uv run --package worker worker-consumer

# EspoCRM search / REPL / batch updates
uv run --package five08 crmctl repl

Docker Compose Workflow

For day-to-day development, prefer ./scripts/dev.sh plus host-run app services. That entrypoint exports deterministic per-worktree localhost ports and service URLs so the apps work without manual overrides. Use the Compose wrapper when you need full container parity, including Coolify-style runs.

Start full stack (discord_bot + api + worker + redis + postgres + minio):

./scripts/docker-compose.sh up --build

Note: the service Dockerfiles use BuildKit cache mounts, so containerized builds require BuildKit-capable Docker / docker compose build support.

Stop stack:

./scripts/docker-compose.sh down

Show the deterministic host ports assigned to the current worktree:

./scripts/docker-compose.sh print-ports

Set *_HOST_PORT or COMPOSE_PROJECT_NAME in .env or the invoking shell if you need fixed values; otherwise the wrapper computes deterministic per-worktree ones.

Testing and Quality

./scripts/test.sh
./scripts/lint.sh
./scripts/format.sh
./scripts/mypy.sh

Adding Bot Features

Bot features remain Discord.py cogs in:

  • apps/discord_bot/src/five08/discord_bot/cogs/

Pattern:

from discord.ext import commands

class MyCog(commands.Cog):
    def __init__(self, bot: commands.Bot) -> None:
        self.bot = bot

async def setup(bot: commands.Bot) -> None:
    await bot.add_cog(MyCog(bot))

Adding Worker Jobs

  1. Add job function in apps/worker/src/five08/worker/jobs.py.
  2. Enqueue from apps/api/src/five08/backend/api.py (or from bot code if needed).
  3. Ensure job type/queue settings and Postgres settings are configured in .env.

Job architecture

  • API layer persists jobs first in Postgres with idempotency keys.
  • Queue layer uses Dramatiq actors over Redis for delivery.
  • MinIO is the current internal transfer mechanism (bucket: internal-transfers) and is intended only for stack-internal file movement; external S3 integrations are separate.

Worker CRM Flow

  • EspoCRM webhooks are accepted at POST /webhooks/espocrm.
  • Each event enqueues five08.worker.jobs.process_contact_skills_job.
  • Jobs use modules under apps/worker/src/five08/worker/crm/ to:
    • fetch contact + attachments from EspoCRM
    • extract text from resume-like files
    • extract skills (LLM when configured, heuristic fallback otherwise)
    • update contact skills field in EspoCRM
  • Manual queueing is available via POST /process-contact/{contact_id}.
  • Human action audit ingest is available at POST /audit/events.

CRM Cleanup CLI

Use crmctl when you want direct EspoCRM contact search/update control outside the Discord UI.

Examples:

uv run --package five08 crmctl search --where timezone__is_null=true --where location__is_not_null=true
uv run --package five08 crmctl batch-update --where timezone__is_null=true --where location__is_not_null=true --update timezone=@location
uv run --package five08 crmctl batch-update --where timezone__is_null=true --where location__is_not_null=true --update timezone=@location --apply

Inside uv run --package five08 crmctl repl, contacts are mutable Python objects:

contacts = search(timezone__is_null=True, location__is_not_null=True)
contact = contacts[0]
contact.timezone = contact.infer_timezone()
contact.save()

Discord CRM Audit Flow

  • CRM slash commands in apps/discord_bot/src/five08/discord_bot/cogs/crm.py emit best-effort audit events for human actions.
  • Audit writing is centralized in apps/discord_bot/src/five08/discord_bot/utils/audit.py.
  • Audit writes must never break command execution; failures are logged as warnings only.

Environment Variables

Use .env.example as source of truth. Key categories:

  • Shared queue/runtime: REDIS_URL, REDIS_QUEUE_NAME, POSTGRES_URL, JOB_MAX_ATTEMPTS, JOB_RETRY_BASE_SECONDS, JOB_RETRY_MAX_SECONDS, LOG_LEVEL, webhook settings. Local defaults target host-run services; docker-compose.yml injects Docker-network URLs for containerized runs.
  • Bot credentials/integrations: Discord, email, Espo, Kimai
  • Discord CRM audit writer: AUDIT_API_BASE_URL, AUDIT_API_TIMEOUT_SECONDS (plus shared API_SHARED_SECRET)
  • Worker controls: WORKER_NAME, WORKER_QUEUE_NAMES, WORKER_BURST
  • Worker CRM processing: MAX_ATTACHMENTS_PER_CONTACT, MAX_FILE_SIZE_MB, ALLOWED_FILE_TYPES, OPENAI_API_KEY, OPENAI_BASE_URL, OPENAI_MODEL, RESUME_EXTRACTOR_VERSION
  • Resume upload UX wiring: BACKEND_API_BASE_URL on bot; worker-side LinkedIn field mapping is fixed in code.

CI Notes

GitHub Actions runs tests, lint, mypy, and security checks against:

  • apps/discord_bot/src/five08/discord_bot/
  • apps/worker/src/five08/worker/
  • packages/shared/src/five08/
  • tests/