Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Thank you for your interest in contributing to SubmitQueue! Whether you are repo
## Getting Started

1. Read the [Development Setup](doc/howto/DEVELOPMENT.md) guide for prerequisites, building, and running tests.
2. Review the [Architecture Guide](CLAUDE.md) to understand project layout, conventions, and code style.
2. Review the [Architecture Guide](AGENTS.md) to understand project layout, conventions, and code style.
3. Check the [Testing Guide](doc/howto/TESTING.md) for testing patterns and requirements.

## Development Workflow
Expand Down Expand Up @@ -34,7 +34,7 @@ Thank you for your interest in contributing to SubmitQueue! Whether you are repo
- Include tests for new functionality.
- Ensure all existing tests pass (`make test`).
- Ensure CI passes before requesting review.
- Follow the existing code style and patterns described in the [Architecture Guide](CLAUDE.md).
- Follow the existing code style and patterns described in the [Architecture Guide](AGENTS.md).
- Fill out the PR template with a description, motivation, and test plan.

## Code Review
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Designed for large monorepos and fast-moving teams where concurrent changes can

## Repository layout

Cross-domain Go code (errors, metrics, consumer framework, HTTP helpers, shared entities, shared extension contracts) lives under [`platform/`](platform/README.md). Each product domain has its own tree (`submitqueue/`, `stovepipe/`, …) and grows into `gateway/`, `orchestrator/`, `entity/`, `extension/`, and domain-local `core/` — though a domain may start smaller (Stovepipe is currently a single Ping-only service with just `controller/`). See [CLAUDE.md](CLAUDE.md) for conventions and import paths.
Cross-domain Go code (errors, metrics, consumer framework, HTTP helpers, shared entities, shared extension contracts) lives under [`platform/`](platform/README.md). Each product domain has its own tree (`submitqueue/`, `stovepipe/`, …) and grows into `gateway/`, `orchestrator/`, `entity/`, `extension/`, and domain-local `core/` — though a domain may start smaller (Stovepipe is currently a single Ping-only service with just `controller/`). See [AGENTS.md](AGENTS.md) for conventions and import paths.

## Quick Start

Expand Down Expand Up @@ -49,7 +49,7 @@ The queue's own logic is real in all three: validation, batching, conflict analy
| [Development Setup](doc/howto/DEVELOPMENT.md) | Prerequisites, build, environment, IDE setup |
| [Contributing](CONTRIBUTING.md) | How to contribute, workflow, guidelines |
| [Testing Guide](doc/howto/TESTING.md) | Unit, integration, and E2E testing patterns |
| [Architecture Guide](CLAUDE.md) | Project layout, patterns, conventions |
| [Architecture Guide](AGENTS.md) | Project layout, patterns, conventions |
| [Examples](service/README.md) | Running services, clients, API reference |
| [RFCs](doc/rfc/index.md) | Design documents and proposals |

Expand Down
26 changes: 13 additions & 13 deletions doc/howto/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,9 @@ make build-all-linux # Build Linux binaries for the local docker-
- Speed: Fast (< 1s typically)

**2. Integration Tests** - Service in isolation with real dependencies
- Location: `test/integration/submitqueue/{service}/`
- Run: `make integration-test-{service}`
- Containers: MySQL + one service
- Location: `test/integration/submitqueue/<area>/` (e.g., `gateway/`, `orchestrator/`, `extension/<ext>/`)
- Run: `make integration-test-submitqueue-gateway`, `make integration-test-submitqueue-orchestrator`, `make integration-test-submitqueue-consumer`, or `make integration-test-extensions`
- Containers: MySQL + one service or the extension's dependencies
- Tests one service isolated from others

**3. E2E Tests** - Complete workflows across all services
Expand Down Expand Up @@ -194,14 +194,14 @@ make local-stop
make local-submitqueue-gateway-start

# Test Ping API (port shown by make local-submitqueue-ps)
grpcurl -plaintext -d '{"message": "hello"}' localhost:<PORT> submitqueue.SubmitQueueGateway/Ping
grpcurl -plaintext -d '{"message": "hello"}' localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Ping

# Test Land API
grpcurl -plaintext -d '{
"queue": "test-queue",
"change": {"source": "github", "ids": ["PR-123"]},
"strategy": "REBASE"
}' localhost:<PORT> submitqueue.SubmitQueueGateway/Land
}' localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Land

# Stop
make local-submitqueue-gateway-stop
Expand All @@ -213,7 +213,7 @@ make local-submitqueue-gateway-stop
make local-submitqueue-orchestrator-start

# Test Ping API (port shown by make local-submitqueue-ps)
grpcurl -plaintext -d '{"message": "hello"}' localhost:<PORT> submitqueue.SubmitQueueOrchestrator/Ping
grpcurl -plaintext -d '{"message": "hello"}' localhost:<PORT> uber.submitqueue.orchestrator.SubmitQueueOrchestrator/Ping

# Stop
make local-submitqueue-orchestrator-stop
Expand Down Expand Up @@ -256,18 +256,18 @@ brew install grpcurl # macOS
grpcurl -plaintext localhost:<PORT> list

# Describe a service
grpcurl -plaintext localhost:<PORT> describe submitqueue.SubmitQueueGateway
grpcurl -plaintext localhost:<PORT> describe uber.submitqueue.gateway.SubmitQueueGateway

# Call Ping
grpcurl -plaintext -d '{"message": "test"}' \
localhost:<PORT> submitqueue.SubmitQueueGateway/Ping
localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Ping

# Call Land
grpcurl -plaintext -d '{
"queue": "my-queue",
"change": {"source": "github", "ids": ["PR-456"]},
"strategy": "REBASE"
}' localhost:<PORT> submitqueue.SubmitQueueGateway/Land
}' localhost:<PORT> uber.submitqueue.gateway.SubmitQueueGateway/Land
```

### Available Commands
Expand Down Expand Up @@ -363,9 +363,9 @@ docker network ls | grep sq-test | awk '{print $1}' | xargs docker network rm

### Adding Integration Tests

1. Add test to `test/integration/submitqueue/{service}/suite_test.go`
2. Use suite's resources (`s.client`, `s.db`)
3. Run: `make integration-test-{service}`
1. Add test to `test/integration/submitqueue/<area>/suite_test.go` (e.g., `test/integration/submitqueue/gateway/suite_test.go` for Gateway, `test/integration/submitqueue/orchestrator/suite_test.go` for Orchestrator, or a subdirectory under `test/integration/submitqueue/extension/` for extension tests).
2. Use suite's resources (`s.client`, `s.db`).
3. Run the matching Makefile target such as `make integration-test-submitqueue-gateway`.

Example:
```go
Expand All @@ -387,7 +387,7 @@ assert.Equal(s.T(), "expected", resp.Value)

## See Also

- [CLAUDE.md](../../CLAUDE.md) - Development guidelines and project structure
- [AGENTS.md](../../AGENTS.md) - Development guidelines and project structure
- [service/submitqueue/docker-compose.yml](../../service/submitqueue/docker-compose.yml) - Full stack service definitions
- [service/submitqueue/gateway/server/docker-compose.yml](../../service/submitqueue/gateway/server/docker-compose.yml) - Gateway isolation
- [service/submitqueue/orchestrator/server/docker-compose.yml](../../service/submitqueue/orchestrator/server/docker-compose.yml) - Orchestrator isolation
68 changes: 34 additions & 34 deletions doc/rfc/runway/workflow.md
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# Runway Workflow

Runway is the landing service: it owns VCS operations — mergeability checking and landing — on behalf of SubmitQueue. Runway is a single service (the domain *is* the service): it subscribes to two inbound topics (`merge-conflict-checker`, `merger`) and publishes results to two outbound topics (`merge-conflict-checker-signal`, `merger-signal`). It is a consumer-only service with no gateway; work arrives via topic queues and results leave via topic queues.
Runway is the landing service: it owns VCS operations — mergeability checking and landing — on behalf of SubmitQueue. Runway is a single service (the domain *is* the service): it subscribes to two inbound topics (`merge-conflict-check`, `runway-merge`) and publishes results to two outbound topics (`merge-conflict-check-signal`, `merge-signal`). It is a consumer-only service with no gateway; work arrives via topic queues and results leave via topic queues.

## Merge-conflict check and merge

The two queues operate at different granularities:
The two queues are the same shape but different commit semantics:

- **merge-conflict-check** is request-level. A merge request carries an ordered sequence of steps (changes + merge strategy). Runway performs a read-only trial merge and publishes per-step mergeability results back.
- **merge-conflict-check** is a dry run. A merge request carries an ordered sequence of steps (changes + merge strategy). Runway performs a read-only trial merge and publishes per-step mergeability results back.

- **merge** is batch-level. A merge request carries the same payload but Runway commits the result and reports the revisions it produced (per-step output IDs).
- **merge** is the committing version. A merge request carries the same payload but Runway commits the result and reports the revisions it produced (per-step output IDs).

A third operation — **promote** pushes a commit to a ref as-is (`--ff-only`). The primary use case is forwarding a landed SHA from `main` to `verified/main` without creating a new merge commit. Promote reuses the merge queue with the `PROMOTE` merge strategy; Runway fast-forwards the target ref and reports the same SHA back as the output ID.
- **promote** is a special `PROMOTE` strategy on the `merge` queue: it pushes a commit to a ref as-is (`--ff-only`). The primary use case is forwarding a landed SHA from `main` to `verified/main` without creating a new merge commit.

These are independent input-output flows. A merge-conflict check can run without a merge ever running, and a merge does not depend on a prior check.

Expand All @@ -23,39 +23,39 @@ The outbound topics partition by SubmitQueue queue name, matching SubmitQueue's
## Workflow

```
┌─────────────────────────────────────────────────────┐
│ submitqueue orchestrator │
└──────────┬───────────────────────────┬──────────────┘
│ │
MergeRequest (dry run) MergeRequest (commit)
│ │
▼ ▼
[merge-conflict-checker] [merger]
│ │
merge-conflict-check ctrl merge ctrl
(read-only) (apply + commit)
│ │
MergeResult MergeResult
│ │
▼ ▼
[merge-conflict-checker-signal] [merger-signal]
│ │
▼ ▼
┌──────────┬───────────────────────────┬──────────────┐
│ merge-conflict-check- merge-signal ctrl │
│ signal ctrl (update batch state, │
│ (update request fan out to conclude) │
│ mergeability) │
│ submitqueue orchestrator │
└─────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────┐
│ submitqueue orchestrator │
└──────────┬───────────────────────────┬──────────────┘
│ │
MergeRequest (dry run) MergeRequest (commit)
│ │
▼ ▼
[merge-conflict-check] [runway-merge]
│ │
merge-conflict-check ctrl merge ctrl
(read-only) (apply + commit)
│ │
MergeResult MergeResult
│ │
▼ ▼
[merge-conflict-check-signal] [merge-signal]
│ │
▼ ▼
┌──────────┬───────────────────────────┬──────────────┐
│ merge-conflict-check- merge-signal ctrl │
│ signal ctrl (update batch state, │
│ (update request fan out to conclude) │
│ mergeability) │
│ submitqueue orchestrator │
└─────────────────────────────────────────────────────┘
```

## Per-controller summary

| Controller | In | Out | One-line role |
|---|---|---|---|
| **merge-conflict-check** | MergeRequest | MergeResult -> merge-conflict-checker-signal | Dry-run merge: check mergeability of ordered steps against the target branch (read-only) |
| **merge** | MergeRequest | MergeResult -> merger-signal | Apply, commit, and report per-step output IDs |
| **merge-conflict-check** | MergeRequest | MergeResult -> merge-conflict-check-signal | Dry-run merge: check mergeability of ordered steps against the target branch (read-only) |
| **merge** | MergeRequest | MergeResult -> merge-signal | Apply, commit, and report per-step output IDs |

The merge-conflict-check controller always publishes a result — even when all steps are mergeable — so SubmitQueue receives a definitive answer. On infrastructure error it nacks for retry.

Expand Down Expand Up @@ -83,8 +83,8 @@ Runway has no persistent state — no request store, no job store, no database.

### Runway

Runway is a single service. It subscribes to two inbound topics (`merge-conflict-checker`, `merger`), performs VCS operations through a pluggable extension, and publishes results to two outbound topics (`merge-conflict-checker-signal`, `merger-signal`). It owns no persistent data.
Runway is a single service. It subscribes to two inbound topics (`merge-conflict-check`, `runway-merge`), performs VCS operations through a pluggable extension, and publishes results to two outbound topics (`merge-conflict-check-signal`, `merge-signal`). It owns no persistent data.

### Shared: the messaging queue

Runway communicates with SubmitQueue only through the messaging queue. The inbound topics are owned by runway; the outbound topics are owned by SubmitQueue.
Runway communicates with SubmitQueue only through the messaging queue. The contract is owned by Runway and published under `api/runway/messagequeue/`; both inbound and outbound topic keys live there. SubmitQueue publishes `MergeRequest` messages and consumes the `MergeResult` signals.
Loading
Loading