Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/concepts/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,14 @@ element usually exposes `Name` / `Value` text **even when it has no stable
`AutomationId`**, so UIA-based identity is viable on most native apps even where
a durable selector is not.

Native authoring uses `openadapt-capture` rather than a second recorder inside
Flow. Capture retains the UIA element observed at each demonstrated action, and
the compiler walks that evidence to the nearest actionable node before storing
the structural target. Window-scoped recordings convert the observation into
the captured window's coordinate space before compilation. RDP and Citrix
recordings intentionally omit local client UIA: the remote application remains
an externally observed pixel surface.

Desktop (and Citrix/VDI) is the **differentiated wedge**: the work that has no
web UI and no usable API, where a computer-use agent is the only alternative and
a wrong write is expensive. Structured desktop text distinguishes an `O` from a
Expand Down Expand Up @@ -130,6 +138,13 @@ its visual floor and the identity gate falls back to its pixel/OCR tiers, which
is why a look-alike identifier can force a [halt rather than a verify](identity-gate.md)
there.

For every consequential remote action, the runtime uses a two-phase actuation
lease. It captures a fresh frame, re-resolves the target, and rechecks identity
on that frame. The backend then captures again under its input lock and refuses
before the first input edge if pixels, dimensions, session readiness, or the
leased context changed. A potentially stale coordinate is never delivered
merely because the remote connection is still alive.

!!! info "RDP qualification evidence"
On one Parallels Windows 11 VM at 1280x800 with Aardwolf 0.2.14, candidate
`82a658a` completed 3/3 trials that created a unique file through the Windows
Expand Down Expand Up @@ -166,6 +181,12 @@ resume. It is pixel-only by construction, so the same visual resolution,
identity, effect, policy, and halt contracts run without pretending a DOM or
accessibility tree exists.

Citrix uses the same two-phase remote actuation contract as RDP. Immediately
before a consequential action, OpenAdapt reacquires the exact Workspace window,
focus, geometry, readiness, fresh pixels, resolved target, and record identity.
The first input edge is refused if any of that evidence changes after
resolution.

!!! info "Citrix qualification evidence"
The accepted no-DOM qualification completed 3/3 healthy effects and 3/3
severe-drift safe-halts, with 0 model calls, silent incorrect successes,
Expand Down
11 changes: 8 additions & 3 deletions docs/concepts/demonstration-compiler.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,14 @@ openadapt flow replay bundle --url https://your.app # replay, local, $0
On the web substrate shown here, `record` opens a headed browser on your own app
and captures what you do. The same loop records native Windows, macOS, or Linux
applications and pixel-only RDP or Citrix sessions by choosing a
[backend](backends.md) instead of a `--url`. `compile` turns the recording into
a bundle. `replay` runs the bundle deterministically on the same substrate and
writes an illustrated report.
[backend](backends.md) instead of a `--url`. Those native and remote
demonstrations reuse `openadapt-capture` for screen, input, timing, and
window-scoped evidence. On native Windows, action-time UIA observations are
retained beside the captured event so compilation can keep the nearest
actionable structural target; RDP and Citrix deliberately suppress local
client UIA because accessibility metadata does not cross the remote boundary.
`compile` turns the recording into a bundle. `replay` runs the bundle
deterministically on the same substrate and writes an illustrated report.

## Vision-first behind a small backend

Expand Down
5 changes: 5 additions & 0 deletions docs/concepts/deployment-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,11 @@ risk work required for the actual deployment.
Every substrate is a first-class execution target and every workflow is
qualified in its real environment. The published qualification evidence to date:

Consequential RDP and Citrix actions share a two-phase remote contract: a fresh
frame is resolved and identity-checked, then a one-shot input lease refuses
changed session/window context, focus where applicable, geometry, readiness, or
pixels before delivery.

- **Browser:** Local engine and public managed substrate, exercised end to end
against a real third-party application in the published engine evidence.
- **Windows UIA:** The counted
Expand Down
7 changes: 7 additions & 0 deletions docs/concepts/substrate-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,13 @@ The released backends cover it behind the same protocol:
the exact-window remote-display backend. It binds the Workspace owner/title
and gates readiness before governed input.

Consequential RDP and Citrix input is two-phase. The runtime acquires a fresh
actuation frame, re-resolves the target, and rechecks record identity. The
backend then verifies the same session/window, focus where applicable,
geometry, readiness, and pixels immediately before the first input edge. A
change invalidates the one-shot lease and halts instead of reusing stale
coordinates.

!!! info "Citrix / RDP is pixel-first: the identity gate adapts to it"
On a pure-pixel substrate the ladder runs on its visual floor and the
identity gate uses its pixel/OCR tiers. When an identifier is genuinely
Expand Down
4 changes: 2 additions & 2 deletions docs/ecosystem/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ module inside it:
| [openadapt-cloud](https://github.com/OpenAdaptAI/openadapt-cloud) | **Supported** | Live control plane for the public managed-browser subscription: organizations, exact-hash admission, runner orchestration, reports, billing, and usage. |
| [openadapt-desktop](https://github.com/OpenAdaptAI/openadapt-desktop) | **Supported** | Public Beta `desktop-v0.10.0` provides Windows MSI/NSIS, macOS arm64/x64 DMG, and Linux AppImage/DEB installers. Every installer path was installed, launched, and uninstalled in the native release workflow; the release includes exact checksums, platform metadata, and build attestations. |
| [openadapt-agent](https://github.com/OpenAdaptAI/openadapt-agent) | **Supported** | Active v2 bridge that exposes governed Flow bundles to MCP clients and Agent Skills. The pre-v2 model-driven execution wrapper is the deprecated line; the repository itself is active. |
| [openadapt-capture](https://github.com/OpenAdaptAI/openadapt-capture) | **Supported** | Optional native mouse, keyboard, and screen recorder behind Flow's Windows and RDP recording path. Browser recording remains inside Flow's Playwright listener. |
| [openadapt-capture](https://github.com/OpenAdaptAI/openadapt-capture) | **Supported** | Canonical native screen, mouse, keyboard, timing, and window-scoped recorder behind Flow's Windows, macOS, Linux, RDP, and Citrix recording paths. Capture 1.1 retains Windows UIA evidence at action time; remote sessions remain externally black-box. Browser recording remains inside Flow's Playwright listener. |
| [openadapt-privacy](https://github.com/OpenAdaptAI/openadapt-privacy) | **Supported** | Optional PII/PHI scrubbing used on configured persist, log, and upload paths. |
| [openadapt-types](https://github.com/OpenAdaptAI/openadapt-types) | **Supported** | Shared interoperability schemas; contributor-facing, not an end-user product. |

Expand All @@ -58,7 +58,7 @@ flowchart LR
E[openadapt-evals<br/>benchmarks] -.measures.-> F
```

The compiler is the product. Capture can feed it demonstrations, the ML layer
The compiler is the product. Capture feeds it native and remote demonstrations, the ML layer
can supply optional on-prem models, Agent gives MCP clients and Agent Skills a
governed route into Flow bundles, and evals measures adjacent research. This page
intentionally does not expose internal developer tools as product components.
Expand Down
6 changes: 3 additions & 3 deletions docs/get-started/what-works-today.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,11 +48,11 @@ customer-controlled runtime connected to the same governance model.
| System-of-record effect verification | **Supported** | REST, FHIR, and document-hash verifiers run in the live replay path and halt on refuted or indeterminate declared effects. | The compiler does not infer effects. A deployment must author effects and configure the matching verifier; otherwise screen checks remain the oracle. |
| `teach` halt-to-correction loop | **Supported** | The deterministic reference inducer covers the optional-dialog correction class behind regression and canary gates. | Arbitrary UI corrections are not generally learned. Unsafe or underdetermined revisions are refused. |
| Workflow-program IR, branches, loops, and multi-trace induction | **Supported** | Implemented against synthetic fixtures, including refusal for underdetermined programs. | No real recording exercises the full Phase-2 program path yet. |
| Windows UIA backend | **Supported** | The counted `20260717-candidate-56759c8-v2` in-tree WinForms matrix completed 3/3 trials; an independent SQLite oracle confirmed 3/3 effects; stale-target and ambiguous-target controls each refused 3/3; there were 0 silent incorrect successes, 0 over-halts, and 0 model calls. [Review the immutable report at the merged Flow commit](https://github.com/OpenAdaptAI/openadapt-flow/blob/defafbae758a75c8e149d9693f2cffe1f2264b8c/benchmark/windows_uia/results.json) and [Flow PR #132](https://github.com/OpenAdaptAI/openadapt-flow/pull/132). | The report also preserves earlier rejected diagnostic runs; their failures are not counted acceptance trials. Each Windows workflow is qualified in its real environment; the published evidence covers the named workflow and Windows 11 ARM VM snapshot. |
| Windows UIA backend | **Supported** | The counted `20260717-candidate-56759c8-v2` in-tree WinForms matrix completed 3/3 trials; an independent SQLite oracle confirmed 3/3 effects; stale-target and ambiguous-target controls each refused 3/3; there were 0 silent incorrect successes, 0 over-halts, and 0 model calls. The native authoring handoff now retains Capture 1.1 action-time UIA observations and compiles the nearest actionable node, including window-scoped recordings. Review the [immutable qualification report](https://github.com/OpenAdaptAI/openadapt-flow/blob/defafbae758a75c8e149d9693f2cffe1f2264b8c/benchmark/windows_uia/results.json), [Flow PR #132](https://github.com/OpenAdaptAI/openadapt-flow/pull/132), and the [capture-to-compiler contract in Flow PR #239](https://github.com/OpenAdaptAI/openadapt-flow/pull/239). | The qualification report preserves earlier rejected diagnostic runs; their failures are not counted acceptance trials. The UIA handoff contract covers `recording.db -> CaptureSession -> convert -> compile`, not a new arbitrary-app reliability matrix. RDP and Citrix recordings suppress local client UIA and remain pixel-only. Each Windows workflow is qualified in its real environment; the published task evidence covers the named workflow and Windows 11 ARM VM snapshot. |
| Native macOS backend | **Supported** | On one macOS 15.7.3 arm64 host, candidate `b1b61a5` completed 3/3 exact-byte TextEdit trials and refused a two-window ambiguity without changing either file; there were 0 silent incorrect successes and 0 over-halts. [Review the hash-bound adjudication at immutable evidence commit `ca1b522`](https://github.com/OpenAdaptAI/openadapt-flow/blob/ca1b522cad215875f7471782283f8f8bb8e6c998/benchmark/macos_native/textedit_counted_3plus1_b1b61a5_20260717.adjudication.json) and [Flow PR #135](https://github.com/OpenAdaptAI/openadapt-flow/pull/135). | The counted candidate is `b1b61a5`; `ca1b522` preserves its reports and adjudication but is not the current PR head. The immutable original report remains `status: failed` because graceful-close cleanup warnings were classified as batch failure. The separate adjudication preserves that result, verifies the exact harness PIDs and temporary root were absent, and accepts the action-effect and ambiguity-refusal evidence. Each macOS workflow is qualified in its real environment. |
| Native Linux backend | **Supported** | Required current-main job [`linux-atspi-x11`](https://github.com/OpenAdaptAI/openadapt-flow/actions/runs/30059807758/job/89378981573) at exact Flow commit `3de5fc67acf3024a621f812c5a6ed9be07fac335` ran one fresh GTK3 process per trial on Ubuntu 24.04 X11/AT-SPI. It confirmed 3/3 exact-file effects, 3/3 ambiguous-target refusals, and 3/3 stale-target refusals, with 0 silent incorrect successes, 0 over-halts, 0 operator interventions, and 0 model calls. | Native receipts prove AT-SPI delivery only; independent exact file bytes or confirmed absence prove effects. This is bounded to the in-tree GTK3 fixture and CI Xvfb image; it does not establish Wayland or arbitrary third-party application support. |
| RDP backend | **Supported** | Two bounded results exercise complementary RDP paths. Aardwolf 0.2.14 over a Parallels Windows 11 VM completed 3/3 Windows Run-dialog file effects with independent guest-tools readback. The accepted full governed lifecycle at mechanism commit `6031fde` recorded, compiled, and replayed a synthetic note workflow through real FreeRDP3-transported pixels/input: 3/3 healthy effects and 3/3 drift safe-halts, with zero model calls, silent incorrect successes, false completions, drift writes, or healthy over-halts. Review the [Aardwolf/Windows report](https://github.com/OpenAdaptAI/openadapt-flow/blob/6610d24cebba27918b8ea507b2f05a094057ac85/benchmark/rdp/results_82a658a_20260718.sanitized.json), the [FreeRDP lifecycle report](https://github.com/OpenAdaptAI/openadapt-flow/blob/affedc5f1f0de533a0744deaa8e30a203c91c6b3/benchmark/rdp_ladder/results.json), [Flow PR #142](https://github.com/OpenAdaptAI/openadapt-flow/pull/142), and [Flow PR #177](https://github.com/OpenAdaptAI/openadapt-flow/pull/177). | The Windows batch qualifies its exact task, snapshot, transport, and oracle. The FreeRDP batch qualifies a synthetic Linux Tk task over a real RDP round trip; it is not Aardwolf, a Windows-app qualification, Citrix ICA/HDX, or WAN-captured drift. |
| Citrix / VDI backend | **Supported** | The dedicated `--backend citrix` path selects the exact Citrix Workspace/Viewer owner, optionally binds an exact title, requires a readiness marker for governed `run`, carries the closed target into durable resume, and uses the shared pixel identity, effect, policy, and halt contracts. The accepted no-DOM qualification completed 3/3 healthy effects and 3/3 severe-drift safe-halts with 0 model calls, silent incorrect successes, false completions, healthy over-halts, or drift writes. Review the [immutable report](https://github.com/OpenAdaptAI/openadapt-flow/blob/f6faac5b900b78cbda5980de0e983a9f987285ac/benchmark/citrix_workspace/results.json) and [Flow PR #183](https://github.com/OpenAdaptAI/openadapt-flow/pull/183). | The accepted artifact explicitly records `code_readiness_accepted: true` and `ica_hdx_accepted: false`. It qualifies the shipped Citrix Workspace-window backend contract over a no-DOM canvas stand-in, not a counted real ICA/HDX batch. The exact ICA/HDX client, codec, latency, DPI, lock/readiness, input, identity, and effect matrix is a separate per-deployment qualification boundary. |
| RDP backend | **Supported** | Two bounded results exercise complementary RDP paths. Aardwolf 0.2.14 over a Parallels Windows 11 VM completed 3/3 Windows Run-dialog file effects with independent guest-tools readback. The accepted full governed lifecycle at mechanism commit `6031fde` recorded, compiled, and replayed a synthetic note workflow through real FreeRDP3-transported pixels/input: 3/3 healthy effects and 3/3 drift safe-halts, with zero model calls, silent incorrect successes, false completions, drift writes, or healthy over-halts. Consequential remote actions now acquire a fresh frame, re-resolve target and identity, and use a one-shot input lease that refuses changed session context, pixels, dimensions, or readiness before delivery. Review the [Aardwolf/Windows report](https://github.com/OpenAdaptAI/openadapt-flow/blob/6610d24cebba27918b8ea507b2f05a094057ac85/benchmark/rdp/results_82a658a_20260718.sanitized.json), the [FreeRDP lifecycle report](https://github.com/OpenAdaptAI/openadapt-flow/blob/affedc5f1f0de533a0744deaa8e30a203c91c6b3/benchmark/rdp_ladder/results.json), [Flow PR #142](https://github.com/OpenAdaptAI/openadapt-flow/pull/142), [Flow PR #177](https://github.com/OpenAdaptAI/openadapt-flow/pull/177), and [Flow PR #238](https://github.com/OpenAdaptAI/openadapt-flow/pull/238). | The Windows batch qualifies its exact task, snapshot, transport, and oracle. The FreeRDP batch qualifies a synthetic Linux Tk task over a real RDP round trip; it is not Aardwolf, a Windows-app qualification, Citrix ICA/HDX, or WAN-captured drift. The two-phase mechanism is separately covered by runtime and backend refusal tests; it does not enlarge those bounded application results. |
| Citrix / VDI backend | **Supported** | The dedicated `--backend citrix` path selects the exact Citrix Workspace/Viewer owner, optionally binds an exact title, requires a readiness marker for governed `run`, carries the closed target into durable resume, and uses the shared pixel identity, effect, policy, and halt contracts. Consequential input reacquires the exact client window, focus, geometry, readiness, fresh pixels, resolved target, and record identity, then refuses any change before the first input edge. The accepted no-DOM qualification completed 3/3 healthy effects and 3/3 severe-drift safe-halts with 0 model calls, silent incorrect successes, false completions, healthy over-halts, or drift writes. Review the [immutable report](https://github.com/OpenAdaptAI/openadapt-flow/blob/f6faac5b900b78cbda5980de0e983a9f987285ac/benchmark/citrix_workspace/results.json), [Flow PR #183](https://github.com/OpenAdaptAI/openadapt-flow/pull/183), and [Flow PR #238](https://github.com/OpenAdaptAI/openadapt-flow/pull/238). | The accepted artifact explicitly records `code_readiness_accepted: true` and `ica_hdx_accepted: false`. It qualifies the shipped Citrix Workspace-window backend contract over a no-DOM canvas stand-in, not a counted real ICA/HDX batch. The exact ICA/HDX client, codec, latency, DPI, lock/readiness, input, identity, and effect matrix is a separate per-deployment qualification boundary. |
| Desktop authoring GUI and tray | **Supported / Beta** | Public `desktop-v0.10.0` provides Windows MSI/NSIS, macOS arm64/x64 DMG, and Linux AppImage/DEB installers. All six installer paths were installed, launched, and uninstalled in the native release workflow; the published set includes checksums and per-platform provenance metadata. | Builds are not yet fully code-signed/notarized, so verify `SHA256SUMS` and provenance before overriding the OS publisher warning. The app and CLI drive the same released compiler/runtime. |
| Hosted CLI connectivity | **Supported / public offer** | `login`, exact-hash artifact preparation/upload, one-time runtime validation, bound replacement activation, and `report-break` connect the local engine to the live control plane. | Upload requires destination policy and an approved sanitized derivative; checkout never bypasses an egress refusal. |
| Artifact sanitation and local review | **Supported / launch gate** | The sanitized-derivative pipeline inventories, transforms, rescans, manifests, hashes, and supports local review/approval. | The raw original remains sensitive; unknown or unresolved content is refused; runtime observations can reintroduce PHI/PII. |
Expand Down
Loading