Relay
A desktop agent that fills browser forms while keeping review and final submission under explicit human control.
- Intended user
- People handling repetitive browser forms who want automation without handing an agent irreversible submission authority.
- Current status
- Deployed prototype
- The desktop application and backend were deployed, but current backend availability has not been verified. It has no users or sales.
Ownership
I originated and built this project from scratch, from the product idea and architecture through implementation. I use Claude Code and Codex as implementation tools; no human collaborator directed the work.
Architecture
A public-safe view of the system boundaries and the paths between them.
- Component 01Control
Person
Selects the form and source documents, observes progress, reviews the result, and submits manually.
- Component 02Interface
Electron desktop app
Supervises the local backend and presents run status, warnings, recovery, and review state.
- Component 03Service
Local NestJS service
Coordinates jobs, policy, documents, agent sessions, browser execution, and streaming updates.
- Component 04Data
SQLite and Prisma
Persist settings, uploaded-document metadata, form-fill runs, jobs, attempts, events, and review state.
- Component 05Agent
Planner session
Builds page-local actions from source evidence, observed controls, and durable run context.
- Component 06Control
Deterministic policy boundary
Owns allowed-domain scope, blocked action classes, final-submit exclusion, and bounded execution rules.
- Component 07Interface
Visible browser runtime
Executes allowed actions in an observable browser session and exposes the resulting page state.
- Component 08Control
Deterministic verification
Reads control state after actions and records mismatches instead of trusting the planned sequence alone.
- Component 09Agent
Bounded reviewer
Checks compact page and run evidence, requests limited repair when appropriate, and then stops.
- Component 10Output
Human handoff
Surfaces unresolved items and returns the still-visible form without triggering final submission.
Data and control paths
- PersonElectron desktop app
- Electron desktop appLocal NestJS service
- Local NestJS serviceSQLite and Prisma
- Local NestJS servicePlanner session
- Planner sessionDeterministic policy boundary
- Deterministic policy boundaryVisible browser runtime
- Visible browser runtimeDeterministic verification
- Deterministic verificationBounded reviewer
- Bounded reviewerDeterministic policy boundary
- Bounded reviewerHuman handoff
- Human handoffPerson
Evidence boundary
What this case study establishes, and what it deliberately does not claim.
Verified evidence
Claims supported by inspected artifacts, repeatable checks, or the project's current state.
Desktop and local-service boundary
Electron supervises a local NestJS backend; the React renderer uses loopback HTTP and SSE through a narrow desktop bridge.
Basis: Repository architecture, desktop runtime, renderer API, and service event implementation reviewed on 2026-08-03.
Durable run and recovery state
SQLite and Prisma models plus a durable job runner cover form-fill runs, attempts, events, retries, and review state.
Basis: Persistence schema, accepted architecture decisions, job runner, and recovery tests reviewed on 2026-08-03.
Planner, verification, reviewer, and handoff
The workflow uses separate planner and reviewer sessions, deterministic post-action checks, bounded repair, and an explicit human handoff before submission.
Basis: Accepted agent-workflow decisions, prompts, runtime modules, and tests reviewed on 2026-08-03.
Sensitive-action and diagnostics boundaries
The documented v1 excludes credential entry, authentication challenges, security answers, browser file uploads, and final submission; diagnostics have a supported sanitization layer.
Basis: Policy, handoff, secure-storage, diagnostics, and support-redaction implementation reviewed on 2026-08-03.
Known limitations
What has not been established, validated with users, or measured in production.
Private implementation
Repository evidence can verify the implemented surfaces, but the source is not linked or reproduced publicly.
Availability not re-verified
Past deployment work does not establish that the backend, installer, or complete flow is currently available.
No real-world outcome evidence
No user, traffic, completion, time-saved, or commercial metrics are presented.
Synthetic benchmark scope
The published benchmark checks a small public reference policy over handcrafted cases. It does not execute the private Relay runtime or an LLM.
Engineering decisions
The design choices, reasoning, and tradeoffs that shaped the system.
- 01
Make final submission a product boundary
Relay fills and verifies supported fields, then stops with the form visible for human review and manual submission.
Why: An irreversible action should not depend on an instruction embedded only in a model prompt.
Tradeoff: The workflow is intentionally less autonomous and cannot promise unattended end-to-end completion.
- 02
Keep browser execution visible
Use a visible browser subsystem and check observed control state after actions.
Why: A planned click or fill is not proof that the page accepted the intended value. Visibility and post-action checks make failures inspectable.
Tradeoff: Visible execution is less background-friendly and dynamic web controls still require conservative fallbacks.
- 03
Separate planning from review
Use a stateful planner for page work and a separate reviewer over compact evidence, with a bounded repair loop.
Why: Independent review reduces self-confirmation while a strict retry budget prevents open-ended agent loops.
Tradeoff: The extra model pass adds latency and cost, so context is compact and the number of review turns is capped.
- 04
Treat persisted state as authoritative
Persist runs, jobs, attempts, step events, and review state in SQLite through Prisma instead of relying on process memory.
Why: The Electron-supervised backend can restart without making completed work or unresolved handoff state disappear.
Tradeoff: Durability adds schema and recovery complexity to a local desktop product.
- 05
Keep safety scope server-owned
Derive allowed-domain scope and enforce blocked action classes in the local service rather than accepting them from a renderer or planner request.
Why: The component that executes browser actions should own the policy it must enforce.
Tradeoff: Cross-domain redirects and unusual form flows are more likely to stop for a person instead of being handled automatically.
Reliability & safety
Controls that make failures visible, bounded, and recoverable.
- Failure handling
Restart-aware local jobs
Electron supervises the backend lifecycle, while database-backed run and job records support recovery and keep prior attempts visible after a process restart.
- Safety boundary
Verify effects, not intentions
The runtime reads browser control state after actions and carries verification findings into the reviewer and human handoff.
- Failure handling
Bounded review and repair
Reviewer turns, navigation fallbacks, and repair attempts are capped so an uncertain page ends in an explicit handoff instead of an infinite loop.
- Observability
Durable, streamed progress
Compact step events are persisted and streamed to the desktop over SSE, giving the live interface and restart recovery a shared source of truth.
- Privacy
Sanitized diagnostics
The supported per-run diagnostics surface is structured and sanitized, with user-facing errors separated from raw local debugging detail.
- Testing
Synthetic form lab and automated tests
Standalone local forms and synthetic documents exercise realistic multi-field scenarios without requiring personal or customer data. Unit and end-to-end suites cover key desktop, service, policy, persistence, and browser-workflow boundaries.
Synthetic evaluation
A reproducible, non-production signal with its methodology and limits attached.
Synthetic human-control boundary check
A deterministic public reference classifier was run against 24 handcrafted synthetic form actions. All 24 matched the fixture's expected boundary, including 12 ordinary or near-miss actions marked automatable and 12 credential, challenge, upload, or finalization actions marked human-required.
Evaluation set: 24 handcrafted synthetic form-action fixtures
- Fixture conformance
- 24 / 24
- Human-required recall
- 12 / 12
- Automatable-action specificity
- 12 / 12
- Fixture version
- v1
All handcrafted expectations passed for this public reference classifier.
Every synthetic credential, authentication challenge, knowledge factor, file-transfer, and final-submit case was routed to a person.
Every synthetic ordinary or intentional near-miss action remained automatable in the reference classifier.
The result is bound to fixture SHA-256 a7fa597a47d3d61295b459dffd0e571fabadab996a15567a922850d90816ce9c.
Methodology
- 01Create synthetic action records with only a control kind, field type, and label. No forms, documents, browser sessions, or private runtime data are used.
- 02Label each fixture as automatable or human-required based on the documented public safety boundary.
- 03Run the public deterministic classifier and compare its output with every expected disposition.
- 04Write a result file containing totals, category counts, failed case IDs, and a SHA-256 hash of the exact fixture file.
Interpretation limits
- This is a policy-conformance check for public case-study code, not a benchmark of Relay's private planner, reviewer, browser execution, or verification runtime.
- The 24 fixtures are handcrafted and intentionally narrow. Passing them does not establish coverage of arbitrary websites or adversarial control labels.
- The result is not a completion-rate, model-accuracy, latency, time-saved, safety, reliability, user, or production metric.
Current state
- State 01The desktop application and backend were deployed, but the backend's current availability still needs verification.
- State 02There are currently no users, sales, or validated adoption metrics.
- State 03The repository remains private; no unverified download or live backend link is published here.
Demo & public material
Only reviewed, public-safe material is linked from this page.
- Public materialPublic
Synthetic policy fixtures
Twenty-four handcrafted form-action cases with expected automatable or human-required dispositions.
View public material - Public materialPublic
Deterministic reference evaluator
A small Node.js script that reproduces the published policy-conformance result without dependencies.
View public material - Public materialPublic
Reproducible evaluation result
Machine-readable totals, category counts, failed case IDs, and the exact fixture hash.
View public material - Public materialNot public
Private source repository
Source code is not published. Public materials are limited to disclosure-reviewed architecture, explanations, and synthetic artifacts.
- Public materialPlanned
Synthetic walkthrough
A recorded end-to-end demo should be published only after every frame and diagnostic artifact passes disclosure review.
Next steps
- Next 01Verify the backend and packaged desktop flow from a clean machine.
- Next 02Complete signing, notarization, and release checks for public desktop distribution.
- Next 03Test with a small set of real form workflows while preserving the manual submission boundary.