skip to main content
Atharv Kulkarni
← all projects

independent project · 02Human-controlled browser agent

Relay

A desktop agent that fills browser forms while keeping review and final submission under explicit human control.

intended user
People handling repetitive browser forms who want automation without handing an agent irreversible submission authority.
current status
Deployed prototype. The desktop application and backend were deployed, but current backend availability has not been verified. It has no users or sales.
ElectronReactViteNestJSSQLitePrismaSSEOpenAI Agents SDK

01

Ownership

I originated and built this project from scratch, from the product idea and architecture through implementation. I use Claude Code and Codex as implementation tools; no human collaborator directed the work.

02

Architecture

A public-safe view of the system boundaries and the paths between them.

The agent runs as a visible, restart-aware desktop workflow. The local service owns policy and durable state; model output proposes work but does not own the browser trust boundary or final submission.
  1. control

    Person

    Selects the form and source documents, observes progress, reviews the result, and submits manually.

  2. interface

    Electron desktop app

    Supervises the local backend and presents run status, warnings, recovery, and review state.

  3. service

    Local NestJS service

    Coordinates jobs, policy, documents, agent sessions, browser execution, and streaming updates.

  4. data

    SQLite and Prisma

    Persist settings, uploaded-document metadata, form-fill runs, jobs, attempts, events, and review state.

  5. agent

    Planner session

    Builds page-local actions from source evidence, observed controls, and durable run context.

  6. control

    Deterministic policy boundary

    Owns allowed-domain scope, blocked action classes, final-submit exclusion, and bounded execution rules.

  7. interface

    Visible browser runtime

    Executes allowed actions in an observable browser session and exposes the resulting page state.

  8. control

    Deterministic verification

    Reads control state after actions and records mismatches instead of trusting the planned sequence alone.

  9. agent

    Bounded reviewer

    Checks compact page and run evidence, requests limited repair when appropriate, and then stops.

  10. output

    Human handoff

    Surfaces unresolved items and returns the still-visible form without triggering final submission.

  • Person → Electron desktop app · starts and monitors a run
  • Electron desktop app → Local NestJS service · localhost HTTP and SSE
  • Local NestJS service → SQLite and Prisma · commits authoritative state
  • Local NestJS service → Planner session · provides compact run evidence
  • Planner session → Deterministic policy boundary · proposes actions
  • Deterministic policy boundary → Visible browser runtime · permits bounded actions
  • Visible browser runtime → Deterministic verification · returns observed DOM state
  • Deterministic verification → Bounded reviewer · provides findings
  • Bounded reviewer → Deterministic policy boundary · requests bounded repair
  • Bounded reviewer → Human handoff · stops after the review budget
  • Human handoff → Person · review and manual submission

03

Evidence boundary

What this case study establishes, and what it deliberately does not claim.

verified

  • Desktop and local-service boundary

    Electron supervises a local NestJS backend; the React renderer uses loopback HTTP and SSE through a narrow desktop bridge.

    basis · Repository architecture, desktop runtime, renderer API, and service event implementation reviewed on 2026-08-03.

  • Durable run and recovery state

    SQLite and Prisma models plus a durable job runner cover form-fill runs, attempts, events, retries, and review state.

    basis · Persistence schema, accepted architecture decisions, job runner, and recovery tests reviewed on 2026-08-03.

  • Planner, verification, reviewer, and handoff

    The workflow uses separate planner and reviewer sessions, deterministic post-action checks, bounded repair, and an explicit human handoff before submission.

    basis · Accepted agent-workflow decisions, prompts, runtime modules, and tests reviewed on 2026-08-03.

  • Sensitive-action and diagnostics boundaries

    The documented v1 excludes credential entry, authentication challenges, security answers, browser file uploads, and final submission; diagnostics have a supported sanitization layer.

    basis · Policy, handoff, secure-storage, diagnostics, and support-redaction implementation reviewed on 2026-08-03.

not claimed

  • Private implementation

    Repository evidence can verify the implemented surfaces, but the source is not linked or reproduced publicly.

  • Availability not re-verified

    Past deployment work does not establish that the backend, installer, or complete flow is currently available.

  • No real-world outcome evidence

    No user, traffic, completion, time-saved, or commercial metrics are presented.

  • Synthetic benchmark scope

    The published benchmark checks a small public reference policy over handcrafted cases. It does not execute the private Relay runtime or an LLM.

04

Engineering decisions

The design choices, reasoning, and tradeoffs that shaped the system.

  1. Make final submission a product boundary

    Relay fills and verifies supported fields, then stops with the form visible for human review and manual submission.

    rationaleAn irreversible action should not depend on an instruction embedded only in a model prompt.

    tradeoffThe workflow is intentionally less autonomous and cannot promise unattended end-to-end completion.

  2. Keep browser execution visible

    Use a visible browser subsystem and check observed control state after actions.

    rationaleA planned click or fill is not proof that the page accepted the intended value. Visibility and post-action checks make failures inspectable.

    tradeoffVisible execution is less background-friendly and dynamic web controls still require conservative fallbacks.

  3. Separate planning from review

    Use a stateful planner for page work and a separate reviewer over compact evidence, with a bounded repair loop.

    rationaleIndependent review reduces self-confirmation while a strict retry budget prevents open-ended agent loops.

    tradeoffThe extra model pass adds latency and cost, so context is compact and the number of review turns is capped.

  4. Treat persisted state as authoritative

    Persist runs, jobs, attempts, step events, and review state in SQLite through Prisma instead of relying on process memory.

    rationaleThe Electron-supervised backend can restart without making completed work or unresolved handoff state disappear.

    tradeoffDurability adds schema and recovery complexity to a local desktop product.

  5. Keep safety scope server-owned

    Derive allowed-domain scope and enforce blocked action classes in the local service rather than accepting them from a renderer or planner request.

    rationaleThe component that executes browser actions should own the policy it must enforce.

    tradeoffCross-domain redirects and unusual form flows are more likely to stop for a person instead of being handled automatically.

05

Reliability & safety

Controls that make failures visible, bounded, and recoverable.

  • Failure handling

    Restart-aware local jobs

    Electron supervises the backend lifecycle, while database-backed run and job records support recovery and keep prior attempts visible after a process restart.

  • Safety boundary

    Verify effects, not intentions

    The runtime reads browser control state after actions and carries verification findings into the reviewer and human handoff.

  • Failure handling

    Bounded review and repair

    Reviewer turns, navigation fallbacks, and repair attempts are capped so an uncertain page ends in an explicit handoff instead of an infinite loop.

  • Observability

    Durable, streamed progress

    Compact step events are persisted and streamed to the desktop over SSE, giving the live interface and restart recovery a shared source of truth.

  • Privacy

    Sanitized diagnostics

    The supported per-run diagnostics surface is structured and sanitized, with user-facing errors separated from raw local debugging detail.

  • Testing

    Synthetic form lab and automated tests

    Standalone local forms and synthetic documents exercise realistic multi-field scenarios without requiring personal or customer data. Unit and end-to-end suites cover key desktop, service, policy, persistence, and browser-workflow boundaries.

06

Synthetic evaluation

A deterministic public reference classifier was run against 24 handcrafted synthetic form actions. All 24 matched the fixture's expected boundary, including 12 ordinary or near-miss actions marked automatable and 12 credential, challenge, upload, or finalization actions marked human-required.

Synthetic human-control boundary checksynthetic · not production or customer data

Fixture conformance
24 / 24
All handcrafted expectations passed for this public reference classifier.
Human-required recall
12 / 12
Every synthetic credential, authentication challenge, knowledge factor, file-transfer, and final-submit case was routed to a person.
Automatable-action specificity
12 / 12
Every synthetic ordinary or intentional near-miss action remained automatable in the reference classifier.
Fixture version
v1
The result is bound to fixture SHA-256 a7fa597a47d3d61295b459dffd0e571fabadab996a15567a922850d90816ce9c.

set · 24 handcrafted synthetic form-action fixtures

method · Create synthetic action records with only a control kind, field type, and label. No forms, documents, browser sessions, or private runtime data are used. Label each fixture as automatable or human-required based on the documented public safety boundary. Run the public deterministic classifier and compare its output with every expected disposition. Write a result file containing totals, category counts, failed case IDs, and a SHA-256 hash of the exact fixture file.

limits · This is a policy-conformance check for public case-study code, not a benchmark of Relay's private planner, reviewer, browser execution, or verification runtime. The 24 fixtures are handcrafted and intentionally narrow. Passing them does not establish coverage of arbitrary websites or adversarial control labels. The result is not a completion-rate, model-accuracy, latency, time-saved, safety, reliability, user, or production metric.

07 – 09

State, material, next

current state

  • The desktop application and backend were deployed, but the backend's current availability still needs verification.
  • There are currently no users, sales, or validated adoption metrics.
  • The repository remains private; no unverified download or live backend link is published here.

demo & public material

  • Synthetic policy fixtures ↗available

    Twenty-four handcrafted form-action cases with expected automatable or human-required dispositions.

  • Deterministic reference evaluator ↗available

    A small Node.js script that reproduces the published policy-conformance result without dependencies.

  • Reproducible evaluation result ↗available

    Machine-readable totals, category counts, failed case IDs, and the exact fixture hash.

  • Private source repositorynot public

    Source code is not published. Public materials are limited to disclosure-reviewed architecture, explanations, and synthetic artifacts.

  • Synthetic walkthroughplanned

    A recorded end-to-end demo should be published only after every frame and diagnostic artifact passes disclosure review.

next steps

  1. 01Verify the backend and packaged desktop flow from a clean machine.
  2. 02Complete signing, notarization, and release checks for public desktop distribution.
  3. 03Test with a small set of real form workflows while preserving the manual submission boundary.
next project · AI Guides Factory →