Writing / AI in practice

QA agents need evidence, not vibes

Designing SuperSquad as an evidence-first QA system: organization-owned state, immutable run snapshots, isolated execution, and findings an engineer can actually reproduce.

Most AI QA demos quit at the least useful moment. The agent opens a browser, clicks around, decides something looks broken, and writes a confident paragraph about it.

I can't hand that to an engineer.

If a QA system is going to interrupt a release or make work for someone, it has to show the journey it attempted, what it expected, what it actually saw, the browser state, the failed requests or console errors, and the exact steps to reproduce. That's the single rule SuperSquad is built around:

No confirmed finding without reproducible steps and linked evidence.

I care about that invariant more than I care about the model. It drives the data model, the execution boundaries, the storage, and the report workflow.

Intent is the input; proof is the output

Traditional browser automation starts with instructions:

Click this selector.
Fill this field.
Submit the form.
Assert that this text exists.

That's useful when a team already knows the exact test and has capacity to maintain it. It's much less useful for exploratory coverage, fast-changing products, or teams that can describe the risk but don't want to hand-author selectors.

SuperSquad starts from product intent instead:

Audit signup for validation gaps.
Check checkout recovery after a failed payment.
Review onboarding for dead ends and contradictory calls to action.
Verify that the account settings journey preserves unsaved changes.

The system has to translate that intent into executable coverage, but natural language doesn't remove the need for deterministic controls. A run still needs an allowed domain set, action limits, credentials, timeouts, an evidence policy, and a refusal path.

flowchart LR
  Intent["Product intent"] --> Coverage["Coverage map"]
  Coverage --> Snapshot["Immutable run snapshot"]
  Snapshot --> Browser["Browser execution"]
  Browser --> Artifacts["Evidence artifacts"]
  Artifacts --> Candidate["Finding candidate"]
  Candidate --> Verify["Evidence and policy verification"]
  Verify --> Report["Report or needs review"]

The model can help interpret intent. It doesn't get to waive the verification stage.

QA is a domain model, not a chat transcript

Chat is fine for entering intent and reading an explanation. It's a poor primary data model for QA.

QA needs durable objects:

  • organizations and members
  • applications and environments
  • product profiles and coverage maps
  • schedules and entitlements
  • test runs and journeys
  • findings and evidence
  • reports, usage, and audit history

These need permissions, filtering, lifecycle state, and links. A conversation can explain a run, but it shouldn't be the only place the run exists. Natural language is the command surface; the product state is the system of record.

Split product state from execution state

There are two different ownership problems here:

  1. Who owns the organization's configuration and admission decisions?
  2. Who owns the live execution of one run?

SuperSquad keeps them apart. An organization state owner manages applications, environments, coverage, schedules, entitlements, concurrency, and accepted terminal results. A run owner executes one immutable snapshot and owns its browser session, retries, artifacts, candidate findings, and terminal proposal.

flowchart LR
  UI["Command center"] --> Org["Organization state owner"]
  Org --> Projection["D1 read models"]
  Org --> Run["Run execution owner"]
  Run --> Browser["Browser session"]
  Run --> Sandbox["Isolated probes"]
  Run --> Model["Model boundary"]
  Run --> R2["Evidence in R2"]
  Run --> Proposal["Terminal proposal"]
  Proposal --> Org

The organization owner shouldn't block synchronously on browser automation or model loops. The run owner shouldn't mutate organization configuration as a side effect of execution. A run proposes a result; the organization accepts it once and projects it to the dashboard. That separation is what makes retries and failures containable.

Immutable snapshots make reports explainable

A scheduled run should still be explainable after it finishes, and that's hard if it reads mutable configuration the whole way through.

The application can change while a run is active. Coverage intent gets edited. Allowed domains get tightened. A schedule gets disabled. If the run keeps reading live organization state, the final report is describing a moving target.

So at admission time, the organization owner takes a snapshot:

type RunSnapshot = {
  orgId: string;
  runId: string;
  application: { id: string; name: string };
  environment: { id: string; baseUrl: string };
  productDescription: string;
  coverageIntent: string[];
  allowedDomains: string[];
  blockedActions: string[];
};

The run executes that snapshot. The report can then say exactly what configuration was tested, even if the product moved on since the run started.

Evidence is a product object

Evidence shouldn't be a folder of screenshots stapled on after the model writes a finding. It should be part of the run and finding model.

type EvidenceType =
  | "screenshot"
  | "browser-recording"
  | "console-log"
  | "network-log"
  | "dom-snapshot"
  | "generated-test";

type FindingCandidate = {
  title: string;
  severity: "low" | "medium" | "high" | "critical";
  expectedBehavior: string;
  actualBehavior: string;
  reproductionSteps: string[];
  evidence: Array<{ type: EvidenceType; uri: string; summary: string }>;
};

The policy is stricter than the type:

Requirement

Why

Reproduction steps

An engineer needs a path, not an accusation.

Visual evidence

The team needs to inspect the observed state.

Execution evidence

Network, console, or recording data reduces guesswork.

Confidence state

Suspicion shouldn't be reported as confirmation.

An observation with no evidence can become needs_review. It doesn't get to become a confirmed defect because the model wrote a persuasive sentence.

Browser automation is a capability, not the product

Browser control is necessary, but it isn't the product. A run may also need model reasoning, deterministic policy checks, isolated code probes, artifact storage, retries, and report generation.

“AI that clicks” describes one tool call. It doesn't describe how the system decides what to click, what result was expected, whether the observation is a defect, or whether the evidence supports the claim.

The boundary matters for security, too. A run needs explicit limits around:

  • allowed domains and navigation targets
  • credentials and secret-bearing fields
  • destructive actions and real transactions
  • captchas, SSO, and email verification
  • browser and model budgets
  • sandboxed code or network probes

The run should fail closed when it hits a boundary it can't safely cross. Reporting green because an important journey never actually ran is worse than reporting an incomplete run.

What I actually measure

The useful metrics aren't model benchmark scores:

  • confirmed-finding precision
  • reviewer agreement with severity
  • share of findings with complete evidence
  • reproduction success rate
  • coverage executed versus coverage requested
  • runs that ended in an explicit blocked state
  • time from finding to engineer confirmation

The last two matter most. An autonomous run has to tell “the product passed” apart from “the system couldn't complete the test.”

The product is the proof loop

The model can describe a page. Browser tools can click buttons. A report can be generated from either. The hard part is trust: durable state, reproducible execution, immutable inputs, narrow capabilities, and evidence that survives the model's prose.

That's why SuperSquad is a stateful QA system and not a chatbot with browser access. The agent is a component. The proof loop is the product.

Bhavin Kamani

A small command line for a curious visitor.