Most automation projects start by recording the operator's movements. It's easy to watch and easy to demo: open a dashboard, click a filter, copy a value, update a ticket, send a message.
It's also usually the least valuable part of the job.
Nobody's paying that operator to click “review.” They're paying them to look at a few signals, apply policy, decide whether the case is ordinary or an exception, and choose what happens next. A script that reproduces the clicks makes the operator faster. A system that owns the bounded decision removes the queue. That difference changes the architecture.
Start with a decision inventory
Before I pick a model or a framework, I write down the decisions in the existing workflow. For each one, I record:
Field | Question |
|---|---|
Trigger | What makes this case eligible for processing? |
Context | Which records, messages, pages, or files does the operator inspect? |
Policy | Which outcomes are allowed, forbidden, or always escalated? |
Action | Which side effects follow from each outcome? |
Evidence | What would let another person verify the decision? |
Recovery | What happens when context is missing or a tool fails? |
This separates two kinds of work. Mechanical work can become a tool. Judgment can become a decision contract. Anything that fits neither form isn't ready for autonomous execution yet.
The decision loop
The model sits inside a controlled loop. It doesn't own the workflow.
flowchart LR
Trigger["Workflow trigger"] --> Load["Load case state"]
Load --> Retrieve["Retrieve relevant context"]
Retrieve --> Policy["Apply policy and permissions"]
Policy --> Reason["Model proposes decision"]
Reason --> Validate["Validate typed decision"]
Validate --> Tools["Execute narrow tools"]
Tools --> Evidence["Persist evidence and outcome"]
Evidence --> Review["Complete or escalate"]
Review --> Memory["Evaluation data"]The order matters. Retrieval before reasoning. Policy before side effects. Validation after the model and before tools. Evidence persisted as part of the action, not assembled later from logs.
A decision is not a paragraph
Free-form text is a bad interface between an agent and the rest of a product. The output should describe the decision the system is allowed to make, not just explain what the model thinks.
type EvidenceRef = {
kind: "record" | "screenshot" | "log" | "source";
uri: string;
observedAt: string;
};
type CaseDecision =
| {
kind: "act";
action: "route_case" | "create_ticket" | "mark_passed";
confidence: "high" | "medium";
evidence: EvidenceRef[];
}
| {
kind: "needs_review";
reason: string;
missingContext: string[];
evidence: EvidenceRef[];
}
| {
kind: "refuse";
reason: string;
};This contract buys three things:
- The model can't invent an arbitrary side effect.
- Ambiguity becomes a valid outcome instead of an error hidden behind confident prose.
- A consequential decision carries an explicit evidence requirement.
The application still validates permissions, state, idempotency, and rate limits. A valid JSON object isn't permission to mutate the system.
Tools should expose capabilities, not internals
A tool should be narrow enough that its preconditions are understandable. “Call the CRM API” isn't a useful boundary. “Create a review case with these fields” is closer.
type Tool =
| {
name: "create_review_case";
input: {
accountId: string;
reason: string;
evidenceRefs: string[];
};
}
| {
name: "send_internal_notification";
input: {
channel: string;
message: string;
caseId: string;
};
};The tool implementation owns authorization and idempotency. The model can request an operation; it can't decide whether the current state permits it.
Autonomy needs a boundary
Not every decision should be automated. A practical rollout has at least three lanes:
- Automatic: high-confidence decisions with reversible side effects and complete evidence.
- Suggested: a proposed decision a person approves before the side effect.
- Escalated: missing context, policy ambiguity, sensitive communication, or an irreversible action.
The threshold is a product decision, not a model parameter. A system that's 95% accurate can still be unacceptable if the 5% of errors are expensive or hard to spot.
SuperSquad as an example
SuperSquad applies this to browser QA. A team describes the behavior that matters. The system turns that intent into coverage, runs browser journeys, captures artifacts, and produces a finding only when it can supply reproducible steps and linked evidence.
The browser click is a tool call. The valuable decision is whether the observed behavior violates the product intent, whether the evidence is sufficient, and whether the finding is confirmed or should be escalated. That's why a screenshot alone isn't enough — a useful finding needs the journey, the expected behavior, the actual behavior, the reproduction steps, and enough browser evidence for an engineer to verify it.
What to measure
Model accuracy isn't enough. Measure the workflow:
Metric | What it reveals |
|---|---|
Auto-action precision | Whether the system is safe to leave unattended. |
Escalation rate | How often context or policy is insufficient. |
Evidence completeness | Whether decisions can be audited. |
Tool failure rate | Whether external systems are limiting reliability. |
Time to resolution | Whether the system actually removes operational load. |
Override rate | Where policy or retrieval needs work. |
Every automatic action should become evaluation data. If outcomes aren't recorded, the system can't improve past prompt tweaks and anecdotes.
The engineering conclusion
Automation becomes valuable when it owns a bounded decision, not when it animates a macro. The design work is making the boundary explicit: retrieve the right context, enforce policy, expose narrow tools, validate typed output, keep evidence, and provide a safe way to refuse. The model supplies the reasoning; the product keeps the authority.