A frontier model once told me to take the production server down. I'd asked how to check whether our traffic was really being proxied through Cloudflare, and its plan was: shut off the origin and see if the site throws a Cloudflare error page. Confident, fluent, and completely unusable — it had no idea it was proposing an outage to satisfy a curiosity.
That story is the whole point. The model is the most visible part of an AI product and rarely the part that decides whether it works. It produces a proposal. Everything around it decides what that proposal is allowed to mean.
A production system has to answer questions the model can't answer by itself:
- Which information is authoritative for this request?
- Which tools is this user allowed to invoke?
- What state must survive between turns or runs?
- What shape must the result have for downstream code?
- Which observations are enough to justify a side effect?
- How do we evaluate a bad result after it has already reached a user?
Those are system-design questions. The model is one dependency inside the answers.
Retrieval is a data contract
“Chat with your documentation” isn't a model feature. It's a data pipeline:
flowchart LR
Sources["Owned sources"] --> Parse["Parse and normalize"]
Parse --> Chunk["Chunk with metadata"]
Chunk --> Embed["Generate embeddings"]
Embed --> Index["Vector and keyword index"]
Question["User question"] --> Query["Query and rerank"]
Index --> Query
Query --> Context["Context pack"]
Context --> Model["Model"]
Model --> Answer["Answer with source references"]The retrieval layer decides what the model is even allowed to know for a request. That makes chunk boundaries, source identifiers, freshness, access control, and ranking part of the product contract.
The Knowledge-Base project was a small version of this shape. It crawled webpages and sitemaps, chunked the content, stored embeddings in Postgres with pgvector, and retrieved context for questions asked in Slack. The model was the last step, turning retrieved material into an answer. The real correctness question was whether the right material had reached it at all. When retrieval returns the wrong context, a stronger model just produces a more convincing wrong answer.
Tool calls are capability boundaries
Tools should represent business capabilities, not raw service clients.
type ToolCall =
| {
name: "create_issue";
input: {
title: string;
reproductionSteps: string[];
evidenceRefs: string[];
};
}
| {
name: "sync_standup_summary";
input: {
channelId: string;
answeredUserIds: string[];
missingUserIds: string[];
summary: string;
};
};The application still has to verify the caller, the current state, authorization, the idempotency key, and the rate limit. Tool selection is a model decision; tool execution is an application decision.
This is exactly what keeps the server-takedown scenario from ever running. A model that receives a broad API client gradually becomes responsible for permissions, state transitions, and validation. That isn't autonomy — it's an unreviewed control plane, and it will eventually propose something confident and catastrophic.
Structured output is an interface
If another system consumes an answer, prose isn't an interface. A QA finding, for example, should carry what's needed to reproduce and triage it:
type Finding = {
kind: "finding" | "needs_review" | "no_issue";
severity: "low" | "medium" | "high" | "critical" | null;
journey: string;
expectedBehavior: string;
actualBehavior: string | null;
reproductionSteps: string[];
evidenceRefs: string[];
};The schema doesn't make the answer true. It makes the answer testable. The service can reject a finding without reproduction steps, require evidence for a non-null severity, and stop downstream code from parsing narrative text with regular expressions.
Memory is state, not conversation history
Conversation history is only one kind of memory. A production workflow usually needs explicit state:
type RunState =
| { kind: "queued"; runId: string }
| { kind: "running"; runId: string; step: string }
| { kind: "needs_review"; runId: string; reason: string }
| { kind: "completed"; runId: string; resultRef: string }
| { kind: "failed"; runId: string; errorRef: string };The state has an owner, valid transitions, retry semantics, and an audit trail. A model can propose the next step; it shouldn't be allowed to invent a transition the domain doesn't support.
This is the same issue behind my merged LangChain.js contribution: when composite chains lose the memory owned by their sub-chains, each component looks correct while the composed workflow loses context. Composition boundaries need explicit memory semantics.
Evidence closes the trust gap
An AI result becomes operationally useful when another person can inspect why it happened. Depending on the workflow, that might mean:
- retrieved source identifiers and versions
- tool inputs and outputs
- screenshots or recordings
- console and network traces
- state transitions
- model and policy versions
- human overrides and final outcomes
Emit evidence during execution. Reconstructing it after the result is disputed is unreliable and expensive.
Model choice comes after the contract
Once the surrounding contract exists, picking a model becomes an engineering trade-off:
Workload | Primary optimization |
|---|---|
Classification | Calibration and false-positive cost |
Retrieval answer | Recall, ranking, freshness, and citations |
Tool execution | Schema adherence and safe refusal |
High-stakes reasoning | Evidence quality and reviewability |
Summarization | Reader usefulness and factual omission rate |
Latency and cost matter, but measure them against a defined task and an accepted error budget. “The model sounded good” isn't an evaluation.
The practical conclusion
The model matters, but it isn't the product boundary. Retrieval determines context. Tools determine authority. State determines continuity. Schemas determine interoperability. Evidence determines whether the result can be trusted.
The model produces a proposal. The surrounding system decides what that proposal means — and, occasionally, stops it from taking down production to prove a point.