Shipping AI Features in Production Apps: Cost, Latency, Evals and Guardrails

A demo takes an afternoon. A feature that thousands of people use every day, with a bill you can predict and a failure mode you can explain to a customer, is ordinary product engineering — and that is the gap most AI projects fall into. The model is rarely the hard part. Retrieval quality, latency budgets, cost control, evaluation and what happens when the answer is wrong are what decide whether the feature survives its first month.

Start by choosing a task rather than a technology. "Add AI to the app" produces a chat box nobody uses. "Summarise a support thread so an agent can reply in one minute instead of five" is a task with a baseline, an owner and a number attached to it. The best early candidates share a shape: a human reviews the output, the cost of being wrong is low, and the work is currently done by someone reading and re-typing.

Then be honest about where quality comes from. In most business applications the difference between a useful feature and a plausible-sounding one is retrieval, not reasoning — whether the right document, order or history reached the model at all. Teams reach for a bigger model when the actual defect is a search index that returns the wrong five chunks. Fix the input first; it is cheaper and it improves every model you might later switch to.

The engineering checklist we work through :

• A written eval set — real inputs with expected outcomes, run on every change
• A latency budget, with streaming so the interface responds before the answer finishes
• Cost per request measured and capped, with alerting on unusual usage
• Prompt caching and a smaller model for the easy majority of requests
• A defined failure path — fall back, degrade, or say plainly that it does not know
• Logging of inputs, outputs and versions, within your data retention policy

Evals are regression tests, and skipping them is the classic mistake

Without an eval set you cannot change anything safely. Every prompt tweak, model upgrade or retrieval change becomes a matter of opinion, and quality drifts in whichever direction the last edit pushed it. It does not need to be elaborate: thirty to a hundred real cases with the outcome you expect, run automatically, is enough to catch the regressions that matter. Add each production failure to the set as it happens and the suite becomes genuinely valuable within a couple of months.

Controlling cost and latency without hurting the feature

Guardrails deserve the same care as any other user-facing behaviour. Constrain outputs to a schema when the result feeds other code, validate before you act on it, and never let a model trigger an irreversible action — a refund, a deletion, an outbound message to a customer — without a person or a hard rule in between. Where the feature touches personal data, decide explicitly what is sent to a provider, what is retained, and what your privacy policy tells users about it. Those answers belong in the design, not in a scramble during a security review.

The projects that work look boring from the outside: one clear task, retrieval done properly, an eval suite in CI, a cost ceiling, a human in the loop where it matters, and a measured before-and-after. The projects that stall are the ambitious ones with no baseline and no way to tell whether they got better. If you are planning AI work in a real product this year, pick the smallest task with a genuine number attached to it and make that one work end to end first.