I've watched two AI features die for reasons that had nothing to do with the model. The answers were good. The feature was slow, and slow killed it before quality ever got a vote.
Teams treat latency as an engineering number to optimize after launch. It isn't. How long a user waits is a product decision, and it's usually made by accident — inherited from a model choice, a prompt length, a retrieval step nobody profiled. By the time it shows up in the metrics, the shape of the feature is already set.
So I budget latency the way I budget cost or scope. On purpose, up front, per action.
Speed is a trust signal, not a nicety
Users don't have a mental model of tokens per second. They have a mental model of competence. A tool that answers instantly feels sure of itself. A tool that hangs for eight seconds feels like it's struggling — and a struggling tool invites doubt about the answer, even when the answer is right.
That's the part teams miss. Latency doesn't just cost patience. It costs credibility. The same output delivered in one second reads as confident and delivered in ten reads as guesswork. You are shaping the user's trust in the content by how you deliver it.
Speed also decides whether a feature gets used at all. Anything a user reaches for many times a day has a hard ceiling on acceptable wait. Cross it and they route around your feature — back to the old workflow, the manual step, the thing that at least doesn't make them wait.
The framework: match the wait to the job
Not every AI action deserves the same latency budget. I sort them into four tiers, and each has a different target and a different design response.
Tier 1 — Inline, under 1 second. Autocomplete, inline suggestions, live classification as someone types. Here latency isn't a feature of the experience; it is the experience. If it isn't near-instant, ship a smaller, faster model or don't ship it. A slow inline suggestion is worse than none — it interrupts instead of helping.
Tier 2 — Interactive, 1 to 4 seconds. The user asked for something and is waiting on the answer: a summary, a drafted reply, a search result. This is where streaming earns its keep. First token fast, the rest flowing — the wait becomes reading time instead of dead time.
Tier 3 — Considered, 4 to 30 seconds. A real task the user knows is real: analyze this document, generate this report. People will wait, but only if you show the work. Progress that names its steps — reading, extracting, checking — buys patience that a spinner never will.
Tier 4 — Background, seconds to minutes. Batch jobs, overnight runs, anything the user starts and walks away from. The budget here is generous, but the contract is strict: tell them when it's done, and never make them sit and watch. A Tier 4 job with a Tier 2 interface — a user staring at a loading bar for ninety seconds — is a design failure, not a speed problem.
The mistake is applying one target to everything, or letting the model's raw speed decide the tier by default. Decide the tier first. Then make the engineering serve it.
Perceived speed is the number that matters
Wall-clock latency is what you measure. Perceived latency is what the user feels — and they are not the same number. Most of your room to improve lives in the gap between them.
Streaming is the biggest lever. A response that takes six seconds to finish but shows its first words in four hundred milliseconds feels fast, because the user starts reading immediately. Same total time, entirely different experience.
Optimistic UI helps too: show the action as done the instant it's reversible to undo, and reconcile in the background. Skeleton states that hint at the shape of the answer beat a blank spinner. And honest, specific progress — checking 3 of 12 sources — turns a wait into a story the user is willing to sit through.
None of this makes the model faster. All of it makes the feature feel faster, which is the number that decides whether people keep using it.
Set the budget before you build
Here's the discipline I hold teams to. Before we build an AI feature, we write down its latency budget — the tier, the target, the fallback when we blow it. That number then constrains everything downstream: model size, prompt length, how many retrieval hops we can afford, whether we pre-compute.
Do it in that order and speed is a constraint you designed around. Do it backwards — build first, measure later — and speed becomes a crisis you discover in production, when the fix is expensive and the users have already decided.
Latency is a product decision. Make it on purpose.
FAQ
What's an acceptable response time for an AI feature? It depends entirely on the job. Inline suggestions need to land under a second; a document analysis can take thirty if you show progress. The wrong question is "how fast can the model go" — the right one is "how long can this specific action make the user wait before they route around it."
Does streaming actually help, or is it a trick? Both, and the trick is the point. Streaming doesn't reduce total time, but it converts dead waiting into reading time by showing the first tokens fast. Perceived speed is what drives trust and usage, and streaming moves it more than almost anything else you can do.
Should I pick a faster model or a smarter one? Match the model to the tier, not the other way around. For inline, sub-second work, a smaller fast model usually wins even at some quality cost, because a slow suggestion goes unused. For considered tasks where the user is already committed to waiting, spend the latency on quality.
How do I stop latency from becoming a launch-day surprise? Write the latency budget before you build — tier, target, and fallback — and let it constrain model size, prompt length, and retrieval. Speed decided up front is a design constraint; speed discovered in production is an expensive rebuild.
I'm Ravi Jadav, Chief Product Officer and Co-Founder at Sunbots Innovations and Co-Founder at Xwits Developers. Get in touch.