THE DIRECT ANSWER

Latency is the delay experienced while a system responds. For AI applications, time to first response, total completion time and delays in tools or retrieval can differ. Measure the complete user journey rather than relying on one model-generation speed figure.

Break down the journey

A request can spend time retrieving data, waiting in a queue, processing context, generating text and calling tools. Time to the first visible response differs from time to a usable result. An early sentence does not help much if the required action remains unfinished.

Measure realistic work

For a document assistant, time the full path from upload to reviewed answer. Repeat with ordinary and unusually large inputs, and examine slow cases separately. A single fast demonstration says little about consistency under normal load.

Improve the bottleneck

Remove redundant calls, shorten irrelevant context and run independent steps concurrently where appropriate. Consider a smaller model when it meets the same acceptance criteria. Give meaningful progress feedback during work that genuinely takes time. Preserve the checks and evidence users need; removing validation can make a fast system unreliable. Record both quality and delay after each change. The best optimisation reduces the wait that matters to the user without moving hidden work onto the reviewer.

A worked example

An assistant may begin producing words quickly but wait several seconds for a database lookup before giving a useful answer. Record both the first output and the time when the task is complete. Repeat under representative load. Improving token generation will have limited value if the slowest part is an unrelated service or an approval step.

COMMON QUESTIONS

Questions readers ask

Is tokens per second the same as user-perceived speed?

No. Startup delay, retrieval, tools and output length also affect the experience. Measure the time to a useful completed result.

SOURCES & NOTES

Follow the evidence.

AI-assisted explanation with linked technical references. Examples are illustrative learning exercises, not reported customer deployments or experimental results.

  1. developers.openai.com — official reference

Public sources support the factual background. Analysis and illustrative exercises are identified in the text. How we assess AI · Editorial standards

Spot a factual error or an outdated detail?