Latency

Definition The time delay between sending a request to an AI model and receiving the first token of the response.

In Depth

Latency matters for real-time applications like chatbots and voice assistants. Typical API latencies: 200-500ms for first token, with streaming delivering subsequent tokens continuously. Smaller models have lower latency. On-device models have the lowest latency but less capability.

Browse more terms

AI Agent AI Alignment Agentic AI Artificial General Intelligence Artificial Superintelligence Attention Mechanism AutoML Autoregressive Model Benchmark Bias (AI) CLIP Chain-of-Thought Computer Vision Constitutional AI Context Window DPO (Direct Preference Optimization) Deepfake Diffusion Model Document AI Edge AI