Throughput

Definition The number of tokens an AI model can generate per second, determining how fast it produces complete responses.

In Depth

Higher throughput means faster responses. Cloud-hosted models on powerful hardware can generate 50-100+ tokens per second. Local models on consumer hardware may generate 10-30 tokens per second. Throughput is a key factor in user experience and cost per request.

Browse more terms

AI Agent AI Alignment Agentic AI Artificial General Intelligence Artificial Superintelligence Attention Mechanism AutoML Autoregressive Model Benchmark Bias (AI) CLIP Chain-of-Thought Computer Vision Constitutional AI Context Window DPO (Direct Preference Optimization) Deepfake Diffusion Model Document AI Edge AI