Tokenizer

Definition The component that breaks text into tokens (subword units) that the AI model processes.

In Depth

Different models use different tokenizers. GPT models use BPE (Byte Pair Encoding). The same text may be different numbers of tokens on different models. Understanding tokenization matters for cost estimation — API pricing is per-token, and the same document costs different amounts on different models.

Browse more terms

AI Agent AI Alignment Agentic AI Artificial General Intelligence Artificial Superintelligence Attention Mechanism AutoML Autoregressive Model Benchmark Bias (AI) CLIP Chain-of-Thought Computer Vision Constitutional AI Context Window DPO (Direct Preference Optimization) Deepfake Diffusion Model Document AI Edge AI