N NexChron

Token

Definition The basic unit of text that a language model processes — typically a word, sub-word, or character, depending on the tokenizer. Models read and generate sequences of tokens rather than raw characters or words.

In Depth

Modern LLMs use sub-word tokenization schemes like Byte-Pair Encoding (BPE) or SentencePiece, which balance vocabulary size against sequence length. The number of tokens in a prompt and response determines computational cost and falls within the model context window limit. A rough rule of thumb is that 1,000 tokens ≈ 750 English words, though this varies by language and tokenizer.

Browse more terms

AI Agent AI Alignment Agentic AI Artificial General Intelligence Artificial Superintelligence Attention Mechanism AutoML Benchmark Bias (AI) Chain-of-Thought Computer Vision Deepfake Diffusion Model Edge AI Embedding Federated Learning Few-Shot Learning Fine-Tuning Function Calling GPT