What is Attention Mechanism?

Attention Mechanism

Definition A neural network component that allows a model to dynamically focus on different parts of its input when producing each output token, assigning learned relevance scores across the full context. Self-attention, used in transformers, computes these scores between all positions in the input simultaneously.

In Depth

The attention score between two tokens is computed as the dot product of their query and key vectors, scaled and softmaxed to produce a probability distribution, then used to weight their value vectors. Multi-head attention runs several attention operations in parallel with different learned projections, capturing multiple types of relationships. The O(n²) cost of full attention over sequence length n drives ongoing research into efficient attention approximations.

Browse more terms

AI Agent AI Alignment Agentic AI Artificial General Intelligence Artificial Superintelligence Attention Mechanism AutoML Benchmark Bias (AI) Chain-of-Thought Computer Vision Deepfake Diffusion Model Edge AI Embedding Federated Learning Few-Shot Learning Fine-Tuning Function Calling GPT