In Depth
MoE models can be very large in total parameters but only use a fraction for any given input, making them faster and cheaper than dense models of equivalent capability. GPT-4 and Mixtral are believed to use MoE architectures.
Mixture of Experts (MoE)
Definition
An architecture where multiple specialized sub-networks (experts) handle different types of inputs, with a router deciding which experts to activate for each request.
Browse more terms
AI Agent
AI Alignment
Agentic AI
Artificial General Intelligence
Artificial Superintelligence
Attention Mechanism
AutoML
Autoregressive Model
Benchmark
Bias (AI)
CLIP
Chain-of-Thought
Computer Vision
Constitutional AI
Context Window
DPO (Direct Preference Optimization)
Deepfake
Diffusion Model
Document AI
Edge AI