In Depth
Quantization can shrink a model by 4-8x, allowing large models to run on consumer hardware or cheaper cloud instances. Popular formats include GGUF and GPTQ. Most open-source model deployments use some form of quantization.
Quantization
Definition
A technique that reduces the precision of numbers in an AI model (e.g., from 32-bit to 4-bit) to make it smaller and faster without significant quality loss.
Browse more terms
AI Agent
AI Alignment
Agentic AI
Artificial General Intelligence
Artificial Superintelligence
Attention Mechanism
AutoML
Autoregressive Model
Benchmark
Bias (AI)
CLIP
Chain-of-Thought
Computer Vision
Constitutional AI
Context Window
DPO (Direct Preference Optimization)
Deepfake
Diffusion Model
Document AI
Edge AI