What is Multimodal AI?

Multimodal AI

Definition AI systems that can process and generate multiple types of data — such as text, images, audio, and video — within a single model. Multimodal models can answer questions about images, generate images from text, or transcribe and summarize audio.

In Depth

GPT-4o, Gemini, and Claude are prominent multimodal LLMs. Building multimodal systems typically involves training separate encoders for each modality and projecting their outputs into a shared embedding space the language model can reason over. Multimodal capability is expanding rapidly into video understanding, real-time speech conversation, and document analysis with mixed text and images.

Browse more terms

AI Agent AI Alignment Agentic AI Artificial General Intelligence Artificial Superintelligence Attention Mechanism AutoML Benchmark Bias (AI) Chain-of-Thought Computer Vision Deepfake Diffusion Model Edge AI Embedding Federated Learning Few-Shot Learning Fine-Tuning Function Calling GPT