In Depth
Adversarial attacks exploit vulnerabilities in AI models by creating inputs that are intentionally designed to cause misclassification or errors. In computer vision, adding carefully computed pixel-level noise (invisible to humans) to an image can cause a model to confidently misclassify it, for example, seeing a stop sign as a speed limit sign. In natural language processing, small word substitutions can flip a model's sentiment prediction.
Adversarial attacks come in several forms: white-box attacks (where the attacker knows the model's architecture and weights), black-box attacks (where the attacker can only query the model), targeted attacks (causing a specific wrong output), and untargeted attacks (causing any wrong output). Research has shown that adversarial examples often transfer between models, meaning an attack crafted against one model may fool a different one.
The existence of adversarial attacks raises serious concerns for safety-critical AI applications like autonomous driving, medical diagnosis, and security systems. Defense strategies include adversarial training (including adversarial examples in training data), input preprocessing, certified defenses (providing mathematical guarantees within bounded perturbations), and ensemble methods. The ongoing arms race between attacks and defenses is a major area of AI safety research.