In Depth
Residual Networks (ResNets), introduced by Kaiming He et al. in 2015, solved a fundamental problem in deep learning: as networks get deeper, they become harder to train and performance can actually degrade. ResNets address this with skip connections (residual connections) that allow information to bypass one or more layers, so each block only needs to learn the residual difference from the identity mapping.
The skip connections provide a gradient highway that allows backpropagation to flow more easily through very deep networks. This simple architectural change enabled training of networks with hundreds or even thousands of layers, when previously networks deeper than about 20 layers were difficult to train effectively. ResNet won the ImageNet competition in 2015 with a 152-layer network.
Residual connections have become a universal design principle in deep learning, extending far beyond their original image classification application. Transformers use residual connections around every attention and feed-forward block. Modern architectures almost universally incorporate skip connections in some form. The concept fundamentally changed how researchers think about depth in neural networks.