ImageNet Classification with Deep Convolutional Neural Networks

You've seen the box name our leopard. Before opening it, pin down exactly what it was asked to do — and what the leopard looks like by the time it reaches the first layer.

ImageNet is a giant labeled photo collection — roughly 15 million images across ~22,000 categories. The contest, ILSVRC, uses a 1000-class slice: ~1.2M training images, 50,000 for validation, 150,000 for test. The job: put each photo in exactly one of those 1000 boxes.

Two scores grade it. Top-1 error is the fraction of photos where the model's single best guess is wrong. Top-5 error is gentler — wrong only if the true label isn't among the model's five best guesses. (For a leopard, “my top guess was jaguar but leopard was second” still counts as a top-5 hit.)

You know the input tensor. Here's the map the rest of the page zooms into — the whole machine as one left-to-right pipeline, before we magnify any single piece.

AlexNet has 8 learned layers: 5 convolutional layers (the part that looks at the image) followed by 3 fully-connected layers (the part that decides). Data flows one way, left to right, each layer feeding the next, until a final softmax turns the last 1000 numbers into class probabilities.

You have the floor plan; now we zoom all the way into the first convolutional layer — the heart of how the machine sees. Before any real sizes, do one by hand.

A convolution takes a small grid of learned numbers — a kernel (also called a filter) — and slides it over the image. At each stop it multiplies the kernel against the patch beneath it, adds up the products, and writes that one number into an output grid called a feature map. The stride is how many pixels it jumps between stops.

Our toy: a 5×5 grayscale patch with a vertical light/dark edge (bright on the left, dark on the right), and a 3×3 vertical-edge kernel. Slide it with stride 1 and watch where it fires.

Now zoom back out. The real conv1 doesn't run one kernel — it runs 96 of them, each 11×11×3, sliding across the 224×224×3 leopard with stride 4. Out come 96 feature maps, each about 55×55. And what did those 96 kernels learn? Almost exactly what you'd hope: edge and orientation detectors, plus coloured blobs.

Your hand-convolution produced a −27 for the wrong-way edge. What happens to negative responses? They meet ReLU — and ReLU is the quiet reason a net this big could be trained in 2012.

ReLU (rectified linear unit) is the simplest nonlinearity imaginable: f(x) = max(0, x). It's a non-saturating function — for large positive inputs it keeps growing, so its slope never flattens. The old choices, tanh and sigmoid, are saturating: push them hard and they flatten toward ±1.

This is the one spot worth slowing down for, because the reason saturation hurts is easy to state wrong. It isn't the flat output that stalls learning — it's the flat slope. Learning flows backward through each layer by multiplying by that layer's slope; where tanh has gone flat that slope is nearly 0, so the correction meant for the early weights gets multiplied down to almost nothing, layer after layer. ReLU's slope is exactly 1 for every positive input, so the signal passes through undiminished. Drag the input below and watch the slope, not just the height:

After conv + ReLU, a feature map is large and a little jittery. The next move shrinks it while keeping what matters — a step called pooling.

Max-pooling is downsampling by summary: tile the feature map into small windows and keep only the largest value in each. That halves the map's size and gives translation tolerance — nudge the object a pixel or two and the strongest response usually survives. Here's a 4×4 map pooled into 2×2:

Convolution finds features, ReLU keeps the strong ones, pooling shrinks the map. One more touch sat between some of those layers in 2012: a normalization borrowed from how real neurons behave.

Local response normalization (LRN) is a form of lateral inhibition — a loud neuron damps the neurons at the same spot in neighbouring feature maps, so the single strongest response stands out instead of many mushy ones. The paper calls it “brightness normalization.” The formula divides each activation by a term built from its neighbours' squared activations:

In words: if a feature map fires hard at some pixel, the same pixel in the maps next to it gets pushed down. The bigger a neuron's own activation, the more it suppresses its neighbours — competition that sharpens the response. It bought 1.4% / 1.2% off top-1 / top-5 error.

You now understand each layer's job. There's one constraint that shaped the whole architecture — and it wasn't about accuracy, it was about memory.

A single GTX 580 in 2012 had only 3GB of memory — too small to hold the whole network. So AlexNet was split across two GPUs: roughly half the kernels live on each card, and the cards read and write each other's memory directly. To keep that cross-talk cheap, the GPUs only communicate at certain layers. Conv3 reads from all conv2 maps across both cards; but conv4 reads only from the same-GPU conv3 maps.

The lovely payoff: because the two halves rarely talk, they specialized on their own — and the same split shows up on every training run. GPU 1's first-layer kernels came out largely colour-agnostic (grayscale edge detectors); GPU 2's came out largely colour-specific (coloured blobs). The legend chips below carry the only two functional colours on this page:

The architecture is complete: it sees (Steps 4–8). But a model with 60M parameters and only 1.2M images has a dangerous temptation — and the paper spends a whole section fighting it.

Overfitting is when a model memorizes the training photos instead of learning the general idea of “leopard” — it aces what it's seen and flops on anything new. With each of 1000 labels giving only ~10 bits of constraint, AlexNet was wide open to it. Two defenses, side by side.

The architecture sees and resists memorizing. The last piece is training — turning 60M random numbers into a leopard recognizer.

It learned by stochastic gradient descent (SGD): show a batch of images, measure the error, nudge every weight a little to reduce it, repeat. Batch size 128, momentum 0.9 (keep rolling in the direction you've been heading), and a small weight decay of 0.0005 — which, unusually, reduced training error here rather than acting purely as a regularizer.

The machine is fully assembled: it sees (Steps 4–8), resists memorizing (Step 9), and was trained (Step 10). Final question — did attention to all those details actually pay off?

First, the leopard. The last fully-connected layer produces 1000 raw scores called logits; softmax turns them into probabilities that sum to 1. Top-1 = the tallest bar; top-5 = is the true label among the five tallest? Here are toy softmax outputs for our leopard:

And the headline numbers. The report card stacks AlexNet against the best prior systems — error bars, so shorter is better:

Five questions to test the wiring. Try each one from memory, then open it to check — the answers run a line each, and they retrace the same path the leopard took, from raw pixels to a label.