Backpropagation Step by Step - A Computational Example

Introduction

Backpropagation is the algorithm that made training deep neural networks practical. The idea is straightforward: compute the derivatives of the loss function with respect to the network's parameters, telling us in which direction to nudge each one to reduce the error.

Worth noting — backpropagation is a general-purpose optimization algorithm. It works for any function built from composable operations, not just neural networks. In transformers, for example, the same mechanism trains both the network weights and the token embeddings, which are simply a table of parameters rather than a neural layer.

The best way to build real intuition for backpropagation is to work through a concrete example by hand. In this post, we'll carry out the complete calculations for a simple function.

Gradient, Derivative, Partial Derivative — Quick Definitions

Before the calculations, a brief math refresher:

Derivative — applies to a function of one variable: \(f(x)\). It describes how fast the function changes as \(x\) changes. Notation: \(\frac{df}{dx}\) or \(f'(x)\).

Partial derivative — applies to a function of multiple variables: \(f(x, y, z)\). It describes how the function changes with respect to one variable, holding the others fixed. Notation: \(\frac{\partial f}{\partial x}\).

Gradient — a vector of all partial derivatives. For \(f(x, y, z)\):

The gradient points in the direction of steepest ascent.

In this post we use "gradient" for the full vector (e.g. "compute the parameter gradients") and "partial derivative" for individual components (e.g. \(\frac{\partial L}{\partial a}\)).

The Example

Consider a loss function depending on three parameters:

Input values:

• \(a = 2\)
• \(b = -3\)
• \(c = 10\)

Our goal is to compute \(\frac{\partial L}{\partial a}\), \(\frac{\partial L}{\partial b}\), \(\frac{\partial L}{\partial c}\) — these tell us how each parameter affects the loss.

Computational Graph

Let's break the function down into atomic operations:

a ──┐
    ├──[×]── d ──┐
b ──┘            ├──[+]── e ──[^2]── L
             c ──┘

Where:

• \(d = a \cdot b\)
• \(e = d + c\)
• \(L = e^2\)

Decomposing into atomic operations is key — the derivative of each individual step becomes trivial to compute.

Step 1: Forward Pass

Compute values from inputs to output:

We store all intermediate values — they'll be needed during the backward pass. Without them, we'd have to recompute everything from scratch.

Step 2: Backward Pass

Now we propagate gradients backwards, from output to inputs. We use the chain rule:

where \(y\) is an intermediate variable between \(L\) and \(x\).

Wikipedia has a nice intuition for the chain rule, courtesy of George F. Simmons: "If a car travels twice as fast as a bicycle, and the bicycle travels four times as fast as a walking person, then the car travels 2 × 4 = 8 times as fast as the person."

2.1 Output Gradient

A function's derivative with respect to itself is always 1. This is our starting point.

2.2 Gradient w.r.t. \(e\)

2.3 Gradients w.r.t. \(d\) and \(c\)

The derivative of a sum with respect to each term is 1:

Applying the chain rule:

2.4 Gradients w.r.t. \(a\) and \(b\)

Product rule:

Applying the chain rule:

Gradient Summary

Interpreting the Gradients

What do these numbers actually mean?

• \(\frac{\partial L}{\partial a} = -24\) — increasing \(a\) by a small \(\Delta\) will decrease \(L\) by roughly \(24\Delta\)
• \(\frac{\partial L}{\partial b} = 16\) — increasing \(b\) by a small \(\Delta\) will increase \(L\) by roughly \(16\Delta\)
• \(\frac{\partial L}{\partial c} = 8\) — increasing \(c\) by a small \(\Delta\) will increase \(L\) by roughly \(8\Delta\)

The gradient points toward the steepest ascent. To minimize \(L\), we move in the opposite direction.

Step 3: Gradient Descent

With the gradients in hand, we update the parameters to reduce the loss.

Update rule:

where \(\eta\) is the learning rate. We'll use \(\eta = 0.01\).

Calculations

Step 4: Verification

Let's confirm the loss actually went down:

Loss dropped from 16 to 8.07 ✓

Repeating this cycle (forward → backward → update) will keep driving the loss down until it reaches a minimum.

A Few Closing Thoughts

Why Subtract the Gradient?

The gradient points toward the steepest increase of the function. Since we want to minimize it, we go the other way — hence the minus sign in the update rule.

Think of it like hiking: if you want to get to the bottom of a valley, you walk in the direction opposite to the steepest slope.

The Role of Learning Rate

The learning rate \(\eta\) controls the step size:

• Too large — you may overshoot the minimum and oscillate or diverge
• Too small — training becomes painfully slow
• Just right — stable convergence to a minimum

Choosing the right learning rate is one of the most important hyperparameter decisions in practice. Modern optimizers (Adam, AdaGrad) handle this automatically by adapting the learning rate per parameter.

From This Example to a Real Neural Network

In a real network you have:
- Thousands or millions of parameters (weights \(w\) and biases \(b\))
- A deep, multi-layer computational graph
- Activation functions (ReLU, tanh, sigmoid)
- Matrix operations instead of scalars

But the mechanism is identical:
1. Forward pass — compute the network output and loss
2. Backward pass — propagate gradients from the loss back to all weights
3. Update — move parameters in the direction opposite to the gradient
4. Repeat — until the loss is small enough

Implementation in PyTorch

One last thing — PyTorch (and other deep learning frameworks) compute all of this for us automatically. Here's our example in code:

import torch

# Define parameters as tensors with gradient tracking enabled
a = torch.tensor([2.0], requires_grad=True)
b = torch.tensor([-3.0], requires_grad=True)
c = torch.tensor([10.0], requires_grad=True)

# Forward pass — PyTorch builds the computational graph automatically
d = a * b
e = d + c
L = e ** 2

print(f"Loss: {L.item()}")  # 16.0

# Backward pass — one call computes all gradients
L.backward()

# Read the gradients
print('---')
print(f'∂L/∂a = {a.grad.item()}')  # -24.0
print(f'∂L/∂b = {b.grad.item()}')  # 16.0
print(f'∂L/∂c = {c.grad.item()}')  # 8.0

Output:

Loss: 16.0
---
∂L/∂a = -24.0
∂L/∂b = 16.0
∂L/∂c = 8.0

Exactly the values we computed by hand! PyTorch handled everything: building the graph, caching intermediate values, and applying the chain rule.

Key elements:
- requires_grad=True — enables operation tracking for a tensor
- L.backward() — kicks off backpropagation from L
- .grad — holds the computed gradient for each parameter

We'll take a closer look at this mechanism in the next post, where we'll build a simple neural network from scratch.

Summary

Backpropagation is an elegant application of the chain rule for efficiently computing gradients in computational graphs. The key steps:

1. Forward pass — compute values from inputs to output, cache intermediates
2. Backward pass — propagate gradients from output to inputs using the chain rule
3. Gradient descent — update parameters: \(\theta = \theta - \eta \cdot \nabla_\theta L\)
4. Iterate — repeat until convergence

This simple algorithm, implemented efficiently on GPUs, is what makes training models with billions of parameters possible.

Further Reading

• Calculus on Computational Graphs: Backpropagation (Chris Olah)
• Neural Networks and Deep Learning, Chapter 2 (Michael Nielsen)
• CS231n: Backpropagation (Stanford)