Neural Networks

Neural Networks

A neural network is a flexible function with a very large number of adjustable numbers (parameters, or weights). Training adjusts those numbers so the function maps inputs to the right outputs: an image to "cat", a sentence to its sentiment, a sequence of words to the next word.

Neural networks aren't magic, and they aren't brains. They're compositions of simple arithmetic steps, tuned by calculus. That's worth understanding concretely, because it explains both their power and their failure modes.

One neuron

A neuron takes inputs, multiplies each by a weight, adds a bias, and passes the result through an activation function:

z = w₁·x₁ + w₂·x₂ + w₃·x₃ + b        (a weighted sum)
a = f(z)                              (a non-linear "activation")

With inputs such as a house's size, number of rooms and age, the weights express how much each input matters, and in which direction. On its own, one neuron is just a linear model: logistic regression, if f is the sigmoid.

Layers: where the power comes from

Neurons are organized into layers. Each layer takes the previous layer's outputs as its inputs:

input features ─▶ hidden layer 1 ─▶ hidden layer 2 ─▶ output layer
   (x)             (h₁ = f(W₁x+b₁))   (h₂ = f(W₂h₁+b₂))   (ŷ)

Each layer transforms its input into a new representation. In an image network, early layers detect edges, middle layers detect textures and parts, and later layers detect whole objects. Nobody programs these features. They're learned because they help reduce the error.

Why activations must be non-linear: stacking linear layers (W₂(W₁x)) is just another linear function (Wx), so depth would add nothing. Non-linear activations let the network represent curved, complex decision boundaries.

The forward pass

Computing a prediction is just matrix arithmetic, layer by layer:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def forward(X, W1, b1, W2, b2):
    H = np.maximum(0, X @ W1 + b1)     # hidden layer with ReLU
    return sigmoid(H @ W2 + b2)        # output probability for a binary task

X holds a batch of examples (one per row), so the whole batch is processed at once. That's why GPUs, which are built for large matrix multiplications, made deep learning practical.

Measuring error: the loss function

Training needs one number that says how wrong the predictions are:

• Mean squared error for regression: the average of (y − ŷ)².
• Binary cross-entropy for yes/no outputs: −[y·log ŷ + (1−y)·log(1−ŷ)]. It heavily penalizes confident wrong answers.
• Categorical cross-entropy for multi-class outputs: −log(probability assigned to the correct class). It's also what language models minimize for the next token.

Learning: backpropagation and gradient descent

Training repeatedly nudges every weight in the direction that reduces the loss:

1. Forward pass: compute predictions and the loss for a batch.
2. Backward pass (backpropagation): apply the chain rule from the output back toward the input, computing the gradient ∂loss/∂w for every weight: how much the loss would change if that weight moved slightly. Frameworks such as PyTorch and TensorFlow do this automatically (autodiff).
3. Update: w ← w − learning_rate × gradient.
4. Repeat over many batches and epochs (passes through the data).

Using small random batches rather than the whole dataset is stochastic gradient descent (SGD). Modern optimizers improve on it: momentum smooths updates, and Adam / AdamW adapts the step size per parameter and is the common default.

The learning rate is the most important setting. Too high, and the loss bounces or explodes. Too low, and training crawls or stalls. Schedules that warm up and then decay the rate are standard for large models.

Generalization: fitting the data without memorizing it

Networks with millions of parameters can memorize training data (overfitting): great training accuracy, poor results on new data. Always evaluate on held-out validation and test sets. The usual defences:

• More, and more varied, data. Data augmentation (flips, crops, noise) helps for images.
• Dropout: randomly zero some neurons during training, so the network can't rely on any single path.
• Weight decay (L2 regularization): penalize large weights.
• Early stopping: stop when validation loss starts rising.
• Normalization layers (batch norm, layer norm): stabilize training, and allow higher learning rates.

Vanishing and exploding gradients: in deep networks, gradients multiplied through many layers can shrink toward zero or blow up. ReLU-family activations, careful weight initialization, normalization and residual (skip) connections (output = layer(x) + x) are what made networks with dozens or hundreds of layers trainable.

Architectures for different data

The universal approximation theorem says a wide enough network with one hidden layer can approximate any continuous function. In practice, depth is what makes learning efficient: deep networks build complex features hierarchically from simpler ones, with far fewer parameters than a single enormous layer.

Follow-up questions this topic invites — and their answers

Q: What is backpropagation, in one sentence? A: It's the chain rule of calculus applied efficiently through the network's computation graph, from the loss backwards, to compute the gradient of the loss with respect to every weight in one backward pass.

Q: Why did ReLU replace sigmoid in hidden layers? A: The sigmoid saturates: for large positive or negative inputs its gradient is nearly zero, so gradients vanish in deep networks. ReLU has a constant gradient for positive inputs, is cheap to compute, and made deep networks train much faster.

Q: How do you tell if a model is overfitting, and what do you do? A: Training loss keeps falling while validation loss rises, or plateaus well above it. Get more data or augment it, add regularization (dropout, weight decay), simplify the model, or stop training earlier.

Q: What does the learning rate do, and how do you choose it? A: It sets the size of each weight update. Too large diverges, and too small is slow or gets stuck. Start from a known-good default for the optimizer (around 1e-3 for Adam on small models), try a small range of values, and use warm-up and decay schedules for large models.

Q: When is a neural network the wrong tool? A: With small tabular datasets, gradient-boosted trees are usually more accurate, faster and easier to interpret. Neural networks shine with large amounts of unstructured data (images, text, audio), or when a pre-trained model can be fine-tuned.