You already know the pieces. A neuron is a weighted sum plus a bias passed through a squashing function, which is exactly logistic regression. Stack a row of them into a layer, feed one layer's outputs into the next, and put a simple nonlinear activation in between, and you have a neural network. It learns by backpropagation, which is just the chain rule wiring gradient descent through every layer. Deep learning is these familiar ideas, composed and scaled.
Without the nonlinear activation, stacking layers is pointless: a chain of linear maps is still just one linear map, a single straight boundary. The nonlinearity between layers is what lets a network bend its decision boundary, compose simple features into complex ones, and fit shapes no line can. Everything deep learning can do rests on that small twist.
From a Neuron to a Network
Start with one neuron. It takes inputs x, multiplies each by a learned weight, adds a
learned bias, and squashes the result: a = f(w·x + b). That is it. With a sigmoid
for f, a single neuron is logistic regression. The power comes from wiring many neurons into
layers and stacking those layers.
Each hidden layer transforms its inputs into a new set of features, and the next layer builds on those. Early layers learn simple patterns; later layers combine them into complex ones. The final output layer reads the last set of features and produces the prediction, a class probability, a number, whatever the task needs. A network is nothing more than this: data flowing forward through a stack of weighted sums and activations, a step called the forward pass.
Activations: Why Depth Needs Nonlinearity
The activation function is the small nonlinear step applied after each layer's weighted sum. It is not a detail, it is the whole reason depth works. Stack two linear layers with nothing in between and the math collapses: a linear function of a linear function is still linear, one flat boundary. Insert a nonlinearity and the layers can no longer be merged, so each one genuinely adds shaping power.
ReLU (max(0, x)) is the modern default: cheap to compute, and it keeps gradients flowing
instead of vanishing the way the saturating S-curves do in deep stacks. Sigmoid and
tanh still appear, sigmoid to turn a final score into a probability, tanh inside some recurrent
layers. With a nonlinearity in place, a network of enough neurons is a universal approximator: given
enough hidden units, it can represent essentially any continuous function. Depth just makes that vastly more
efficient than width.
How a Network Learns: Backpropagation
Training is a loop. Run a forward pass to get predictions, measure how wrong they are with a loss function, then figure out how to nudge every weight to shrink that loss, and take a small step. Repeat for many epochs. The only new machinery is how the gradients are computed.
- ●Forward pass: push the batch through the layers to get predictions.
- ●Loss: score the predictions against the truth (cross-entropy for classification, squared error for regression).
- ●Backpropagation: apply the chain rule backward through the network to get the gradient of the loss with respect to every weight, reusing each layer's work.
- ●Update: step every weight a little way downhill (gradient descent, usually the Adam optimizer), and loop.
Backpropagation is the reason deep networks are trainable at all: it computes millions of partial derivatives in one efficient backward sweep rather than one at a time. Modern frameworks do it for you through automatic differentiation, so you define the forward pass and the gradients come free. The mechanics of the downhill step, learning rate, momentum, Adam, are exactly the optimization ideas from the Optimization & Gradient Descent chapter, now driving thousands of parameters at once.
Real-World Example: A Boundary No Line Can Draw
The clearest way to see what a network buys you is a problem a linear model cannot solve. The companion notebook trains a small network on it with PyTorch and watches it succeed where logistic regression stalls.
1,200 parts, two sensor readings each (sensor_a,
sensor_b) and a label passed_qc. A part passes only when its combined sensor magnitude sits
in a mid-band, a ring in the two-sensor plane, which no straight line can carve out.
- ●Linear model: logistic regression reaches only 61% accuracy, barely above guessing, because a single straight boundary cannot enclose a ring.
- ●Neural network: a tiny MLP (two hidden layers of 16 ReLU units, just 337 parameters) reaches 91%, learning a curved, ring-shaped boundary.
- ●Training: over 300 epochs the loss falls from about 0.72 to 0.22 as gradient descent tunes the weights.
Same data, same optimizer, same loss. The only difference is the two hidden layers and their ReLU activations, and that difference is the entire jump from 61 to 91 percent. Scale this idea up, millions of neurons, specialized layer types, mountains of data, and you have the models behind modern image, speech, and language systems.
The Architecture Landscape
The neuron-and-layer recipe stays the same; what changes across deep learning is how the layers are wired to match the data. A handful of architectures cover most of the field.
| Architecture | Wiring idea | Best for |
|---|---|---|
| Feedforward (MLP) | Fully-connected layers, every neuron to every neuron | Tabular data, the general-purpose default (this chapter) |
| Convolutional (CNN) | Small filters slid across the input, sharing weights to detect local patterns | Images, audio spectrograms, any grid with spatial structure |
| Recurrent (RNN / LSTM / GRU) | A loop that carries a hidden state along a sequence, a memory | Text and time series (now often replaced by transformers) |
| Transformer | Self-attention: every position looks at every other and decides what matters | Language, vision, and the backbone of modern foundation models |
| Embeddings | Learned dense vectors that place similar items near each other | Words, users, products; the input layer of most modern models |
| Autoencoders / GANs / diffusion | Networks that learn to compress or generate data | Representation learning, image and audio generation |
The defining trend is scale: the transformer plus enormous data and compute produced foundation models, pre-trained once and adapted to countless tasks, and the large language models built on them. Research now pushes on scaling laws (how performance grows with size), multimodal models that fuse text, images, and audio, and making these systems cheaper, safer, and more reliable. Every one of them is still, at bottom, the neuron and the backward pass in this chapter, composed billions of times.
Build and train a neural network in Python
The companion notebook computes a single neuron's forward pass by hand, plots the sigmoid, tanh, and ReLU activations, shows a logistic regression failing on the ring dataset, then builds a small network in PyTorch, trains it with an explicit loop (forward, loss, backward, step), plots the falling loss, and draws the curved decision boundary it learns, every step with a picture.
View opens the rendered notebook instantly.
Open in Colab runs it live (with a free GPU if you want one). To run locally, install
numpy, pandas, matplotlib, scikit-learn, torch, and
openpyxl.
🎓 Key Takeaways
- ✓A neuron is logistic regression: a weighted sum of inputs plus a bias, through an activation.
- ✓A network stacks layers of neurons; early layers learn simple features, later layers compose them.
- ✓Nonlinear activations are the point: without them, stacked layers collapse to one straight boundary. ReLU is the modern default.
- ✓Training is a loop: forward pass, loss, backpropagation (the chain rule backward), gradient-descent update, repeat.
- ✓Depth buys shape: on the ring data a linear model got 61% and a tiny network got 91%, purely from two hidden layers.
- ✓Architectures specialize the wiring: CNNs for grids, RNNs for sequences, transformers for almost everything now.
Practice Challenges
Five exercises on the sensor-QC network. Full solutions are in the companion solutions notebook.
Neuron by hand
Given weights, a bias, and an input vector, compute one neuron's output with a sigmoid activation. Confirm it matches a one-unit model.
sigmoid(w @ x + b).Linear baseline
Fit logistic regression on the ring data and report its accuracy. Why can it not beat about 60%?
Train the network
Build a 2-16-16-1 MLP in PyTorch and train it with a forward/loss/backward/step loop. Plot the loss per epoch.
nn.Sequential, BCEWithLogitsLoss, Adam.Remove the nonlinearity
Rebuild the network with the ReLU layers deleted (linear only). What accuracy do you get, and why?
Draw the boundary
Predict over a grid of sensor values and plot the network's decision boundary. Does it trace the ring?
np.meshgrid + contourf.Solutions notebook
All five challenges worked in code, the hand-computed neuron, the linear baseline, the trained network with its loss curve, the ablation that removes the nonlinearity, and the decision-boundary plot, each with a short explanation.
Quiz: Test Yourself
Eight questions on neurons, activations, backpropagation, and architectures. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved.
You can build and train a neural network. Next, Chapter 147 · Recommendation Systems puts these tools to work on one of the most valuable problems in industry: predicting what a user will want next, from collaborative filtering to the neural recommenders behind modern feeds.