Every model in this book eats numbers. Text is not numbers, so the first and most consequential decision in NLP is how to represent it. The history of the field is essentially three answers, each richer than the last: count the words, embed them as dense vectors that place related meanings near each other, and finally let the model compute, for every word, which other words matter. That last idea is attention, and it is what makes a large language model work.
An LLM is trained on a task a child could state: given the text so far, predict the next token. Run that over a large fraction of the written internet, with enough parameters, and grammar, facts, translation, summarizing, and a passable imitation of reasoning all fall out as side effects. No part of the recipe says “learn to reason”, it emerges from relentless next-token prediction.
From Text to Numbers
Start simple. Tokenize the text into words, build a vocabulary, and represent each document by how often each vocabulary word appears. That is the bag-of-words vector: it throws away word order entirely and still works remarkably well for classification.
TF-IDF refines the raw counts. A word matters if it is frequent in this document (term frequency) but rare across the corpus (inverse document frequency). That pushes down “the” and “this”, which appear everywhere and distinguish nothing, and pushes up the words that actually carry the signal. The result is still a sparse vector with one dimension per vocabulary word, and still order-blind, but it is a genuinely strong baseline for classification.
Embeddings: Meaning as Geometry
Bag-of-words has a fatal blind spot: every word is its own dimension, so excellent and superb are as unrelated as excellent and toaster. Embeddings fix this by mapping each token to a dense vector of a few hundred numbers, learned so that words used in similar contexts land near each other. Meaning becomes geometry.
Two useful consequences. First, similarity becomes a number: the cosine of the angle
between two vectors tells you how related two words or documents are, which is what powers semantic search and the
retrieval step in modern AI systems. Second, directions in the space are meaningful, the classic
demonstration being that king − man + woman lands near queen. Modern models go further
with contextual embeddings: the vector for “bank” differs in river bank and
savings bank, because the representation now depends on the surrounding words.
Attention, Transformers, and How an LLM Works
Contextual embeddings need a mechanism that lets each token look at the others and decide what is relevant. That mechanism is self-attention. For every token the model forms a query, and every token offers a key and a value. Compare a query against all keys, turn the scores into weights with a softmax, and the token's new representation is that weighted blend of values.
Stack that operation, many heads in parallel and many layers deep, add a feed-forward block and positional information, and you have a transformer. Because attention compares every token to every other in parallel, it trains far faster than the sequential recurrent models it replaced, which is exactly what made scaling possible. A large language model is such a transformer pre-trained on a huge corpus to predict the next token, then usually fine-tuned, often with human feedback, to follow instructions helpfully. Generation is just that prediction run in a loop, one token at a time.
Real-World Example: Classifying Product Reviews
Before reaching for a language model, know what the classical pipeline gets you. The companion notebook runs the full text-classification workflow with scikit-learn.
800 product reviews labeled positive or negative (404 / 396), in three
columns: review_id, review_text, and sentiment. About 8 percent of
the labels are deliberately flipped, so perfect accuracy is impossible, exactly like real annotated data.
- ●The vocabulary is tiny: just 80 distinct words across all 800 reviews, enough for a sparse vector per document.
- ●Both representations tie: bag-of-words and TF-IDF each reach 92.9% accuracy (F1 0.93), right at the ceiling the 8% label noise allows.
- ●The model is readable: its strongest positive words are great, highly, works, excellent; its strongest negative words are poor, waste, terrible, not.
Two honest lessons hide in that result. First, TF-IDF did not beat raw counts here: on short, clean documents with a small vocabulary there are no long rambling texts for it to down-weight, so the reweighting has nothing to fix. Its advantage shows up on longer, noisier corpora. Second, the model stalls at about 93 percent not because it is weak but because 8 percent of the labels are wrong, an irreducible error no architecture can beat. A transformer would not do better here; knowing that saves you from reaching for one.
Evaluating Language Models & the AI Landscape
Classification has accuracy and F1. Generated text is harder: there is rarely one right answer, so the field uses a family of proxies, each measuring something different and each easy to game.
| Metric | What it measures | Used for |
|---|---|---|
| Accuracy / F1 | Right-or-wrong against a label, balancing precision and recall | Classification: sentiment, topic, intent, spam |
| Perplexity | How surprised the model is by real text; lower is better (roughly, how many words it is choosing among) | Language modeling and pre-training progress |
| BLEU | N-gram precision against reference text: how much of what you produced appears in the reference | Machine translation |
| ROUGE | N-gram recall: how much of the reference your output covered | Summarization |
| BERTScore | Similarity of embeddings rather than exact words, so paraphrases still score well | Generation, where wording legitimately varies |
| Human eval / LLM-as-judge | People (or a strong model) rate helpfulness, accuracy, and safety | Open-ended assistants, where n-gram overlap fails |
The word-overlap metrics share a weakness worth remembering: they reward matching tokens, not matching meaning. A perfect paraphrase can score badly on BLEU, and a fluent, confidently wrong answer can score well. That is why embedding-based scores and human judgment have become essential for anything open-ended.
NLP is the fastest-moving corner of AI. The live threads: scaling laws and how far next-token prediction goes; instruction tuning and RLHF to make raw models useful and safe; retrieval-augmented generation, which grounds answers in real documents to fight hallucination; efficiency work (quantization, distillation, small capable models); and multimodal systems that treat text, images, and audio as one token stream. Running underneath is an evaluation crisis: benchmarks saturate almost as fast as they are built, so measuring genuine capability, and genuine reliability, is itself an open research problem.
Do NLP in Python
The companion notebook tokenizes the reviews and builds bag-of-words and TF-IDF vectors, trains and compares classifiers on both, reads out the most predictive words, builds a small embedding space from the corpus and finds nearest neighbors by cosine similarity, computes self-attention by hand with softmax over query-key scores, and finally implements perplexity, BLEU, and ROUGE from scratch so the metrics stop being magic.
View opens the rendered notebook instantly.
Open in Colab runs it live. To run locally, install numpy, pandas,
matplotlib, scikit-learn, and openpyxl (the transformers library
adds pre-trained models and tokenizers).
🎓 Key Takeaways
- ✓Representation is the first decision: tokenize, then counts (bag-of-words), then TF-IDF weighting, then dense embeddings.
- ✓Bag-of-words is order-blind: “dog bites man” and “man bites dog” get identical vectors, yet it is still a strong baseline.
- ✓Embeddings make meaning geometric: similar words sit close, cosine similarity measures relatedness, and directions carry analogies.
- ✓Self-attention lets every token weigh every other (softmax over query-key scores); stacking it gives a transformer.
- ✓An LLM is next-token prediction at scale, pre-trained then fine-tuned; generation is that prediction looped.
- ✓Match the metric to the task: accuracy/F1 for classification, perplexity for language modeling, BLEU (precision) for translation, ROUGE (recall) for summarization, BERTScore or humans for open-ended text.
Practice Challenges
Five exercises on the reviews corpus. Full solutions are in the companion solutions notebook.
Bag-of-words vs TF-IDF
Vectorize the reviews both ways, train a classifier on each, and compare accuracy. Why is the difference so small here?
CountVectorizer vs TfidfVectorizer.Read the model
Pull the largest positive and negative coefficients and list the words behind them. Do they make sense?
get_feature_names_out() + clf.coef_.Similar words by cosine
Build a term-document matrix, reduce it with SVD, and find each word's nearest neighbors by cosine similarity.
TruncatedSVD then cosine_similarity.Attention by hand
Given small query, key, and value matrices, compute softmax(QKT/√d)V and confirm the weights sum to 1.
Perplexity and BLEU
Compute perplexity for a unigram and a bigram model, and BLEU and ROUGE for a candidate sentence against a reference.
Solutions notebook
All five challenges worked in code, the two representations compared, the model's most predictive words, cosine nearest neighbors, self-attention computed by hand, and perplexity, BLEU, and ROUGE implemented from scratch, each with a short explanation.
Quiz: Test Yourself
Eight questions on text representation, embeddings, attention, and evaluation metrics. Answer them, hit Check Answers, and keep refining until you score 100%. Your progress is saved.
Language models are trained on data far too large for one machine. Big Data & Scaling closes this part with the engineering that makes that possible: distributed storage and compute, parallel processing, and the practical limits of scale. Browse the full Contents for what is published and what is on the way.