How LLMs Work

How LLMs Work

A large language model (LLM), like the ones behind ChatGPT, Claude or Gemini, is a neural network trained to do one thing: given some text, predict what comes next. It predicts one small piece (a token), appends it, and repeats. Everything impressive these models do (answering questions, writing code, summarizing, translating) emerges from getting extremely good at that single prediction task, at enormous scale.

Knowing how this works explains their everyday behaviour: why they sometimes state false things confidently, why wording changes answers, why long inputs cost more, and why the same prompt can give different outputs.

Step 1: text becomes tokens

Models don't read letters or whole words. A tokenizer splits text into tokens, frequent chunks of characters learned from data (commonly by byte-pair encoding, which repeatedly merges the most common adjacent pairs). Each token has an ID:

"Unbelievable results!"  →  ["Un", "believ", "able", " results", "!"]  →  [2436, 48132, 481, 3135, 0]

Rules of thumb for English: 1 token ≈ ¾ of a word, and 100 tokens ≈ 75 words. Consequences you'll meet in practice:

• Pricing and limits are in tokens, not characters or words.
• Unusual words, code, numbers and non-English text often use more tokens per word.
• Tasks that look at individual letters (counting the r's in "strawberry", reversing a word) are awkward, because the model never directly sees letters, only chunks.

Step 2: tokens become vectors

Each token ID is mapped to an embedding, a vector of thousands of numbers learned during training. Positions in the vector space encode meaning: tokens used in similar contexts end up close together. Information about position is added too, since word order matters ("dog bites man" vs "man bites dog").

Step 3: the transformer and attention

The core of modern LLMs is the transformer, a stack of dozens of identical layers. Each layer has two main parts:

1. Self-attention: every token looks at the other tokens before it and decides how much each one matters for understanding it.
2. A feed-forward network: processes each token's updated representation. Much of the model's stored "knowledge" is thought to live in these weights.

Attention, intuitively. In "The trophy didn't fit in the suitcase because it was too big", the representation of "it" should draw heavily on "trophy". Mechanically, each token produces three vectors: a query ("what am I looking for?"), a key ("what do I contain?") and a value ("what do I pass on?"). A token's query is compared with every earlier token's key. The similarity scores, normalized with softmax, decide how much of each value flows into that token's new representation:

Attention(Q, K, V) = softmax(Q·Kᵀ / √d) · V

Models run many attention "heads" in parallel, each free to learn different relationships (syntax, coreference, and so on), and stack many layers, so meaning is built up gradually. Comparing every token with every other token costs O(n²) in the sequence length. That's the main reason long inputs are expensive and why context windows have limits.

Step 4: predicting the next token

After the last layer, the model turns the final token's representation into a score for every token in its vocabulary (typically 50,000–200,000 of them). Softmax turns the scores into a probability distribution, for example "Paris" 92%, "the" 3%, and so on. One token is chosen and appended, and the whole process repeats for the next token. Generating 500 tokens means 500 of these steps. Implementations cache earlier computation (the KV cache) so each step doesn't recompute everything from scratch.

How the next token is chosen (why outputs vary)

• Greedy / temperature 0: always take the most likely token. Near-deterministic, but it can be repetitive.
• Temperature: rescales the probabilities. Low (0.2) is focused and consistent; high (1.0+) is more varied and creative, and more error-prone.
• Top-p (nucleus) sampling: sample only from the smallest set of tokens whose probabilities add up to p (e.g. 0.9), ignoring the long tail of unlikely ones.

For extraction, classification and code, use low temperature. For brainstorming and creative writing, use higher.

How a model is trained

1. Pre-training: next-token prediction over trillions of tokens of text and code. The model adjusts its billions of weights to reduce prediction error. It learns grammar, facts, styles and reasoning patterns, because all of these help predict text. The result is a powerful but raw "document completer".
2. Instruction tuning (supervised fine-tuning): further training on example conversations of instructions and good responses, so the model follows requests instead of just continuing text.
3. Preference tuning: humans (or AI feedback) compare responses, and the model is optimized toward preferred ones. Methods include RLHF (reinforcement learning from human feedback) and DPO. This shapes helpfulness, tone and safety behaviour.

Why LLMs behave the way they do

• Hallucinations: the model produces the most plausible continuation, not a verified fact. When its training data is thin or the question is outside it, a fluent, confident, wrong answer is still "likely text". Remedies include providing sources in the prompt (RAG), asking for citations, and verifying outputs.
• Knowledge cut-off: a model knows only what was in its training data, up to a certain date, unless you supply newer information in the prompt or it has tools like web search.
• Context window: the maximum number of tokens (input + output) the model can consider at once. It has no memory between separate API calls. "Memory" in chat apps is the app resending earlier messages.
• Sensitivity to wording: the prompt is the context the prediction is conditioned on. Clearer instructions, examples and structure change the probability distribution, which is why prompt engineering works.
• Uneven long-context use: models can pay less attention to information buried in the middle of very long inputs, so put key instructions and facts where they're easy to find.

Follow-up questions this topic invites — and their answers

Q: If an LLM only predicts the next token, how can it reason? A: To predict text well (proofs, code, explanations), the model has to internally represent the patterns that produce such text. Step-by-step reasoning in its own output also helps, because each written step becomes context for the next prediction, which is why "think step by step" prompts improve multi-step tasks. It's powerful but not guaranteed to be correct, so verify important results.

Q: What is a context window, and why is it limited? A: It's the maximum tokens the model can attend to at once. Attention compares tokens pairwise, so compute and memory grow roughly quadratically with length, and models are trained with a maximum length. Longer windows are available but cost more and can be slower.

Q: Why does the same prompt give different answers? A: Output tokens are sampled from a probability distribution. With temperature above zero, different runs pick different tokens, and one different early token changes everything after it. Lower temperature for consistency. Even at zero, small non-determinism can come from how computations are batched on hardware.

Q: What's the difference between pre-training and fine-tuning? A: Pre-training learns general language and knowledge from huge unlabelled text via next-token prediction, which is expensive and done once. Fine-tuning continues training on a much smaller, targeted dataset to change behaviour, such as following instructions, a style, or a narrow task. It's far cheaper.

Q: What are embeddings used for outside the model itself? A: Separate embedding models turn whole sentences or documents into vectors whose distances reflect meaning. They power semantic search, clustering, deduplication, recommendations, and the retrieval step of RAG systems.