A looks like magic from the outside and like a spreadsheet from the inside. Every answer it gives, every line of code it writes, comes from one repeated operation: given a stretch of text, guess the next fragment, then add that fragment and guess again. Everything else, the training runs that cost millions, the routing tricks, the reasoning behaviour, is built on top of that single move.
In this article. A working tour of a large language model from the inside: how the field arrived here, what a model is actually made of, how one is trained and then post-trained into an assistant, the design choices (dense versus , reasoning, specialisation) that decide what it costs and what it is good at, where it falls down, how you run and customise one, and what it now takes to build a small one from scratch.
How we got here
The pieces arrived one at a time over roughly a decade. In 2013, word2vec showed that you could turn words into vectors of numbers such that similar words sat near each other, and that arithmetic on those vectors carried meaning . That gave models a usable representation of language. The next problem was sequence: how to read a sentence in order and produce another sentence in order. Recurrent networks did this, reading one word at a time and carrying a running summary, but they struggled with long inputs because everything had to be squeezed through a single fixed-size memory. In 2014, an attention mechanism was bolted onto these translation models, letting the decoder look back at any part of the input rather than relying on one compressed vector .
The 2017 paper that reset the field took the attention idea and threw away the recurrence around it. "Attention Is All You Need" proposed the transformer: a network built entirely on attention, with no sequential bottleneck, which meant it could be trained on far more data in far less time because the work parallelised cleanly across hardware . The title was a statement of subtraction. The mechanism that had been an add-on became the whole machine.
What followed was a scaling story. In 2018 the architecture split into two camps: like BERT, tuned for understanding text, and decoder models in the line, tuned for generating it . In 2020, GPT-3 pushed the decoder side to 175 billion parameters and demonstrated something that surprised people: a big enough model could be told what to do with a few examples in the , no retraining required . That property, in-context learning, is most of why prompting works at all. Two years later, wrapped a tuned version of this in a chat box and the technology stopped being a research curiosity .
The open-weight story has a better plot than people remember. Meta released the first to researchers on 24 February 2023, gated behind an application form and a non-commercial licence . About a week later, on 3 March, someone posted a torrent of the weights on 4chan, and a user opened a pull request on Meta's own repository to add the magnet link, ostensibly to save bandwidth . Meta filed takedown requests; the weights were already everywhere. Within weeks, researchers at Stanford had fine-tuned the leaked model into an instruction-follower for a few hundred dollars. Meta's response was not to retreat but to lean in: Llama 2 arrived in July 2023 as a deliberate open release . The accident seeded a community that has been releasing competitive open models ever since.
What a model is made of
Strip away the interface and a trained model is a large bag of numbers called parameters, or weights. A 7-billion-parameter model is 7 billion of these numbers; GPT-3 had 175 billion of them . They are not a database of facts you can look things up in. They are the learned settings of a function that, given some text, produces a probability distribution over what comes next.
The text the model sees is , not words: fragments produced by a scheme called that splits text into common chunks, so "tokenisation" might be one token and an unusual surname might be four.
Each token is then turned into an : a long list of numbers that acts as the token's coordinates in a space with thousands of dimensions. Nobody can picture thousands of dimensions, but the intuition survives in three. Imagine every word as a pin on an enormous map, arranged during training so that words with similar meanings end up near each other. More surprisingly, the relationships between words settle into consistent directions across that map. The step from "king" to "queen" points the same way as the step from "man" to "woman". The model is never told this; it falls out of the work of predicting text.
The number of tokens a model can consider at once is its , and it is a hard limit: text beyond it is invisible to the model unless something outside the model puts it back in.
One objective
A base model is trained to do exactly one thing, predict the next token, billions of times over. Summarising, translating, writing code, answering questions: none of these are separate capabilities that were programmed in. They are all the same prediction, applied to different starting text.
How they are made
Training starts with data, and the quantities are hard to picture. -V3, a recent open model, was pre-trained on 14.8 trillion tokens . The mix is mostly web text, filtered and de-duplicated, plus code, books, and reference material. The quality of that filtering now matters as much as the raw volume; a smaller, cleaner corpus often beats a larger, dirtier one.
The model itself is the transformer described earlier, repeated in a stack. The part worth understanding is attention. As the model reads, each word looks around at the other words and decides which of them matter for working out what comes next. Take the sentence "the trophy didn't fit in the suitcase because it was too big." Attention is how the model works out that "it" means the trophy and not the suitcase, by letting "it" weigh the other words and lean on the relevant one. For anyone who has written database queries: attention is a soft self-join, except the match is on learned similarity rather than exact keys, and the weights are graded rather than all-or-nothing. This runs for every word, at every layer. Between the attention layers sit ordinary feed-forward layers that do much of the storage and computation, and stacking dozens of these blocks gives you the network.
How big should the model be, and how much should you train it on? For a while the assumption was that bigger was simply better: add parameters and the model improves. The 2022 work put numbers on this: for a fixed compute budget, model size and training data should grow together, at a ratio of roughly 20 training tokens per parameter . By that measure GPT-3, trained on around two tokens per parameter, was badly undertrained for its size. It was too big for the data it saw . The rule of thumb is not a law, and in practice teams now deliberately overtrain small models on far more than 20 tokens per parameter, because a smaller model is cheaper to run for years after the one-off training cost is paid. The maths shifted the whole industry toward smaller, better-fed models.
A model behaves more like a student than a hard drive: a bigger brain is wasted if you never give it enough to read.
A freshly pre-trained model is called a base model, and it is a strange thing to talk to. It will happily your text, but it has no notion of being a helpful assistant; ask it a question and it might reply with five more questions, because that is a plausible continuation of a list of questions. Turning it into something useful is post-training, and it happens in stages.
shows the model thousands of example conversations so it learns the shape of a helpful reply. Preference tuning then nudges it toward the answers people prefer, either through or a simpler method called direct preference optimisation. This stage is where most of a model's personality and refusals come from. It is also a small fraction of the total compute. The expensive part is pre-training; post-training is the polish.
The design choices that matter now
A few choices in how a model is trained and structured shape what it costs to run and what it is good at.
The first is dense versus Mixture-of-Experts. A dense model uses all of its parameters for every token. An MoE model splits much of the network into many "expert" sub-networks and, for each token, a small picks just a few to run. Mixtral, the model that brought the pattern into in December 2023, has 46.7 billion total parameters but uses only 12.9 billion per token, so it computes at the speed of a 13-billion model while holding the knowledge of a much larger one . By 2025 the design was the default for any serious model. 's gpt-oss-120b, its first open-weight release since GPT-2, activates 5.1 billion of its 117 billion parameters ; Moonshot's Kimi K2 runs 32 billion of roughly a trillion, fewer than one parameter in thirty active at a time . The everyday version: a dense model is one generalist who personally handles every question; an MoE model is a large practice that keeps many specialists on staff but only calls in the few a given case needs.
| Model | Total params | Active per token | Held in memory | Computed per token |
|---|---|---|---|---|
| Llama 2 70B (dense, 2023) | 70B | 70B | all 70B | all 70B |
| Mixtral 8x7B (2023) | 46.7B | 12.9B | all 46.7B | 12.9B |
| DeepSeek-V3 (2024) | 671B | 37B | all 671B | 37B |
| gpt-oss-120b (2025) | 117B | 5.1B | all 117B | 5.1B |
| Kimi K2 (2025) | ~1T | 32B | all ~1T | 32B |
Total and active parameter figures from each model's release notes [R10, R11, R17, R18].
MoE saves you compute per token, not memory, and that is the catch that trips up capacity planning. The router can pick any expert for any token, so every expert has to be loaded and ready. You pay for the active parameters in speed and for the total parameters in VRAM.
MoE doesn't shrink the box
It is tempting to size hardware off the active parameter count, because that is what runs per token. Memory does not work that way. Kimi K2 activates 32B parameters but you must hold all trillion in memory to serve it . MoE moves the bottleneck from compute to memory; it does not remove it.
The second shift is reasoning, or . OpenAI's o1, released in September 2024, was trained to produce a long internal before answering, spending more computation at the moment of the query rather than only at training time . In January 2025, DeepSeek published R1 and, unusually, the method behind it: the reasoning behaviour could be coaxed out almost entirely through reinforcement learning, with the model learning to think longer on hard problems and to check its own work . This is a real change in where the cost sits. A can burn far more tokens answering one question than a standard model does, which is excellent for maths and code and wasteful for "what's the capital of Norway". By 2026 this is the norm rather than a novelty: open models such as gpt-oss ship with reasoning levels you dial up for hard problems and down for easy ones .
The third is general versus specialised. The coding variants you see, GPT-5- and its successors, are not smaller or stripped-down models. Each is the same base model with extra post-training: more code in the mix, and reinforcement learning on the actual loop of software work, where the model proposes a change, runs the tests, reads the failure, and tries again . What gets added is task-specific skill and tool fluency, not a subtraction of general knowledge. The reason is practical. Understanding a vague request and reasoning about an unfamiliar codebase both lean on broad language and world knowledge, so coding ability and general reasoning tend to rise together rather than trade off. OpenAI's own line bears this out: the coding-tuned work was folded back into the general GPT-5.5, pitched as strong at both . So "how little general knowledge can you keep" has an awkward answer: less than you might hope. Cut enough that the model forgets how an HTTP request or a binary tree works and it can no longer follow what you are asking it to build.
Where they fall down
Two failure modes follow straight from how a model works, and both are worth understanding before you trust one with anything that matters.
The first is : a model stating something false with complete confidence. It happens because the model is built to produce a plausible next token, not a true one, and nothing in the basic mechanism checks the output against the world. The way models are trained and graded makes it worse. A benchmark that scores "I don't know" the same as a wrong answer teaches the model to always answer, because a confident guess sometimes scores and an admission of ignorance never does . The result is fluent, well-formed citations to papers that were never written and methods that do not exist. Reasoning reduces it, and grounding the model in retrieved sources reduces it more, but no current model is free of it .
The second is arithmetic and multi-step logic. Ask a model to multiply two large numbers or work through a puzzle with several dependent steps, and it will often hand back a confident, wrong answer that looks like working. The cause is the same: it is matching patterns from training, not computing. Apple's GSM-Symbolic study made this concrete. Change only the numbers in a grade-school maths problem and accuracy wobbles; add a single clause that sounds relevant but changes nothing, and accuracy across every leading model fell by as much as 65% . The same brittleness shows up with riddles: alter a famous one slightly and the model often answers the original, memorised version rather than the one in front of it.
The model is good at deciding what to compute and poor at doing the computation; the tool is the reverse.
There are two fixes, which is why this is not a dead end. The first is reasoning: a model that works step by step is far more reliable on maths than one that answers in a single pass. The second, and the more dependable, is to stop asking the model to compute at all. Give it a calculator. Let it write a few lines of code, run them in a sandbox, and read back the exact result, which is what the tool in nanochat and the tools in any coding are for .
A fluent reasoner, not an oracle
A model is a strong text predictor with broad knowledge, not a database and not a calculator. Much of the engineering around it exists to cover exactly these gaps: retrieval for facts, tools for computation, reasoning for multi-step work.
Running one
, meaning running a trained model, is the same next-token loop the model was trained on, repeated. The model predicts a distribution over the next token, one is chosen, it is appended to the text, and the process repeats. The choosing is governed by a setting: low temperature picks the most likely token almost every time and reads as deterministic and dry; higher temperature samples more freely and reads as more varied, sometimes to the point of nonsense. To avoid re-reading the whole conversation on every step, the model keeps its working notes in what is called the , which is why generation speeds up after the first token and why long contexts eat memory.
The number that decides whether you can run a model on your own machine is straightforward arithmetic. Each parameter stored at 16-bit precision takes two bytes, so a 7-billion-parameter model needs about 14 GB of memory just for its weights . shrinks this by storing each parameter at lower precision, much like saving a photograph at a lower resolution to make the file smaller. At four bits, roughly half a byte per parameter, the same 7B model drops to around 4 GB and fits a consumer graphics card, with only a few percent of quality lost on most .
| Model size | 16-bit (≈2 bytes/param) | 4-bit (≈0.5 bytes/param) |
|---|---|---|
| 7B | ~14 GB | ~4 GB |
| 13B | ~26 GB | ~7 GB |
| 70B | ~140 GB | ~35 GB |
Weights only; the KV cache and runtime overhead add several GB more, especially at long context. Figures follow the standard bytes-per-parameter calculation [R14]. For MoE models, size against total parameters, not active ones.
This single calculation explains the whole local-model hobby. Tools like Ollama and llama.cpp package the quantised weights so that running a capable model at home is a download and a command, not a research project. It also explains why a 70B model is a different proposition: even quantised it wants 35 GB, which is past a single consumer card and into multi- or rented territory.
Customising behaviour and knowledge
When a model does not do what you want, there is a ladder of options, and it pays to start at the cheap end.
The cheapest is prompting: tell it more clearly what you want, give it examples, give it the relevant material in the context window. Next is retrieval-augmented generation, where you fetch relevant documents at query time and paste them into the prompt, so the model answers from current, specific material it was never trained on. Above that sits , where you adjust the weights themselves on your own examples; methods like LoRA and make this affordable by training a small set of additional parameters rather than the whole model. At the top, and almost never the right answer for a single team, is continued pre-training on a large new corpus.
The pattern, after a few years of people reaching for the impressive-sounding option first, is that most problems labelled "we need to fine-tune" turn out to be a prompting or retrieval problem. Fine-tuning changes how a model behaves and what style it adopts; it is a poor and expensive way to teach it new facts, which retrieval does better and keeps current. Reserve fine-tuning for when you need a consistent format or behaviour that prompting cannot hold reliably.
A good deal of what makes a model useful is not in the model at all. The tools it can call, the documents it can retrieve, the memory it carries between turns, and the loop that lets it act and observe results all live outside the weights, in the scaffolding around them.
Smaller, specialised models
All of that assumes you want a large, general model in the first place. Often you do not. A parallel world rarely makes the headlines: small models trained to do one narrow job. Many are encoder models from the BERT branch , built to read and label text rather than generate it. They run from a few million to a few hundred million parameters, orders of magnitude below anything in the table earlier, and they need no GPU. A sentiment or topic classifier, an entity extractor, an embedding model for search, a speech-to-text model: each fits in tens of megabytes and answers in well under a second on an ordinary laptop.
Small enough, in fact, to run inside a web page. 's Transformers.js loads these models straight into the browser and runs them on the CPU through WebAssembly, or on the GPU through WebGPU, with no server in the loop . The model downloads once, is cached, and then works offline, so a site can do translation, classification, or transcription with the user's text never leaving their machine.
Building one is within reach in a way that pre-training a large model is not. You rarely start from scratch. You take a small pre-trained base and fine-tune it on a few thousand labelled examples, which finishes on a laptop or a free notebook in minutes to hours. Raschka's book covers this path directly, turning a small model into a text classifier .
Right-size the model
For a narrow, well-defined task, a small specialised model is usually faster, cheaper, more private, and more predictable than calling a frontier model through an API. Reaching for the largest model by default is often the expensive way to solve a problem a small local classifier would handle.
Rolling your own
The best way to turn these ideas into running code is to build a small model yourself. Two routes are realistic, depending on whether you want to understand every line or watch the whole pipeline run.
The understanding route is Sebastian Raschka's book Build a Large Language Model (From Scratch). It walks you through coding a GPT-2-class model in plain PyTorch (tokenisation, the attention mechanism, the transformer block, pre-training, then fine-tuning), and the model is small enough to train on an ordinary laptop . You need intermediate Python and some patience. Andrej Karpathy's older nanoGPT, around 300 lines that train a character-level model on a tiny corpus, is the even-lighter version if you just want to read the training loop end to end.
The whole-pipeline route is Karpathy's nanochat, released in October 2025. A single script runs the entire lifecycle: train a tokeniser, pre-train the base model, mid-train it on conversational and data, run supervised fine-tuning, optionally do a round of reinforcement learning on grade-school maths, evaluate, and serve the result in a small chat web interface . Each stage lines up with a concept from earlier: tokenisation, pre-training, post-training, and serving, running in order on a model you made.
This is not an at-home setup. You rent an eight-GPU at roughly $24 an hour; the base run finishes in about four hours for around $100, and Karpathy's own run came in at three hours fifty-one minutes . Turn one dial and the model scales with the budget.
| Budget | Wall-clock (8×H100 node) | Roughly what you get |
|---|---|---|
| ~$100 | ~4 hours | a small model that holds a conversation |
| ~$300 | ~12 hours | about GPT-2 level on the CORE benchmark |
| ~$1,000 | ~40 hours | basic maths and coding ability |
nanochat speedrun tiers as reported by the author [R15].
The cost collapse
nanochat reaches GPT-2-class capability for about $48 of compute. The same level of model reportedly cost on the order of $43,000 to train in 2019 . That cost floor has dropped by roughly three orders of magnitude in six years.
Most readers will run neither, and that is fine. Reading the speedrun script top to bottom is itself the quickest way to see how the pieces fit together, and that the whole assembly now fits in a rented afternoon.
One operation, repeated
One operation, repeated: predict the next token. Scaled up with the transformer, fed according to the token-per-parameter maths, shaped by post-training, made cheaper to run by quantisation and Mixture-of-Experts routing, and extended by reasoning and the tooling around the model. The architecture has been stable since 2017; the rest has been making it bigger, cheaper, and better wrapped.
Final thought
A technology that needed a research lab in 2020 now has a from-scratch version you can run for the price of dinner.
References22
- 1Mikolov, T., Chen, K., Corrado, G., Dean, J., "Efficient Estimation of Word Representations in Vector Space" (word2vec), 2013. arxiv.org ↗
- 2Wikipedia, "Attention Is All You Need" (for the 2014 Bahdanau et al. attention mechanism predating the transformer). en.wikipedia.org ↗ Accessed 2026-06-15
- 3Vaswani, A., et al., "Attention Is All You Need", 12 June 2017. arxiv.org ↗
- 4Devlin, J., Chang, M.-W., Lee, K., Toutanova, K., "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding", 2018. arxiv.org ↗
- 5Brown, T. B., et al., "Language Models are Few-Shot Learners" (GPT-3, 175B parameters), May 2020. arxiv.org ↗
- 6Wikipedia, "ChatGPT" (public launch, late November 2022). en.wikipedia.org ↗ Accessed 2026-06-15
- 7DeepLearning.AI, The Batch, "How Meta's LLaMA NLP Model Leaked" (release to researchers 24 February 2023 under a non-commercial licence; leak about a week later), March 2023. deeplearning.ai ↗
- 8Wikipedia, "Llama (language model)" (initial release 24 February 2023; weights torrent shared via 4chan 3 March 2023; Llama 2 released July 2023). en.wikipedia.org ↗ Accessed 2026-06-15
- 9Hoffmann, J., et al., "Training Compute-Optimal Large Language Models" (Chinchilla; ~20 tokens per parameter; 70B model on 1.4T tokens), March 2022. arxiv.org ↗
- 10Mistral AI, "Mixtral of experts" (46.7B total / 12.9B active parameters; released 8 December 2023). mistral.ai ↗
- 11DeepSeek-AI, "DeepSeek-V3 Technical Report" (671B total / 37B active parameters; pre-trained on 14.8T tokens), December 2024. arxiv.org ↗
- 12Heigold, S., "How to Train LLMs to 'Think' (o1 & DeepSeek-R1)", Towards Data Science (cited for OpenAI o1's release in September 2024), 2025. towardsdatascience.com ↗
- 13DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning", January 2025 (peer-reviewed version in Nature, 2025). arxiv.org ↗ · nature.com ↗
- 14llmhardware.io, "LLM Quantization Explained: Q4, Q8, FP16 and VRAM Tradeoffs" (16-bit ≈ 2 bytes/parameter so a 7B model ≈ 14 GB; 4-bit ≈ 0.5 bytes/parameter so ≈ 3.5–4 GB). llmhardware.io ↗ Accessed 2026-06-15
- 15Karpathy, A., "nanochat" repository and "Introducing nanochat" discussion (full pipeline; 8×H100 node at ~$24/hr; base speedrun ~4 hours / ~$100, measured 3h51m; ~$300 and ~$1,000 tiers; GPT-2-class capability for ~$48 versus ~$43,000 to train in 2019), October 2025. github.com ↗ · github.com ↗
- 16Raschka, S., *Build a Large Language Model (From Scratch)*, Manning, 2024 (GPT-2-class model in PyTorch, runs on a laptop; covers tokenisation, attention, pre-training, and fine-tuning). manning.com ↗
- 17OpenAI, "Introducing gpt-oss" and "gpt-oss-120b & gpt-oss-20b Model Card" (gpt-oss-120b: 117B total / 5.1B active parameters; OpenAI's first open-weight release since GPT-2; released 5 August 2025 under Apache 2.0; ships with low/medium/high reasoning levels). openai.com ↗ · arxiv.org ↗
- 18Kimi Team (Moonshot AI), "Kimi K2: Open Agentic Intelligence" technical report (≈1.04 trillion total / 32 billion active parameters; 384 experts, 8 routed per token; pre-trained on 15.5 trillion tokens), 2025. arxiv.org ↗ · huggingface.co ↗
- 19Hugging Face, "Transformers.js" (JavaScript library that runs ONNX-exported models in the browser, on the CPU via WebAssembly or the GPU via WebGPU, with no server; supports classification, named-entity recognition, embeddings, translation, and speech recognition). github.com ↗ · huggingface.co ↗ Accessed 2026-06-15
- 20OpenAI, "Introducing upgrades to Codex" (GPT-5-Codex is a version of GPT-5 further optimised for agentic software engineering, trained on real-world engineering tasks) and "Introducing GPT-5.5" (combining the coding strength of the Codex line with general reasoning), 2025–2026. openai.com ↗ · openai.com ↗
- 21Kalai, A. T., Nachum, O., Vempala, S. S., Zhang, E., "Why Language Models Hallucinate", OpenAI, September 2025 (training and evaluation reward confident guessing over admitting uncertainty). openai.com ↗ · arxiv.org ↗
- 22Mirzadeh, I., et al. (Apple), "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models", October 2024 (accuracy drops by up to 65% when an irrelevant clause is added to a maths problem). arxiv.org ↗