LLM Basics: Tokens, Context Window, Temperature, Hallucinations

Four terms come up in almost every AI PM interview: tokens, context window, temperature, and hallucinations. Most candidates can half define them, and half a definition is exactly what gets you a two on the scorecard. What actually gets you hired is attaching a tradeoff and a real product decision to each one. That's the gap between knowing a word and being able to ship with it.

So let's cover all four properly. For every concept, four beats: what it is, the tradeoff it forces, why it changes a product decision, and one real number. Learn the shape once, and you can answer any technical round question about these on demand. Start with tokens. A token is the chunk of text the model actually reads and writes.

It's not a word and it's not a character. It's a piece produced by the tokenizer, roughly four characters, or about three quarters of an English word on average. Cat is one token. Tokenization might be three. Rare words, code, and non English text all split into more tokens per word. Now for the tradeoff. Everything the model does is priced and limited in tokens.

You pay per input token and per output token, and every model has a hard ceiling on how many it can handle at once. More tokens means more money and more latency. Fewer tokens means you had to cut context somewhere. So what does that mean for a PM? Token cost is your unit economics. If your feature stuffs a fifty page document into every request, you're paying for fifty pages on every single call.

That's the whole reason chunking, caching, and retrieval exist, to avoid resending tokens you don't need. When someone asks how you would reduce cost, the first lever is nearly always to send fewer tokens. And the number. As of 2025, a frontier model might cost on the order of a few dollars per million input tokens, and more for output.

A million tokens is roughly 750,000 words, about seven or eight full novels. So one heavy request that packs 200,000 tokens of context is not free, and at scale it's the line item your finance partner will ask you about by name. Next is the context window. This is the maximum number of tokens the model can consider at once, input plus output combined.

Think of it as the working memory of the model for a single request. Once you exceed it, the earliest tokens fall off the back, or the request just gets rejected. The tradeoff here is sneaky. A bigger window lets you pass more, like a whole codebase, a long transcript, or a stack of retrieved passages. But it costs more, it runs slower, and it has a real recall problem.

Models are measurably worse at using information buried in the middle of a long context than information at the start or the end. That's the lost in the middle effect. A bigger window is not automatically a smarter model. It's just more room, and the model uses that room unevenly. The PM implication is the one people miss. Do not treat a large context window as a replacement for retrieval.

If you dump 500,000 tokens in and the answer you need is on page 200, the model may miss it, and you paid full price to watch it miss. Well designed retrieval that puts the five most relevant passages near the top of the prompt often beats just using the big window. Window size is one factor in picking a model.

It's not a solution on its own. For the number, windows in 2025 run from about 8,000 tokens on older or small models, up to 128,000 or 200,000 on frontier models, and as high as a million on the largest. 200,000 tokens is roughly a 500 page book. But recall on a fact buried mid context can drop noticeably versus the exact same fact placed at the edges.

Which is exactly why saying we will just use the one million window is not a complete answer in the room. Third concept is temperature. Temperature is a setting, usually from zero to about two, that controls randomness in how the model picks its next token. At temperature zero the model is near deterministic, meaning it picks the single most likely next token every time.

As you raise it, the model samples more from lower probability options, so the output gets more varied and more surprising. The tradeoff is a straight line. Low temperature gives you consistency and repeatability, at the cost of sounding a bit samey and being less creative. High temperature gives you variety and creativity, at the cost of consistency, and past a point, coherence.

There's no correct temperature in the abstract. There's a correct temperature for a given job. And this is the fun one, because it's a product decision disguised as a knob. For a factual or compliance sensitive feature, like a code assistant, a data extractor, or a legal summary, you want low temperature so the same input gives a stable and defensible output.

For a brainstorming or creative feature, like marketing copy, story ideas, or name generation, you want higher temperature so users get variety and don't see the same phrasing twice. Shipping a legal tool at high temperature, or a brainstorm tool at zero, isn't a tuning detail, it's a product bug. The numbers to carry are these. Common defaults sit around 0.7.

Extraction and classification pipelines often run at zero to 0.3 for repeatability. Creative generation often runs 0.8 to 1.2. The exact value matters far less than knowing which direction your feature needs, and being able to say why. Fourth is hallucinations. A hallucination is output that's fluent, confident, and false. The model generates text that reads as authoritative but isn't grounded in any real source.

It'll invent a citation, a statistic, a court case, or an API that doesn't exist, and it'll do it in a completely calm voice. Here's the tradeoff that separates strong candidates. Hallucination is not a bug you fully remove. It's a property of how these models work. They predict likely next tokens, and they don't look up ground truth by default.

The more you constrain and ground a model to reduce it, with retrieval, low temperature, or say I don't know instructions, the safer it gets. But you can also make it more cautious and less fluent, and sometimes it'll refuse things it could actually have answered. Safety and helpfulness pull against each other right here. So the PM implication is about stakes.

Where the cost of being confidently wrong is high, think health, legal, finance, or anything a user acts on directly, you can't ship a bare model. You gate it. You ground answers in retrieved sources with citations, you set a confidence threshold below which the thing says I'm not sure or hands off to a human, and you keep an eval set that measures how faithful the model stays to its sources.

Where the cost of a wrong answer is low, like draft me a poem, you can let it run freer. The product question is never does it hallucinate. It's what does a hallucination cost here, and what gate do I put in front of it. The number is this. Even strong models hallucinate on a measurable share of factual questions, and the rate climbs on niche or recent topics the model wasn't trained on.

Grounding with retrieval and citations can cut the rate substantially, which is why adding RAG is the standard first mitigation. But it never reaches zero, so human in the loop for high stakes decisions stays in the design. Let me put all four together. You're the PM for an AI feature that answers employee questions about the HR policies of your company.

Policies are long, so you don't stuff all 200 pages into every request. That's token cost, and lost in the middle if you did. Instead you retrieve the three most relevant policy passages and pass just those. You run temperature at 0.2, because asking how many sick days I get has to return the same answer every time, not a creative variation.

You ground every answer in the retrieved passage and show the source, because a confidently wrong HR answer is a genuine problem for a real person. And you set one hard rule. If retrieval finds nothing relevant, the model says it couldn't find a policy on that and gives the contact for HR, rather than inventing one. Four concepts, one coherent design.

These four words are the atoms of the technical round. They show up directly in questions like define hallucinations, or what's the effect of context window size. And they're load bearing in every product sense answer that touches an LLM. If you can run define, tradeoff, PM implication, and number on demand, you clear the AI depth column without breaking a sweat.

Quick myth busting, because each of these is a trap. A token is not a word. It's roughly three quarters of a word, and it swings hard by language and content. Is a bigger context window strictly better? No, it costs more, runs slower, and recall sags in the middle. It complements retrieval, it doesn't replace it. Temperature zero makes the model consistent, not correct, and a consistently wrong answer is still wrong.

Can hallucinations be turned off? No, they can be reduced and gated but never eliminated, so design for the wrong answer instead of assuming it away. Put it in one flowing thought. Tokens are the meter, the context window is the working memory, temperature is the creativity knob, and hallucinations are confident wrong answers you gate but never fully remove.

Each one carries a tradeoff and a product decision, and if you can say both out loud, that's the whole round. Memorise this. For every concept, cover what it is, the tradeoff it forces, the product decision it changes, and one real number. Do that four times and you've answered the technical basics like someone who's shipped, not someone who's just read.