How LLMs Work Explained Simply: Tokens, Context Windows, and Hallucinations

At some point in an interview for an AI enabled role, the conversation usually drifts from "what have you built" to "do you actually understand what's happening underneath it." Not a request for a computer science lecture, but a real, simple check: can you explain, in plain language, what a token is, why a model's context window matters, and why it sometimes states something confidently wrong.

Most candidates who have only used AI tools, without ever being asked to explain them, fumble this gently. They can operate the tool but cannot open the hood, even a few inches, which reads to an interviewer as surface level familiarity rather than genuine understanding. The gap is smaller than it looks. Four concepts, explained without math or jargon, cover almost everything that actually comes up.

This guide explains all four the way a good colleague would explain them over coffee, not the way a textbook would.

Why conceptual understanding gets tested, not just tool skill

A candidate who can operate a tool without understanding roughly how it works is in a specific kind of trouble: they can follow a known pattern successfully, but they cannot reason about a new situation the pattern does not cover. When a task fails in an unfamiliar way, understanding why the underlying system behaves the way it does is the difference between diagnosing the problem in two minutes and being stuck for an hour.

This matters more, not less, in procurement and operations roles specifically, because these roles increasingly involve explaining AI assisted work to people who do not use the tools themselves: a manager, a compliance reviewer, a supplier relations team. "The tool made a mistake" is not an acceptable explanation in a room like that. "The document was longer than the model's context window could hold in one pass, so it missed a clause near the end" is a real, credible explanation that only someone with basic conceptual understanding can give.

Interviewers test for this because it predicts something real: whether a candidate can be trusted to use AI tools independently, on tasks nobody has specifically walked them through, without supervision catching every mistake before it matters.

There is also a simpler, more immediate reason this comes up so often: these four concepts explain nearly every strange or surprising thing a new user runs into during their first weeks with an AI tool. Why did it forget something I told it earlier in a long conversation. Why did it confidently give me a wrong date. Why does the same question sometimes get a slightly different answer. A candidate who can answer these questions for themselves troubleshoots faster and asks for help less often, which is a real, practical advantage independent of any interview.

Tokens: the units a model actually reads

A language model does not read text the way you do, letter by letter or word by word. It breaks text into small chunks called tokens, roughly word sized pieces that might be a whole short word, part of a longer word, or a punctuation mark. "Procurement" might be one token or split into two or three, depending on how common that exact word is in the model's training. You do not need to memorize how the splitting works. You need to know that it happens, because it explains two things that otherwise seem strange.

First, it explains why AI tools are priced and measured in tokens rather than words or pages, since the model's actual unit of work is the token, not the word. Second, it explains why very unusual words, names, technical jargon, or a language the model saw less of during training, sometimes get handled slightly less reliably than common English words: they often break into more, smaller tokens, giving the model less of a clean pattern to work from.

Context window: the model's working memory for one conversation

A context window is the total amount of text, measured in tokens, a model can actually consider at one time within a single conversation or request. Think of it as a desk: everything currently on the desk is available to work with, but anything that never made it onto the desk, or that got pushed off to make room for something new, is simply not part of what the model is reasoning about right now.

This has a direct, practical consequence that explains a lot of confusing AI behavior. If you paste a 400 page contract into a tool with a context window too small to hold all of it, the model may only ever actually "see" the first portion, and any answer about a clause near the end is either missing entirely or based on a guess rather than the real text. This is not the model being careless. It is the model correctly reporting on the only part of the document that was actually on its desk.

The practical fix is straightforward once you understand the cause: for long documents, either use a tool with a context window comfortably larger than the document, or deliberately break the document into sections and process each one, the same chaining technique covered in the seven prompt engineering skills employers test.

Training versus inference: two very different phases

These two words get used loosely, and mixing them up leads to real misunderstandings about what a model can and cannot do. Training is the one time, resource intensive process where a lab teaches a model general patterns of language and reasoning, using enormous amounts of text, over weeks or months, before the model is ever released to the public. Inference is what happens every single time you send the model a message afterward: it applies what it learned during training to your specific input, in seconds, without learning anything new from that one conversation.

This distinction explains a specific, common confusion: a model does not "remember" your conversation from yesterday unless the tool you are using explicitly saves and resends that history, because inference does not change the underlying model at all. It also explains why a model's knowledge has a cutoff date tied to when its training data was collected, and why asking it about something that happened after that cutoff, without giving it the information directly, will produce either an honest "I don't know" or, less helpfully, a guess.

Hallucination: why a model states something confidently wrong

A hallucination is when a model produces a statement that sounds fluent and confident but is not actually true or not actually supported by anything it was given. This is the single most consequential concept in this guide for real workplace use, because a hallucinated number in a procurement report can be acted on before anyone notices it was never real.

The underlying cause connects directly back to training and inference. A model does not store facts the way a database does, in a lookup table it can check for certainty. It stores patterns, learned during training, about what kind of text plausibly follows other text. When it is asked something it genuinely does not have reliable information about, it does not have a built in mechanism to simply say "I don't know" by default, because generating a plausible sounding continuation is what the underlying pattern matching is built to do. The result can be a specific, confident, entirely invented number, citation, or fact, delivered in exactly the same tone as a correct one.

This is worth sitting with for a moment, because it is genuinely counterintuitive the first time it clicks. A wrong answer from a person usually comes with some tell, hesitation, a qualifier, a change in tone. A hallucinated answer from a model carries none of those tells, because the model is not experiencing uncertainty the way a person would, it is simply completing a pattern, and a wrong completion looks exactly as fluent as a right one. This is precisely why the verification habit matters more here than it would with a human colleague giving you an answer.

Practice these interview questions

You don't need to explain transformer architecture to pass this kind of interview question, but you do need to show you understand the practical mechanics well enough to make good decisions with these tools. Work through your own plain-language answer first, then compare it with the sample.