AI Learning Hub › Learn AI › Self-Paced Courses › Introduction to AI › Beginner › How Generative AI Works

Level 2 of 4 · Beginner · Lesson 2 of 6

How Generative AI Works

A conceptual, nontechnical explanation of how a typical generative language-model assistant is trained and how it produces output.

About 20 minutes

Skip to lesson

By the end of this lesson you should be able to

  • Describe, in order, how a language model is trained, adjusted, and then used
  • Explain tokens, parameters, and the context window in plain language
  • Distinguish training, the context of a conversation, and application memory, and check an answer’s claims against the sources available

Why this matters

Nobody writes the answers a chat assistant gives. Yet it can explain a doctrine, draft an email, or summarize a reading in seconds. Knowing roughly how that happens helps you anticipate the kinds of errors to expect and what you need to check. Knowing the process does not reveal the origin of each claim in a particular answer. Comparing the answer with sources helps you assess its support, not reconstruct the model’s internal process. It also clears up common misunderstandings: that the system looks answers up in a stored database, that uploading a file teaches it permanently, or that it is “just autocomplete.”

This lesson follows a large language model as it typically works inside a chat assistant. Products differ in the details, and other kinds of generative AI work differently, as a note below explains.

From training to answer, in five steps

1. Training adjusts the model’s parameters

The model is shown an enormous amount of text, broken into small pieces called tokens. Over and over, it predicts the next token in a passage, compares its prediction with the token that actually came next, and a training algorithm nudges the model’s internal numbers so that the right answer becomes a little more likely. Those numbers are its parameters, and large models have billions of them. No one sets them by hand.

What training leaves behind is a set of adjusted numbers that encode patterns of language and information—not a filing cabinet of the documents it read.

2. Post-training shapes its behavior and abilities

A model that can only continue text is not yet a helpful assistant. Developers train it further—for example on examples of good responses, on feedback about which answers people prefer, or on particular tasks such as coding—so that it follows instructions, keeps to a conversational format, declines some requests, and performs some tasks better. Methods vary between developers, and this stage can change what a model is able to do, not only its manner.

3. At use, it reads the context it is given

When you ask a question, the application sends the model a context: your message, plus whatever the application selects, such as the product’s own instructions, earlier turns of the conversation, or a document you uploaded—in full, or as a summary or selected passages. Systems that handle images or audio include those too. The context window is the limit on how much the model can take into account at once, measured in tokens; for many models it covers the response being written as well as the input. Material the application leaves out entirely plays no part in the answer, but a summary or excerpt it supplies does.

The answer is shaped by both: the patterns in the parameters from training, and the context for this request.

4. It generates one token at a time

Using its trained parameters, the model calculates how likely each possible next token is, given everything in the context. One token is selected and added, and the process repeats until the response is finished. The selected token is not always the single most likely one: many applications sample from among the likely candidates, which reads more naturally and is one reason the same question can get different answers.

5. The application may add tools

Many products wrap the model in other software: a web search, a search of a document collection (often called retrieval), a calculator or code runner, or a memory of your preferences. These are features of the application around the model. They can put fresh, relevant material into the context, but they do not replace what the model learned in training: it still generates the answer from that context together with its trained parameters.

The terms, briefly

Token
The unit a model reads and writes. A token can be a whole word, part of a word, a single character, or punctuation, so a long or unusual word may take several tokens.
Parameters
The internal numbers adjusted during training. The “large” in large language model refers largely to how many there are.
Embedding
A numerical representation of a token inside the model: a long list of numbers. Training tends to give tokens used in similar ways similar representations.
Attention
The step that lets the representation of each token be adjusted by the other tokens around it, so that “bank” near “river” is treated differently from “bank” near “loan.”
Inference
Running a trained model to produce output, as opposed to training it.
Context window
The limit, measured in tokens, on how much the model can take into account in a single request. Depending on the model, it may count the response as well as the input.

Sources for these definitions: Google’s Introduction to Large Language Models (opens in a new tab) (tokens can be words, subwords, or single characters; applications may choose the most likely prediction or sample among likely ones) and 3Blue1Brown, “Large Language Models explained briefly” (opens in a new tab) (parameters, training, embeddings, and attention). Google’s page on fine-tuning (opens in a new tab) describes further training on task-specific examples.

Worked example: likely is not the same as true

A constructed illustration; the continuations are examples, not output recorded from any model.

Suppose a model has to continue this sentence, with nothing else in its context:

The library is open on Saturdays until…

Plausible continuations include:

  • “5 p.m.”
  • “6 p.m., except in the summer.”
  • “noon during exam periods.”
  • “midnight.”

Depending on the tokenizer, a continuation may contain one token or several. The model builds the continuation one token at a time, each time assigning probabilities to the possible next tokens. Those probabilities describe how plausible a piece of text is, given the patterns the model learned; they are not a measure of how likely it is that the library actually closes at that time.

No particular library or authoritative current schedule has been supplied. These plausible completions alone do not establish the actual opening hours or what informed the answer. Depending on the system, it might produce specific hours anyway, ask which library you mean, say it cannot tell you, or, if it has a search tool, retrieve the library’s hours page. The review problem is the missing authoritative schedule: without one, any specific time is unsupported. If the application does place the current hours page in the context, the answer can be based on that page—and the page, not the model, is what you would check it against.

What it is not doing

It is not looking up a stored answer. Output is generated fresh from learned patterns and the current context. A model can sometimes reproduce passages from its training data—researchers call this memorization, and it raises privacy and copyright questions—but it is not pasting answers from a database of documents. (See NIST AI 600-1 (opens in a new tab), sections on data privacy and intellectual property.)

It is not “just autocomplete”—or rather, that phrase describes the mechanism while hiding the scale. Predicting the next token well, across billions of parameters and with attention over a long context, turns out to support translation, summarization, explanation, and drafting. The same mechanism also explains the central weakness: the model is trained to produce likely text, and nothing in that mechanism by itself checks the text against the world.

Not all generative AI predicts words. Image generators, for instance, often use diffusion models, which learn to turn random visual noise step by step into a clear image. (See Stanford HAI, “What are Diffusion Models?” (opens in a new tab)) Designs also differ among language models; the five steps above are a typical outline, not a description of any one product.

Training, context, and memory are different things

Training
What it is: patterns built into the parameters.
When it changes: when the developer trains the model further or releases a new version. It reflects data up to some point in time.
Context
What it is: what is in front of the model for this request: your messages, uploaded text, retrieved passages.
When it changes: every request. Start a new conversation and the earlier context is not there unless the application supplies it again.
Application memory
What it is: notes or past conversations a product saves and later adds back into the context.
When it changes: when the product saves or deletes them. Whether it exists, and what it keeps, depends on the service and its settings.

Supplying a file or a correction in an ordinary chat normally changes the context used for the response, not the model’s parameters. Whether the service keeps that material, or may use it to train later models, is a separate question, governed by the service, any agreement your institution has with the provider, and your settings.

One provider’s documentation, for example, says it may use content from its services for individuals to train its models unless the user opts out, and by default does not use content from its business and education workspaces. (OpenAI Help Center, “How your data is used to improve model performance” (opens in a new tab), checked September 24, 2026.) So “it does not retrain the model on the spot” is not a promise of privacy, deletion, or exclusion from future training.

Optional: see it animated

This eight-minute video from 3Blue1Brown covers the same ground visually: next-word prediction, parameters, pre-training and human feedback, and a first look at attention. It is optional, and it briefly mentions a few more technical ideas (such as backpropagation) that you do not need for this lesson.

“Large Language Models explained briefly” (opens in a new tab) by Grant Sanderson (3Blue1Brown), published November 20, 2024; 7 minutes 58 seconds. YouTube lists English captions for it. For a text alternative, 3Blue1Brown publishes a written adaptation of the lesson (opens in a new tab) (text adaptation by Justin Sun), which is an adaptation rather than a transcript.

Try it: what does the flyer support?

A constructed scenario. The flyer, people, and response are fictional.

A program coordinator pastes the text of this flyer into a chat assistant and asks for a two-sentence announcement for a newsletter.

Lunchtime Talk · “Water Rights in the West” · Professor Dana Rivera · Tuesday, March 3, 12:45–1:50 p.m. · Room 190 · Lunch for registered attendees

The assistant replies:

Join us Tuesday, March 3, at 12:45 p.m. in Room 190 for “Water Rights in the West,” a talk by Professor Dana Rivera, a leading authority on Western water law. Lunch is available for registered attendees, so be sure to sign up.

  1. Which factual claims in the announcement does the flyer support?
  2. Which wording organizes or presents those facts without adding a new factual claim?
  3. Which claims does the flyer not support, and what should happen to them?
  4. If the coordinator starts a new conversation next week, what can and cannot be assumed about whether the flyer is still available to the assistant?
Discussion
  1. Supported by the flyer: the title, the speaker’s name, the date, the start time, the room, and lunch for registered attendees. Check each against the flyer anyway: a copied detail can still be copied wrongly.
  2. Presentation: “Join us,” the order of the details, and the sentence structure arrange the flyer’s facts without adding new ones. The closing nudge, “so be sure to sign up,” is close to presentation but rests on an inference—that people can still register. That is a reasonable reading of “registered attendees,” but the flyer does not say registration is open or how to do it, so confirm before sending.
  3. Not supported: “a leading authority on Western water law.” Nothing in the flyer says so. The flyer cannot tell you why the phrase appeared, and asking the assistant will not settle it: its account of its own process is also generated text. The phrase may be true or false. Confirm it from another source, such as the speaker’s faculty profile or the event’s host, or remove it.
  4. The scenario does not establish it. Supplying the flyer did not change the model’s parameters. Whether the application saves the conversation, adds it back through a memory or chat-history feature, or keeps it on the provider’s side depends on the service, any agreement, and the settings. Do not assume it is remembered, and do not assume it is gone. If the assistant needs the flyer next week, supply it again.

What to carry into review. Check which claims the available sources support. Do not assume that an unsupported statement is a remembered fact from training, and do not treat the assistant’s explanation of where it got an answer as a reliable provenance record. Some unsupported statements are legitimate inferences rather than inventions; check the premises and the reasoning before you rely on them.

Sources and further reading