How AI writes its answers: one piece at a time

Same question, 2 different answers

Watch this part · 51 sSame question, two different AI answers. Why?

Ask an AI chat app this: "Plan 5 days in New Zealand, with the places where The Lord of the Rings was filmed." The plan might open at the Hobbiton set near Matamata, in the North Island. Open a new chat and send the same words. This time day 1 could be in the South Island.

The films were shot at more than 150 places across both islands, so both plans are reasonable. But which one is right, and is the app broken?

Nothing is broken. This is how these apps write, whether you use ChatGPT, Claude, Gemini or another. Once you see how, other things fall into place: why an answer appears word by word, why a long chat runs out, and why Thinking takes longer.

Autocomplete that has read a library

Watch this part · 1:15How AI writes answers: autocomplete that read a library

Type "see you" on your phone and the keyboard offers "tomorrow", "soon", "later". It is guessing the next word from the words before it.

A language modelLarge Language ModelA model trained on a huge amount of text to predict the next token. Further training turns it into an assistant that follows instructions. does a far bigger version of the same job. Given the text so far, it predicts what comes next. For the trip question it gives every possible next piece a score. "Day" might come first, "Here" close behind, "New" a little lower, and thousands of other pieces share what is left.

One piece is picked and added to the text. Then the whole text goes back in, and the next piece is picked. This repeats until the answer is finished.

That is why you see an answer arrive word by word. The app shows you the start while the rest is still being written.

The scores come from training, which is the subject of the next lesson. The point for now is what the model is not doing. It is not looking your answer up. It is predicting, piece by piece, what a good answer looks like. That is also why it can sound sure and still be wrong.

It reads pieces, not words

Watch this part · 1:31What are AI tokens? Explained in 90 seconds

Those pieces are called tokensTokenThe small piece of text a language model reads and writes, often part of a word. Prices, limits and the context window are all counted in tokens.. A token can be a whole word, part of a word, a single character or a punctuation mark. Even a space changes how text is cut.

Why not whole words? There are too many. English alone has over 500,000, before you count names and typing mistakes. Why not single letters? Every word would turn into a long chain, and the text would be too long to work with. Pieces are the middle ground. Common words stay whole. Rare words split into parts that still mean something, such as "annoyingly" into "annoying" and "ly".

As a rule of thumb for English, 1 token is about 4 characters, or three-quarters of a word.

Our own count

OpenAI publishes its tokenizerTokenizerThe part that cuts text into the tokens a model reads, and turns the model's tokens back into text. Each model family has its own., so we ran our sentences through it on 14 October 2026.

• The trip question is 20 tokens. Most are whole words, each with the space in front of it. The "5" is a token of its own.
• In "How many r's are in strawberry?", the word strawberry is 1 token.
• Typed alone, with no space in front, the same word is 3: "st", "raw" and "berry".

So when you ask how many r's are in strawberry, the model gets 1 piece. It never sees the letters. To count them it has to take that piece apart without being able to look at it. A question a child can answer tripped these models up for a long time for this reason.

The same sentence costs more in some languages

Take one line from the trip: "Where can I see the Hobbit village?"

It is the same question, and in Greek it takes more than twice the pieces. Each maker also cuts text in its own way, so one sentence can be a different number of tokens in ChatGPT, Claude and Gemini. Pictures and sound become tokens too. For Gemini a small image is 258 tokens and each second of audio is 32.

Tokens are the currency

Watch this part · 1:38Tokens are AI's money

Everything is counted in tokens: the words you send, the words you get back and, as you will see below, the thinking.

In the apps you do not see a price per token. You see limits, and the limits are counted in the same currency. Claude and Gemini both measure your usage by how long and how complex your conversations are. On both, the allowance refills every 5 hours, up to a weekly limit. So a plan works like a wallet that refills. Spend it on long, heavy chats and it runs out sooner.

There is a second limit of a different kind: how much one conversation can hold at once. This is the context windowContext WindowEverything a model can see at once while it writes an answer, counted in tokens. When the window is full, something has to be left out.. In the Gemini app it runs from 32,000 tokens with no plan to 1 million on the top plans. Claude's newest models hold up to 1 million on paid plans.

Here is the part that surprises people. With every message you send, the model reads the whole conversation again from the top: each earlier question, each earlier answer, then your new one. So a long PDF pasted into a chat is not paid for once. Its tokens come along with every message after it.

3 habits follow from this.

• Start a new chat when the topic changes.
• Paste the part you need, not the whole file.
• Do not drag a giant conversation along when a fresh one would do.

A weighted die picks each piece

Watch this part · 2:04Why AI gives a different answer every time: temperature explained

Back to the 2 trip plans. Where did the difference come from?

The simplest rule would be to take the top-scoring piece every time. That gives the same answer on every run, but on longer text it starts to repeat itself. So most apps roll a weighted die. The favourite gets the most sides, the runner-up fewer, and every piece with any chance at all gets a sliver.

Most rolls land on the favourite. Some do not. And one different pick early on changes everything after it, because each new piece is built on the ones before. Start with "Day 1: Auckland" and the plan heads for Hobbiton. Start with "Here is a plan" and it may open in the south.

A setting called temperatureTemperatureA setting that controls how much chance goes into each word a model picks. Low gives focused, predictable answers; high gives more varied ones. reshapes the die. Turn it down and the favourite gets even more sides, so answers are focused and predictable. Turn it up and the sides even out, so answers are more varied. DeepSeek, for example, recommends 0 for coding and maths and 1.5 for creative writing and poetry. You want the same answer to a sum every time, and a few surprises in a poem.

In the chat apps this is set for you. Developers used to set it themselves, and that is going away. Anthropic's newest Claude models return an error for anything but the default, Google deprecated it for Gemini in July 2026, Kimi K3 fixes it at 1, and DeepSeek's models ignore it in thinking mode. Even at 0, the exact same words are not guaranteed.

The cat sat on the …

Temperature1.0

• mat50%
• sofa22%
• floor17%
• roof7%
• keyboard4%
• moon1%

Pick the next word

Low temperature: the likeliest word nearly every time. High: the unlikely words get a real chance. The scores are made up to show the effect.

So a different answer is not a broken answer. You can use this.

• For ideas (a trip, a name, an opening line): ask 2 or 3 times and keep the best parts.
• For a fact: ask once, then check the source.

What "Thinking" is

Many AI apps now have a Thinking switch, under one name or another. The idea is the same in all of them. Before the final answer, the model writes rough notesChain of ThoughtHaving a model write out the steps of its reasoning before it gives an answer. Once a prompting trick, now built into "thinking" models.. It breaks the problem down, plans the steps and weighs the options, and only then answers. It is like doing a sum on scrap paper before you write the result in the box.

For the trip, the notes might run: 5 days. The filming places are spread over both islands. Hobbiton is in the north and many others are in the south. That is too much driving for 5 days, so pick one island or fly between them. The answer then comes out already organised.

The notes are written the same way as everything else, one token at a time, and they are charged as output tokens. Some apps show you the notes and some show only a summary, but the notes were written either way. That is why Thinking takes longer and uses up your limit faster.

What you get back is better answers on problems with steps in them: maths, code, planning. And strawberry? With room to think, a model can write the word out letter by letter and then count.

More and more, the model decides for itself. Claude weighs each request: a simple factual question may get no thinking at all, and a multi-step maths problem gets a lot. Gemini 3.8 Flash thinks by default at a medium level. Kimi K3 thinks on every answer, and you can only turn it down.

A simple rule covers most cases.

• A word, a fact, a quick rewrite: fast mode is quicker and just as good.
• A plan, a comparison, a tricky sum: let it think.

The 4 things to keep

• An AI writes one token at a time, like autocomplete that has read a library. It reads the same whether it is right or wrong.
• Tokens are pieces of words, and every limit is counted in them. A long chat is read again with each message.
• A weighted die picks each piece, which is why one question can come back with 2 different answers.
• Thinking is scrap paper before the answer. It is slower and uses more of your limit, and it is better on hard problems.