Skip to content

Stockertim Dictionary · Artificial intelligence · Technology

Token

Also called tokens, text token

In short

A token is the small unit of text a language model reads and writes, such as a short word, part of a longer word, a number or a punctuation mark.

  • Beginner
  • Concept
  • Checked

In detail

Language models do not read letters or whole sentences. Before any text reaches the model, a tokeniser splits it into tokens: common short words often become one token, while longer or rarer words are broken into two or more pieces. Numbers, spaces and punctuation marks are tokens too. The model then works entirely in tokens, predicting one likely token after another until its reply is complete.

Tokens matter in three practical ways. They measure how much text fits in the context window, they set how long a reply can be, and they are usually the unit on which paid AI services charge. A common rule of thumb for English is that 100 words come to roughly 130 to 140 tokens, but the ratio varies with the model, the language and the kind of text. Tables, code, unusual names and languages other than English often take more tokens per word.

Because the model handles pieces of words rather than letters, some tasks that look simple for a person, such as counting the letters in a word or reversing a string, can go wrong. Knowing that text is tokenised also explains why long pasted documents can be cut off and why a reply sometimes stops in mid-sentence when it reaches its length limit.

Worked example

An analyst at Harbour Foods wants an AI assistant to summarise a 6,000-word report, using the rule of thumb of about 4 tokens for every 3 words.

  1. Report: 6,000 x 4 / 3 = 8,000 tokens
  2. Instructions and the expected summary: 300 + 1,200 = 1,500 tokens
  3. Total budget: 8,000 + 1,500 = 9,500 tokens

The job needs roughly 9,500 tokens, a planning estimate rather than an exact count, since the ratio varies by model and language.

In practice

Teams estimate tokens before pasting long material, to check it fits the context window and to forecast the cost of a paid service. Many tools show a token count, and finance teams use token volumes to compare usage between departments.

Watch out

  • A token is not the same as a word; one word can be several tokens.
  • The word token also means a digital asset or a security code in other fields, which is a separate meaning.

Learn it properly

Official sources

Facts last checked on 3 November 2026.

Education, not advice: the Stockertim Dictionary explains what terms mean. It never recommends buying or selling anything.

English