What Are AI Tokens? The Unit Behind AI Limits and Prices

Every AI chatbot and API measures your text in the same unit, and it isn't words. It's tokens. They decide how much you can paste into one conversation, when a long chat starts forgetting its beginning, how much an API call costs and why the same question can cost more in Spanish than in English. This guide explains what AI tokens are, how text gets split into them, how they connect to context windows and pricing, how to count them and the prompts that cut the number you use without making answers worse.

What an AI token is

A token is the chunk of text a language model reads and writes as a single unit. Before a model sees your prompt, a program called a tokenizer cuts it into these chunks and turns each one into a number. The model works only with those numbers, and it writes its answer one token at a time.

A token can be a whole word, part of a word, a punctuation mark or a space attached to the start of a word. OpenAI's widely quoted rule of thumb for English is that one token is about four characters, or roughly three quarters of a word. So:

• 100 tokens is about 75 English words, a short paragraph.
• 1,000 tokens is about 750 words, a page and a half of ordinary prose.
• 100,000 tokens is about 75,000 words, the length of a full novel.

Those are averages, not rules. Common words such as "the" or "because" are usually one token each. Rare words, names, typos and long compounds split into several. "Tokenization" itself might come out as "Token" and "ization". If you want to see it rather than take it on trust, paste a paragraph into our token counter and watch how the count compares with the word count.

How text becomes tokens

Most current tokenizers use a method called byte-pair encoding (BPE). It starts with individual characters, finds the pairs that appear together most often in a huge sample of text and merges them into new tokens, again and again, until the vocabulary reaches a fixed size. The tokenizer used by GPT-4, for example, has a vocabulary of about 100,000 tokens. Frequent words end up as a single token; rare ones are built from pieces.

That training sample explains most of the quirks you'll notice:

• Other languages cost more: tokenizers learn mostly from English, so the same meaning in other languages tends to need more tokens. A 2023 study by Petrov and colleagues found that translations of the same text can differ in token length by up to 15 times between languages. European languages are usually only modestly more expensive; languages with less training text and non-Latin scripts fare worse.
• Numbers and code fragment: long numbers, IDs, dates and timestamps often split into several pieces, and so do indentation, brackets and unusual variable names in code.
• Every model counts differently: each company trains its own tokenizer, and they change between generations. Anthropic notes that its tokenizer from Claude Opus 4.7 onwards produces roughly 30% more tokens for the same text than earlier Claude models. A count from one model is only an estimate for another.

Tokens also explain a famous weakness. Ask a model how many r's are in "strawberry" and it may get it wrong, because it never sees the letters, only a couple of token numbers. Spelling, counting characters and rhyming are harder for models than they look for the same reason.

Tokens and the context window

The context window is the maximum number of tokens a model can work with in one request. It isn't only your latest message. It includes the system instructions, the whole conversation so far, any files or images you've attached, any tool results and the answer the model is writing. Current flagship models advertise windows from a few hundred thousand tokens to a million or more; Anthropic, for instance, lists a 1M-token window for its current Claude models.

A few things catch people out:

• Chat apps resend the whole conversation: the model has no memory between turns, so every new message is processed together with all the earlier ones. In a 20-turn chat where each exchange adds about 500 tokens, the last message carries around 10,000 tokens of history, and the conversation as a whole processes more than 100,000.
• Attachments count: a long PDF, a spreadsheet or a pasted codebase can fill most of the window before you've asked anything. Images are tokens too: Claude's documentation puts a 1,000 × 1,000-pixel image at about 1,300 tokens.
• Reasoning counts: models that "think" before answering produce hidden reasoning tokens. OpenAI says they take up space in the context window and are billed as output, and it recommends reserving at least 25,000 tokens for reasoning and output when you start using those models.
• When the window fills, something gives: an API rejects a prompt that's too long, while chat apps tend to drop or summarise older messages quietly. That's why a long conversation can start ignoring decisions you made at the start.

A bigger window doesn't mean the model uses all of it equally well. The 2023 paper "Lost in the Middle" found that models answer best when the relevant information sits at the beginning or end of a long input and worse when it's buried in the middle. Anthropic's documentation calls the broader effect context rot: as the token count grows, accuracy and recall decline. Choosing what goes into that space is the whole subject of context engineering.

How token pricing works

Subscription chat apps hide tokens behind usage limits, but every API bills by them, usually as a price per million tokens. Four items appear on almost every price list:

• Input tokens: everything you send, including the system prompt and conversation history.
• Output tokens: everything the model writes, including any reasoning tokens. Output usually costs several times more than input, because the model generates it one token at a time.
• Cached input: text repeated at the start of many requests, such as a long system prompt, can be cached and billed at a discount. On Claude, cache reads cost a tenth of the normal input price on most models.
• Batch processing: several providers discount requests that don't need an instant answer and can run within hours.

A worked example with made-up but realistic prices of $3 per million input tokens and $15 per million output tokens: a support assistant with a 2,000-token system prompt, about 1,500 tokens of conversation history and a 300-token customer message sends 3,800 input tokens per reply and gets back about 400. That's $0.0114 for input plus $0.006 for output, under two cents a reply, or about $174 for 10,000 replies a month. Halve the system prompt and the bill falls by roughly a sixth; cache it and the saving is bigger still.

The lesson isn't that tokens are expensive. It's that the cost sits in the parts you repeat: system prompts, history and attached documents, multiplied by every request.

How to count tokens before you send

Most people never need an exact count. You need one when you're close to a limit, pricing an automation or comparing models. The options, from rough to exact:

1. The rule of thumb: words × 1.33 for English. Fine for "will this fit?" questions, unreliable for code, numbers or other languages.
2. An estimator: our token counter runs in your browser, so your text isn't uploaded, and shows estimated tokens and cost for several models at once.
3. The provider's own tool: OpenAI publishes a tokenizer page and its open-source tiktoken library; Anthropic offers a free token-counting endpoint in its API; Google's Gemini API has a count-tokens method. These use the real tokenizer for the model you name.
4. The usage report: every API response tells you how many input and output tokens it used. For live systems, logging those numbers beats any estimate.

In chat apps you rarely see token counts at all. The signs that you're near the limit are indirect: warnings about long conversations, files that get truncated, or answers that forget earlier details.

Prompts that save tokens without losing quality

Cutting tokens is useful only if the answers stay as good. The safest savings come from removing repetition and controlling length, not from writing prompts in telegraph style. The first prompt turns a sprawling chat into a compact brief you can paste into a fresh conversation:

Long system prompts and reusable templates are the next place to look, because you pay for them on every request:

Output tokens are the expensive ones, and models are happy to write more than you need. Say what length and shape you want:

Run the result past our prompt analyzer if you're unsure whether a shortened prompt is still clear. The basics of a well-built prompt are in our guide to prompt engineering fundamentals.

Ways to use fewer tokens

Beyond prompts, a few habits make the biggest difference:

• Start new chats for new topics: one conversation per task keeps the history short and relevant.
• Send the relevant part, not the whole file: paste the three sections that matter rather than the 80-page report. For large document collections, retrieval does this automatically; see what RAG is.
• Clean up pasted text: web pages and PDFs come with menus, footers, page numbers and repeated headers. Removing them is free accuracy.
• Put stable content first: on an API, keep the system prompt and reference material identical and at the start of each request so caching can apply.
• Match the model to the task: smaller models cost a fraction per token and handle classification, extraction and short rewrites well. Save the large ones, and reasoning modes, for problems that need them.
• Ask for structured output: a JSON object or a short table is usually shorter than paragraphs, and easier to use downstream.

Limits worth knowing

• Token counts are approximations across models: the same text produces different counts on different tokenizers, and even the official counting tools describe themselves as estimates. Leave a margin.
• A big window isn't perfect memory: fitting a document into the context doesn't mean every detail will be used. Ask for quotes or page references when accuracy matters.
• Fewer tokens isn't always better: cutting context the model needs, or demanding very short answers to complex questions, saves pennies and costs quality. Reasoning tokens in particular often buy real accuracy on hard problems.
• Chat plan limits aren't simple token maths: subscription apps set their own usage caps, which vary by plan, model and demand, and aren't always expressed in tokens. Long chats and large files still use them up faster.
• Prices and limits change often: context windows, rates and tokenizers have changed several times a year. Check the provider's current pricing page before you budget, and treat any figure in an article, including this one, as a starting point.
• Tokens don't measure truth: a model can produce a fluent, well-formatted answer in exactly the length you asked for and still be wrong. Token limits and AI hallucinations are separate problems.

Once you think in tokens, a lot of odd AI behaviour makes sense: the chat that forgets, the bill that grows with every feature, the answer that's oddly bad at spelling. You don't need to count them daily, but knowing where they go lets you write prompts that fit, cost less and keep the important parts where the model will actually notice them.

References

• Anthropic — Context windows (Claude documentation)
• Liu et al. — Lost in the Middle: How Language Models Use Long Contexts (2023)
• Wikipedia — Byte-pair encoding