Token counting explained for builders and budget owners

See also / canonical cluster. Pillar for token & cost estimation: How to estimate LLM token costs before you ship. Interactive: token & cost estimator. This page stays on a different intent — see the intro.

This is the fundamentals spoke: what a token is, why model families disagree, and how counting relates to context windows. It is not the pre-ship monthly cost spreadsheet — that lives on the estimation pillar. Use the token & cost estimator when you need interactive numbers.

What is a token?

A token is a chunk of text produced by a tokenizer — often a subword piece. Rough intuition for English prose:

• ~4 characters per token
• ~0.75 words per token (or ~100 tokens ≈ 75 words)

These are rules of thumb, not laws. Code, URLs, non-English text, and whitespace-heavy JSON tokenize differently.

Chinese often uses fewer characters per token than a naive English rule suggests — still measure with a real tokenizer when precision matters.

Why vendors count differently

Each model family ships (or implies) a tokenizer. The same string can yield different token counts on different APIs. Therefore:

• Budget with the tokenizer of the model you call
• Do not reuse GPT heuristics blindly for other families
• When vendors expose usage.prompt_tokens in responses, treat that as ground truth for billing

Offline estimators (including ours) approximate for planning. They are for forecasts and teaching, not invoices.

What consumes tokens in a request?

Typical chat/completions request:

1. System message
2. Developer / tool instructions (if any)
3. Conversation history
4. Retrieved documents / RAG chunks
5. Tool schemas and tool results
6. The new user message
7. Output tokens from the model reply (billed separately, often at a different rate)

Agents multiply (4)–(6) across loop iterations. That is why research agents need max-call budgets (system prompts for research agents).

Context window vs cost

• Context window — hard maximum tokens the model can consider (input + output constraints vary by API)
• Cost — usually dominated by how many tokens you actually send and generate, not the maximum window

A 128k window does not mean you should send 128k tokens. More context is not free and often hurts quality when noisy.

Practical estimation workflow

1. Paste a representative prompt (system + user + sample RAG) into an estimator
2. Set expected output length (max_tokens or observed average)
3. Multiply by monthly calls × (1 + retry rate)
4. Apply vendor $/1M input and output rates
5. Add 10–20% buffer for prompt creep

Do this per feature, not as one blob.

Why your estimator disagrees with the API

Counting for non-English and code

• CJK text: do not assume 4 chars/token; sample real counts
• Code: operators and identifiers can be token-heavy; indentation matters less than you think, but long minified lines can be worse
• Base64 / hex: extremely token-inefficient — avoid stuffing blobs into prompts
• Markdown tables: usually fine; huge HTML dumps are not

Output token control

Output is often pricier per token than input on frontier models.

• Set max_tokens to the task (titles vs essays)
• Ask for compact formats (bullets, JSON without comments)
• Forbid restating the user question
• For creative tasks, accept higher caps deliberately — and budget them

Token budgets in product design

Examples:

• Free tier: 50k input tokens / day / user
• Support bot: hard cap 8k context; summarize older turns
• Doc assistant: top-5 chunks ≤ 3k tokens combined
• Eval suite: separate budget so it cannot starve production

See LLM cost control for teams.

Teaching teammates a shared language

Print this on the wiki:

• “We’re not short on ideas; we’re short on tokens and edit time.”
• “Expanding the system prompt by 1k tokens at 1M calls/month is a budget change — review it.”
• “Estimator ≠ invoice; API usage = invoice.”

Quick lab (10 minutes)

1. Take a 500-word blog paragraph; estimate tokens at 4 chars/token
2. Run the same text through your provider’s tokenizer or a library when available
3. Compare error %
4. Repeat with 50 lines of Python and a Chinese paragraph
5. Update your team’s rule of thumb per content type

Related tools and guides

• LLM token cost estimation (cluster pillar)
• Token estimator tool
• Prompt cleaner — reduce junk tokens before counting
• Prompt engineering checklist