What Are Tokens? The Complete Guide

The unit your bill is actually measured in

Every LLM API charges by the token, not the word or character. Before you can estimate a cost, forecast a budget, or debug why a request got truncated, you need to know what a token is and how your text turns into them.

A token is a chunk of text, not a word

Tokenizers split text into subword pieces based on how frequently character sequences appear in the training data. Common English words are often a single token. Rare words, numbers, and non-English scripts frequently split into multiple tokens.

As a rough rule of thumb for English text: 1 token ≈ 4 characters ≈ 0.75 words. So 100 words of English is roughly 130–140 tokens, not 100.

"The invoice total is ₹4,500."
→ roughly 9 tokens: [The][ invoice][ total][ is][ ₹][4][,][500][.]

Notice the ₹ symbol and the number get split into several tokens — non-ASCII characters and digit sequences are frequently less token-efficient than plain English words. This matters directly for Indian applications: Hindi, Tamil, or other Indic-script text can use 2–4x more tokens for the same amount of meaning compared to English, because most tokenizers were trained predominantly on English/Latin-script text.

Why this matters for cost

Every price on every model card — ₹X per million input tokens, ₹Y per million output tokens — is a token count, not a word count or a character count. Two prompts that look similarly long in your editor can cost meaningfully different amounts if one is in English and the other has embedded Hindi text, code, or heavily punctuated data like JSON.

# A rough Python estimate without an API call
def estimate_tokens(text: str) -> int:
    return len(text) // 4  # crude approximation for English text
 
prompt = "Summarise this customer complaint in two sentences."
print(estimate_tokens(prompt))  # ~13 (actual: 11-12, close enough to budget with)

For an exact count rather than an estimate, use the free token counter — it runs the actual tokenizer for GPT, Claude, and Gemini model families and shows you the real number before you spend anything.

Input tokens vs. output tokens

Almost every provider prices these differently — output tokens typically cost 4-5x more than input tokens, because generating text is more computationally expensive than reading it. This is why a chatty system prompt matters less than a chatty model: 2,000 tokens of static system prompt sent 100 times is often cheaper than 100 verbose 500-token responses.

Context window: the token budget per request

Every model has a maximum context window — the total tokens (input + output) it can handle in a single request. Exceed it and the request fails, not gets truncated silently. If you're stuffing long documents or conversation history into a prompt, track your running token count against the model's limit — the context window tool helps visualize how much of a model's budget your prompt is using.

Frequently Asked Questions

How many tokens is one page of English text?

Roughly 500–600 tokens for a standard page (about 400–500 words), using the ~0.75 words-per-token rule of thumb.

Do all models use the same tokenizer?

No. GPT, Claude, and Gemini each use different tokenizers, so the same text can produce a different token count on each. Always check the specific model's tokenizer if you need an exact count — the free token counter supports multiple model families.

Why does my Hindi or Tamil prompt cost more than an equivalent English one?

Most tokenizers were trained on primarily English/Latin-script data, so non-Latin scripts often split into more tokens per character. Budget extra headroom for Indic-language workloads compared to an equivalent English prompt.

Does whitespace or formatting count as tokens?

Yes. Spaces, newlines, and punctuation all consume tokens, though usually as part of adjacent word tokens rather than standalone ones. Dense formats like JSON or Markdown tables typically use more tokens per character than plain prose.

Get started

• Count tokens for your own prompt before sending a real request.
• Estimate total request cost once you know your token counts.
• Related: Context Window Management: Don't Waste Tokens.