Tokens and Context Windows

Text length is not token count

Suppose a documentation helper needs to answer one question from a large manual. Sending the whole manual may exceed the model’s input allowance or crowd out the answer. Before shortening anything, identify what is being counted and which information the task actually needs.

Tokenization converts input into units represented by token IDs. A tokenizer may split an unfamiliar identifier into several pieces while representing a common word with one token. Whitespace, punctuation, code and different writing systems can produce very different counts. A fixed words-to-tokens ratio is only a rough estimate.

Use the tokenizer or counting facility for the specific model when precision matters. Include system instructions, user messages, retrieved passages, tool results and message-format overhead. Multimodal inputs may have their own accounting rules. Do not assume that counting only the visible user question gives the request size.

Leave room for the answer

The context window limits what the model can process in a request. Provider rules differ in how they combine input, output and other token categories, and models can also have a separate maximum output length. Check those limits before sending a large request. A long accepted input still does not guarantee that every detail will be used effectively.

For a hypothetical model whose shared input/output budget is 16,000 tokens, reserve the output first and account for instructions and overhead. The example below is a planning exercise, not a current provider specification. Real applications should measure their assembled requests and handle over-limit failures explicitly.

Total shared budget:        16,000 tokens
Reserved response:          2,000
Instructions + question:    1,000
Tool/message overhead:        500
Safety margin:                500
Available source material: 12,000

Conversation history must be managed

Many APIs do not automatically remember every earlier request. The application sends history, references stored conversation state, or maintains its own summary. Treat that mechanism as an implementation detail to verify. A browser chat interface may look continuous even when older messages no longer reach the model.

Separate durable facts from disposable conversation. Keep explicit user constraints, the current task and source references in a controlled state. A generated summary can lose a negation or turn a tentative guess into a fact. Compare critical fields with the original messages and let users correct retained information.

Prefer relevant context to maximum context

For a coding question, start with the failing function, its caller, the exact error and the relevant dependency version. Sending every source file increases cost and can obscure the important detail. Retrieval and chunking help select material, but preserve enough surrounding context for definitions and qualifications to make sense.

Test long-context behaviour with conflicting versions, a crucial fact near the middle and a question whose answer is absent. Check both answer accuracy and evidence selection. A request fitting within the advertised limit is a capacity result, not a correctness result.

Choose passages against a concrete question

A user asks whether version 3 supports CSV export. Assume your measured source allowance is 6,500 tokens. The version-3 export page takes 2,000 tokens, its permissions section takes 1,000, an old version-2 guide takes 3,500 and unrelated installation notes take 1,500. Keeping the first two is both smaller and better supported than filling the allowance with whatever appears first.

Preserve the version heading, any conditions and the source URL with the selected passages. The export page alone may say “available” while the permissions section limits which accounts can use it. If the current documentation does not settle the question, more irrelevant text will not help; retrieve the missing evidence or ask for clarification.

When summarizing conversation history, retain whether a statement was a requirement, observation or hypothesis. A summary such as “the server cache is broken” changes the meaning of “the page looks stale; the cache may be responsible.” Store critical constraints explicitly and compare them with the original messages before using a summary for consequential work.

Sources & further reading

Primary references for the concepts in this guide. Examples and exercises are illustrative; verify provider-specific behaviour against the version you use.

• Hugging Face: Tokenizers
• Hugging Face: Padding and truncation