30-SECOND SUMMARY
What to take away
- Context is the token window available to the model.
- Longer context increases KV-cache memory.
- Measure the smallest sufficient value for the workload.
How context consumes memory
Budget tokens before raising limits.
- 01PROMPT
Instructions and text
- 02TOKENS
Shared budget
- 03KV CACHE
Length and concurrency
- 04LIMIT
Smallest sufficient value
Budget tokens
System instructions, history, retrieved text, and the answer share one context budget. Characters do not map to tokens uniformly across languages.
The working rule for “Budget tokens” is: Context is the token window available to the model. Keep the model, quantization, context length, and concurrency fixed so that a device or model comparison has a clear cause.
For verification, save the model and runtime versions, source input, relevant settings, and observed output together. Repeat the step while changing only one factor, and record unexpected results and untested limits as carefully as successes before applying the guidance to private or production data.
Account for KV cache
The runtime stores prior-token attention state in a KV cache. Longer context and more concurrent requests consume more memory even with the same model file.
The working rule for “Account for KV cache” is: Longer context increases KV-cache memory. Keep the model, quantization, context length, and concurrency fixed so that a device or model comparison has a clear cause.
Preserve the before-and-after state and the time of the check so that another run can reproduce the result. Include at least one failure condition—such as empty input, constrained resources, or a restart—to reveal the boundary of the step rather than documenting only the happy path.
Inspect Ollama
Ollama documents VRAM-based defaults and warns that larger context requires more memory. Use `ollama ps` to inspect allocation and offloading.
The working rule for “Inspect Ollama” is: Measure the smallest sufficient value for the workload. Keep the model, quantization, context length, and concurrency fixed so that a device or model comparison has a clear cause.
One successful run is not enough: repeat it after a restart and send one invalid input to confirm a controlled failure. Before connecting production data, test timeouts and cleanup so that an interrupted command does not leave stale processes, files, or application state.
OLLAMA_CONTEXT_LENGTH=8192 ollama serve
ollama psIncrease only as needed
Measure representative documents plus instructions and answer headroom. Consider retrieval or hierarchical summaries instead of inserting everything.
The working rule for “Increase only as needed” is: Context is the token window available to the model. Keep the model, quantization, context length, and concurrency fixed so that a device or model comparison has a clear cause.
For verification, save the model and runtime versions, source input, relevant settings, and observed output together. Repeat the step while changing only one factor, and record unexpected results and untested limits as carefully as successes before applying the guidance to private or production data.
Frequently asked questions
Does more context always improve answers?
No. It costs memory and latency, and models may not use it effectively. For a practical check, follow the “Budget tokens” section, change one condition at a time, and record the result.
Does it use RAM or VRAM?
That depends on the runtime and offloading; inspect the running process. Longer context increases KV-cache memory. For a practical check, follow the “Account for KV cache” section, change one condition at a time, and record the result.
What if long chats slow down?
Compare a new chat and a smaller context with the same prompt. Measure the smallest sufficient value for the workload. For a practical check, follow the “Inspect Ollama” section, change one condition at a time, and record the result.
Primary sources
Check the original documentation for version-specific details.
Ollama Context Length Ollama FAQ