All chapters
  1. Start hereEverything the model reads is text. Only some of it is an instruction
  2. Foundations · 1From a sentence to a prompt: the parts and what each one is for
  3. Foundations · 2Roles and authority: who is allowed to tell the model what to do
  4. Foundations · 3Prompt composition: how an application builds the prompt
  5. Foundations · 4Prompt patterns: five ways to ask, and when each one earns its cost
  6. Surfaces · 5Tool-aware prompting: the tool description is a prompt too
  7. Surfaces · 6MCP prompts and resources: a template you chose versus content that was fetched
  8. Security · 7Prompt injection: when data starts giving orders
  9. Security · 8Defense in depth: assume one layer fails
  10. Operations · 9Long context and prompt placement: where an instruction sits changes what it does
  11. Operations · 10Versioning and provenance: which prompt produced this answer
  12. Operations · 11Testing prompts: golden cases, injection probes and comparisons
  13. Operations · 12Portability: the same prompt on another model is another prompt
  14. Operations · 13Production operations: rolling a prompt out, and rolling it back
  15. Reference · 14Frameworks and tooling: what each one does with your prompt
  16. Reference · 15Advanced topics: other languages, images, hand-offs and attackers who adapt
  17. Reference · 16The insurance lab: one poisoned roof note, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Operations · 9

Long context and prompt placement: where an instruction sits changes what it does

A context window is a budget. What gets cut when the prompt is too long, where to put the instructions, how prompt caching reads a prefix, and how to count before you send.

What you will understand
A budget
The context window is finite, and what gets cut when a prompt does not fit is decided by code, often badly.
Placement
Where an instruction sits changes how it is read and what a provider can cache.
Counting
Why you count before you send, with the provider's own counter, and keep the estimate for planning.

The insurer adds a new kind of evidence: full inspection reports, attached as PDFs, often thirty pages long. The first week goes well. Then an underwriter notices that recommendations on the longest submissions have stopped citing rules. A developer looks at the composed prompts and finds the cause in a helper function written months ago: when the prompt is longer than the limit, it drops text from the end. With a thirty-page report in the user message, the end that got dropped was not the report. The report sits in the middle. The end was the output contract and the closing request, which a later change had moved after the evidence.

No error was raised. The model received a prompt with no output format and answered in the best way it could. This page is about the budget every prompt lives inside, and about why the order of its parts is a design decision.

The window, the budget, and what gets cut#

A context window is the maximum amount of text, counted in tokens, that a model can take in for one call, including the answer it writes. Tiny example: if a model's window is a few hundred thousand tokens and the prompt uses all but a hundred, the answer can be at most about a hundred tokens. Each model documents its own window.

A token budget is the share of the window the application plans to spend on each part of the prompt, with room reserved for the answer. Tiny example from the example code: a window of 700 estimated tokens with 150 reserved for the answer leaves a budget of 550 for the prompt. The numbers are illustrative and deliberately small, so that a cut is visible.

Truncation is what happens when the prompt does not fit: something is removed, by the application before sending or by the provider's rules. Tiny example: the helper in the story removed text from the end. Which part gets removed is the whole question, and a naive rule removes whatever happens to be last.

The example's budget.py shows this on prompt 1.4.0 and case UW-101. Its fit function cuts from the end, as naive code does. Token counts are estimates by tokens.py, not a real tokenizer.

cd examples/uw-prompts
python3 budget.py --version 1.4.0 --window 700
prompt 1.4.0 · UW-101 · 778 estimated tokens · window 700 - reserve 150 = budget 550
  Task                      system  static      68  kept
  Required steps            system  static      47  kept
  Authority rule            system  static      62  kept
  Never guess               system  static      30  kept
  Policy rules              system  static     122  kept
  Output contract           system  static      68  kept
  Success criteria          system  static      37  kept
  Worked example            system  static     117  partly cut
  Case line                 user    dynamic     13  cut
  Submission                user    dynamic     73  cut
  Roof description          user    dynamic     39  cut
  Evidence (tool results)   user    dynamic    102  cut
second request (UW-003): 8 leading sections equal · 551 of 773 estimated tokens reusable (71%)

Read the cut column. With the instructions first, a cut from the end removes the case: the submission, the roof note and the evidence. The model would get perfect instructions and no facts, and, with the never-guess rule intact, it would ask for the submission. That is a visible failure. Now imagine the opposite order, with data first and instructions last, as in the story: the cut removes the output contract and the never-guess rule, and the model answers fluently without them. That failure is invisible.

The lesson is not "put instructions first". It is that cutting is a decision, and the application should make it on purpose: fail loudly when the instructions do not fit, and shorten data with a rule you chose (summarize, drop the oldest evidence, page through a long report), never by losing whatever happens to be last. Choosing what data to keep is context engineering, and it has its own site.

What this is not. This chapter is about where instructions sit and what a budget does to them. Selecting, retrieving, summarizing and compacting the information that fills the window belongs to context.sammuthu.com. Saving state across calls belongs to memory.sammuthu.com.

Where an instruction sits changes how it is read#

Position matters even when everything fits. A research paper read today, "Lost in the Middle: How Language Models Use Long Contexts" (2023), reports that performance "is often highest when relevant information occurs at the beginning or end of the input context" and "significantly degrades when models must access relevant information in the middle of long contexts." The effect has a name: lost in the middle. Tiny example: a rule stated once, between two long documents, is the rule most likely to be missed.

Models have changed since that paper, and the size of the effect differs by model. What has not changed is the advice providers give. Anthropic's prompting guide, read today, says that for long inputs you should place long documents "near the top of your prompt, above your query, instructions, and examples", and notes that queries at the end improved response quality in its tests. Read together with the paper, the practical layout is:

Diagram: One prompt, one order · static first for the cache, long data next, the request last

Read left to right as the order of the prompt. The static part is identical on every call and can be cached. The dynamic part changes per case. A naive cut removes whatever is last, so the policy for cutting must be chosen. Click a box to see it in prompt 1.4.0.

  1. 1 · Static: the cache prefix
    1. Instructions: Task, process, authority, never guess, rules, format, criteria, example. Same text on every call.
  2. 2 · Dynamic: the case
    1. Case data: Case line, submission, roof note and evidence. Different on every call; long documents go here.
    2. The request: The specific ask, at the edge the model reads best, after the data.
  3. 3 · The limit
    1. Budget: The window less the room kept for the answer. Here 700 - 150 = 550 estimated tokens (illustrative).
    2. Truncation: A naive cut removes whatever is last. A policy never cuts instructions and shortens data by rule.
  4. 4 · The cache
    1. Prompt caching: The provider reuses the processed prefix when the next request starts with exactly the same text.

The insurer's prompt follows that layout: the system message holds the rules; the user message holds the case line, the marked data, the computed eligibility result and, when the underwriter typed one, the request, last. A team could go one step further and restate the two most important rules at the very end ("Answer with one JSON object; never guess a missing value"); that is a design option, not part of version 1.4.0, and like any change it would be tested on the golden cases first.

Production pitfall. A reminder at the end is text in the user message. If an attacker's text also sits near the end, it competes with the reminder on equal terms. Keep untrusted text marked and escaped wherever it sits, and keep enforcement in code; placement improves the odds, it does not decide them.

Prompt caching reads a prefix#

Every call sends the whole prompt again. For the insurer, about 550 of the roughly 775 estimated tokens of prompt 1.4.0 are identical on every call: the instructions, the rules, the format, the example. Providers offer a way to avoid processing that repeated part every time.

Prompt caching is a provider feature that stores the processed form of the beginning of a prompt and reuses it when a later request starts with exactly the same text. Tiny example: the second submission of the day reuses the system message processed for the first. The reusable beginning is the cache prefix: everything up to the first difference.

The documentation read today agrees on the central rule:

ProviderWhat its documentation says
AnthropicCaching "covers the full prefix: tools, system, and messages (in that order)"; a hit needs "100 percent identical prompt segments" up to the marked block; changes at one level invalidate that level and all later ones; modifying tool definitions "invalidates the entire cache"; default lifetime 5 minutes, with a 1-hour option; a model-specific minimum length, below which a request is processed without caching and no error is returned
OpenAI"Prompt caching is enabled by default for supported OpenAI models"; "Cache reuse requires the entire rendered prefix to match"; put "stable developer instructions and shared reference material first" and dynamic content "at the end rather than the beginning"; entries are not stored indefinitely
Google GeminiThe documentation read for this page does not cover its caching feature; the frameworks and portability fact logs record what was read from Google's pages today

So the order that protects instructions from truncation is also the order that caching rewards: static first, dynamic last. The budget.py output above measures it on the example. Two requests in a row, UW-101 then UW-003, share 8 leading sections: 551 of 773 estimated tokens, 71 percent. The same two requests under version 1.2.0, which put the case line inside the system message near the top, share 2 sections: 115 of 656 estimated tokens, 18 percent.

python3 budget.py --version 1.2.0 --window 700
prompt 1.2.0 · UW-101 · 661 estimated tokens · window 700 - reserve 150 = budget 550
  Task                      system  static      68  kept
  Required steps            system  static      47  kept
  Case line                 system  dynamic     13  kept
  Authority rule            system  static      62  kept
  Never guess               system  static      30  kept
  Policy rules              system  static     122  kept
  Output contract           system  static      68  kept
  Success criteria          system  static      37  kept
  Submission                user    dynamic     73  kept
  Roof description          user    dynamic     39  partly cut
  Evidence (tool results)   user    dynamic    102  cut
second request (UW-003): 2 leading sections equal · 115 of 656 estimated tokens reusable (18%)

One 13-token line in the wrong place costs most of the cacheable prefix, because everything after the first difference counts as new. That is why version 1.4.0 moved the case line into the user message; its changelog says so: "Static text first, dynamic text last."

Try the order yourself. Move a section up or down, change the window, and watch the cut and the reusable prefix.

Section of 1.4.0KindEstimated tokens on UW-101
Task, required steps, authority rule, never guessStatic68, 47, 62, 30
Policy rules, output contract, success criteriaStatic122, 68, 37
Worked exampleStatic117
Case line, submission, roof description, evidenceDynamic13, 73, 39, 102

At a 700-token window with 150 reserved, a cut from the end removes part of the worked example and all four dynamic sections. Prefix reuse for UW-101 then UW-003: 551 of 773 estimated tokens (71 percent) static first; 0 if the dynamic sections come first.

For a backend engineer. Prompt caching behaves like an HTTP cache keyed on the request body's prefix, or like a prepared statement whose plan is reused. The prepared-statement comparison is the useful one: the plan is reused only when the statement text is identical, so you keep the variable parts in parameters at the end. It breaks in one way: a prepared statement's parameters are a separate channel, and a prompt's dynamic text is not. Caching changes cost and latency, never what the model is allowed to do.

Count before you send#

Every number on this page so far is an estimate from tokens.py, written for teaching. A real budget needs a real count, and real counts differ by model: the same text can be a different number of tokens on two models from the same provider.

Anthropic's token counting documentation, read today, describes an endpoint, /v1/messages/count_tokens, that "accepts the same structured list of inputs for creating a message, including support for system prompts, tools, images, and PDFs" and returns the number of input tokens. It also says three things a budget must respect: the count "is an estimate" that "might differ by a small amount" from actual use; a newer tokenizer used by recent models "produces approximately 30 percent more tokens than on earlier models" for the same text; and: "Recount prompts against the model you plan to use rather than reusing counts measured against earlier models."

That last point turns counting into part of a model change. A prompt that fit with room to spare can stop fitting when the model changes, with no edit to the prompt. The portability chapter covers the model change itself; here the rule is simple: count with the provider's counter for the model you will call, at the size of your largest real inputs, and fail loudly in code when the instructions would not fit.

The same idea for a coding agent. A coding agent's window fills with file contents and tool output as a session goes on. Harnesses compact or summarize old turns to make room, and a careless compaction drops the repository's instruction file or the original task. Teams that rely on rules in a repository instruction file make sure the harness re-reads it, or keeps it at the start of the window, after compaction. How harnesses manage that is covered at harness.sammuthu.com.
Deeper: caching, placement and injection pull in the same direction, mostly

Three forces shape the order of a prompt, and in this design they agree.

  • Caching wants static text first and anything that varies last.
  • Authority wants the builder's text in the system slot, apart from data.
  • Attention wants long data before the request, and the request with its key rules at the end.

They disagree in one place: the end of the prompt is both where the model reads best and where untrusted data tends to sit, right before the request. The insurer's layout puts the request after the data, so the last thing the model reads before answering is the application's text, not the applicant's. When a request must include untrusted text (a user question that quotes a document), mark it like any other data.

One more interaction: any change to the static prefix invalidates the cache from that point on. A prompt version that edits one sentence of the system message pays a cold cache on the first calls after rollout. That is a cost to expect during a canary, not a regression in the prompt.

Deeper: a truncation policy that fails safe

A sound policy names what may never be cut, what may be shortened and how, and what happens when that is not enough.

SectionPolicy
System instructions, policy rules, output contractNever cut. If they do not fit, fail the call with an error the operator sees
Worked exampleMay be dropped, with a recorded flag, before any case data
Evidence from toolsShortened by rule: keep fields the policy needs, drop free-text notes, or summarize with a separate call
Long documentsPaged: process in sections and extract typed facts, rather than one giant prompt
The requestNever cut; it is short and it sits last

Record the decision in the run record: the estimated or counted tokens, what was dropped and why. A recommendation made on a shortened prompt is a different kind of evidence, and the underwriter should be able to see that.

Deeper: does a larger window make this go away?

Larger windows move the limit; they do not remove the three costs of a long prompt. Every token is processed on every call that is not cached. Latency grows with input length. And the lost-in-the-middle effect is about position, not capacity: a rule buried between two long documents can be missed in a window with plenty of space left.

The design question to ask is not "does it fit?" but "what does the model need to see, in what order, for this call?". For an underwriting recommendation the answer is short: the rules, the facts the rules need, the evidence ids, and the request. A thirty-page inspection report is better turned into typed facts by a separate step (see isolation on the defense page) than sent whole to the step that recommends.

Checklist: is this prompt placed and budgeted?#

QuestionGood answer
Is the order static first, dynamic last?Yes: instructions, rules, format and examples before the case
Is the specific request at the very end, after the data?Yes, with the one or two key rules restated
What does the code cut when the prompt is too long?A named policy; instructions are never cut silently
Are counts from the provider's counter for the model in use?Yes, rechecked when the model changes
Is any per-call value placed before static text?No: it would break the cache prefix
Does the run record show what was dropped?Yes

In one breath#

The context window is a budget, and when a prompt does not fit, whatever the code cuts is gone without an error, so cut on purpose and never cut the instructions. Order matters twice: models read the beginning and the end of a long prompt best, and providers cache only an identical prefix, so put static instructions first, long data next, and the request last. Count with the provider's counter for the model you call, because counts change between models even when the text does not.

The prompt now has a shape, a place for every part and a cost for each. It also changes over time, and every change is a new prompt. The next chapter is about versioning and provenance: knowing which exact text produced which answer.