Prompt Caching

If you send the same long prefix on every request — a system prompt, a tool list, a document you keep asking questions about — the provider can keep it warm and skip re-processing it. On models that list a cache-read price, cached input tokens bill at that reduced rate; on other models they bill at the normal input rate. On a Variable price model, cached tokens are billed like the rest of the request: in most cases at the provider's cost plus a service fee, and never more than the model's maximum prices.

The catch is the write. Putting something into the cache costs 1.25× the normal input rate, so a cached prefix that is never read again made that request 25% more expensive. Caching pays when the prefix is reused, and loses money when it is not.

Do you need to do anything?

Depends on the model.

The models that need marking are the ones where the provider will not guess for you. If you send nothing, you simply get no caching — not an error.

Automatic caching: read the result

Nothing to send. Check whether you got a hit:

from openai import OpenAI
 
client = OpenAI(api_key="sk-...", base_url="https://backend.sovereigneg.com/v1")
 
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[
        {"role": "system", "content": LONG_SYSTEM_PROMPT},
        {"role": "user", "content": "What does clause 7 say?"},
    ],
)
 
usage = response.usage
print(usage.prompt_tokens)                        # total input, cached included
print(usage.prompt_tokens_details.cached_tokens)  # how many were served warm

prompt_tokens is inclusive — it already contains cached_tokens. Do not add them together.

The first request of a conversation reports cached_tokens: 0. Send the same prefix again within the cache lifetime and the count jumps.

Claude: mark the prefix with cache_control

Put cache_control on the last content part of the stable prefix. Everything up to and including that part is cached; everything after stays fresh. Order matters — put your stable content first. The marker can go on a part of a system, user or assistant message, or on a tool definition (see Marking the system prompt and tools).

response = client.chat.completions.create(
    model="claude-sonnet-5",
    messages=[
        {"role": "system", "content": "You answer questions about the contract below."},
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": LONG_CONTRACT_TEXT,
                    "cache_control": {"type": "ephemeral"},
                },
                {"type": "text", "text": "What does clause 7 say?"},
            ],
        },
    ],
)

Note the content is a list of parts, not a plain string — a marker has to attach to something, and a bare string has nowhere to put it.

In user and assistant messages, markers work on text and image_url parts. A system message takes text parts only, so the marker goes on a text part there. On a tool, cache_control goes on the tool definition.

Marking the system prompt and tools

A long system prompt and a fixed tool list are often the prefix most worth caching. Give the system message a list of parts and mark the last part you want cached, and put cache_control on the last tool you want cached, next to its type and function:

response = client.chat.completions.create(
    model="claude-sonnet-5",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": LONG_SYSTEM_PROMPT,
                    "cache_control": {"type": "ephemeral"},
                }
            ],
        },
        {"role": "user", "content": "What does clause 7 say?"},
    ],
    tools=[
        {
            "type": "function",
            "function": {
                "name": "find_clause",
                "description": "Look up a clause of the contract by its number.",
                "parameters": {
                    "type": "object",
                    "properties": {"number": {"type": "integer"}},
                    "required": ["number"],
                },
            },
            "cache_control": {"type": "ephemeral"},
        }
    ],
)

On Claude models served directly by Anthropic:

• The system prompt is split into blocks only when a part of it is marked, so each marker has a block to sit on. Parts are joined with a blank line either way, so marking a part does not change the system text the model reads. A system prompt with no marker goes out as one plain string.
• A marked tool caches the tool list up to and including that tool. A cache_control inside function is read only when the tool itself has none; if both are set, the one on the tool is used and the one inside function is ignored without being checked.

Rules

• At most 4 markers per request, counting every marked system part, tool and message part together. A fifth is a 400.
• type must be ephemeral. Anything else is a 400.
• Send ttl: "5m" or omit ttl entirely. Do not send 1h — see The 1-hour cache below.
• Below the model's minimum, the marker does nothing. No error, no cache.

These rules hold for markers on system parts and tools too: on Claude models served directly by Anthropic, a type other than ephemeral or a ttl other than 5m (such as 1h) on a system part or a tool is a 400, exactly as on a message part.

On Claude models served directly by Anthropic, a request with one marked system part, one marked tool and three marked message parts carries 5 markers and gets a 400. Before system and tool markers were forwarded, the same request went through, with those two markers silently dropped.

Depending on which provider serves the model, a rejection may come from us — naming the exact field, e.g. messages[].content[].cache_control.type — or be relayed from the upstream provider in its own wording. The rule is the same either way; only the error text differs.

Where a marker is dropped

A marker in these places is not supported yet. It is dropped with no error, and nothing is cached because of it:

• cache_control on the message object itself, next to role, instead of on one of its content parts (on every model);
• cache_control at the top level of the request (on every model);
• a part of a role: "tool" message, on Claude models served directly by Anthropic — a tool result is sent there as plain text.

Models that support cache control

These are the models where markers do something. The minimum prefix column is the floor below which the provider ignores your marker silently — no error, no cache, so the marker is simply wasted.

Claude — mark with cache_control:

OpenAI — mark with prompt_cache_breakpoint plus prompt_cache_options:

For a model not listed above, check its maker's documentation for the minimum. We add no markers of our own and do not check prefix length, so the provider decides: below its minimum a marker is simply ignored, with no error, no cache and no write charge.

The floors are not ordered by generation. Haiku 4.5 needs 4,096 while the older Haiku 3.5 needs 2,048; Opus 5 needs 512 while Opus 4.5 needs 4,096. Any rule you derive from two models will be wrong for a third — read the table.

Every other model on the platform either caches automatically (send nothing) or does not cache at all. In both cases a marker is unnecessary; the only way to tell them apart is to send the same prefix twice and look at cached_tokens.

GPT-5.6: explicit mode

GPT-5.6 and newer support explicit control, which requires both a marker and the option that switches explicit mode on:

response = client.chat.completions.create(
    model="gpt-5.6-terra",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": LONG_SYSTEM_PROMPT,
                    "prompt_cache_breakpoint": {"mode": "explicit"},
                }
            ],
        },
        {"role": "user", "content": "Summarise section 3."},
    ],
    extra_body={"prompt_cache_options": {"mode": "explicit", "ttl": "30m"}},
)

prompt_cache_breakpoint is an object, not a boolean. Sending prompt_cache_breakpoint: true is rejected with a 400 invalid_request_error; {"mode": "explicit"} is the required shape, and explicit is the only valid mode.

Sending mode: explicit without a breakpoint gives you no caching at all — it switches off the implicit behaviour you would otherwise have got for free. That is worse than sending nothing. Send both or neither.

Breakpoints are accepted on text, image_url, input_audio, file and refusal parts. Marking anything else returns a 400.

Unlike Claude's 5-minute window, GPT-5.6 cached prefixes have a minimum 30-minute TTL — that is the value to send in prompt_cache_options.ttl.

prompt_cache_key — routing affinity

Optional. It is forwarded to models served through OpenAI-compatible providers, where it can route requests that share a prefix toward the same machine and raise your hit rate (OpenAI documents this; other providers may ignore it). It is not sent to Claude models served directly by Anthropic or to models on our own GPUs, where it has no effect.

response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[...],
    extra_body={"prompt_cache_key": "support-bot-v3"},
)

Requests with different keys can land on different machines and each warm their own cache.

Use one key per stable prefix — per assistant, per document, per tenant. Do not put a per-request value in it; a unique key on every call defeats the point.

Your key is namespaced to your organisation before it leaves us, so it can never collide with another customer's cache. The value you send is an input to that namespace, not what travels upstream. Two organisations sending the identical key get unrelated caches.

What it costs

Per million input tokens, relative to that model's normal input rate:

With the 5-minute cache, one read already pays for the write: 1.25× + 0.1× = 1.35× for the prefix across two requests, against 2× uncached, and every further read adds to the saving. Written and never read, you paid 25% extra for nothing.

A cache hit refreshes the clock for free, so a cache renews itself indefinitely for as long as a conversation stays active. The window is idleness, not total elapsed time — 5 minutes on Claude, 30 minutes on GPT-5.6.

Your exact per-component rates are on the model's page in the console — cache read and write are priced per model, not as a fixed ratio.

The 1-hour cache

Claude offers a 1-hour lifetime. It is not supported here — do not send cache_control.ttl: "1h".

A 1-hour write costs 2× the input rate instead of 1.25×, which needs two subsequent reads to break even rather than one. Since a 5-minute cache already renews itself on every hit, the longer tier only helps when your gaps genuinely exceed five minutes.

Be aware that "not supported" is enforced on some routes and not others: on models we serve directly from Anthropic, ttl: "1h" is rejected with a 400 naming the marker's field: messages[].content[].cache_control.ttl on a message or system part, tools[].cache_control.ttl on a tool, or tools[].function.cache_control.ttl when the marker is inside the tool's function. On models reached through an aggregator, the same value may be accepted upstream — in which case you are charged the 2× write premium with none of the break-even maths done for you. Since which provider serves a given model can change, treat 1h as unavailable regardless of what a particular request appears to accept.

If your workload has long idle gaps and you want the 1-hour tier enabled properly, contact support rather than sending it speculatively.

Verifying it works

Send the same request twice and compare:

for i in range(2):
    r = client.chat.completions.create(model=MODEL, messages=MESSAGES)
    d = r.usage.prompt_tokens_details
    cached = (d.cached_tokens or 0) if d else 0
    print(f"call {i}: prompt={r.usage.prompt_tokens} cached={cached}")
 
# call 0: prompt=8421 cached=0
# call 1: prompt=8421 cached=8192

Some models omit prompt_tokens_details entirely when nothing was read from or written to the cache (Claude models served directly by Anthropic do), so treat a missing field as 0.

If the second call still reports cached_tokens: 0, work through these in order:

1. Is the prefix long enough? Below the model's minimum, the marker is ignored silently. This is the most common cause.
2. Is the prefix byte-identical? Caching is prefix matching. A timestamp, a session id, or a reordered JSON key at the start of your prompt invalidates everything after it. Move volatile content to the end.
3. On Claude, did you send cache_control, and where? There is no implicit caching. Put it on a content part or a tool: a marker on the message object itself or at the top level of the request is dropped on every model, and so is one in a tool message on Claude models served directly by Anthropic.
4. On GPT-5.6, did you send both the breakpoint and prompt_cache_options? And is the breakpoint the object {"mode": "explicit"} rather than true? A boolean is a 400, not a silent miss.
5. Did the window lapse since the last hit — 5 minutes on Claude, 30 on GPT-5.6?

Errors

A marker on a system message part is reported under messages[].content[], like any message part, and a request with more than 4 markers is reported on messages[].content[].cache_control even when some of them are on system parts or tools. A marker placed inside a tool's function, on a tool with no cache_control of its own, is reported as tools[].function.cache_control (and its .type and .ttl).

These are the errors we raise ourselves, with the offending field named in error.param. The same mistakes can instead surface as an upstream 400 in the provider's own wording, depending on which provider serves the model — so match on the status and your own request shape, not on the message text.

A request that is simply too short to cache is not an error — it returns 200 with no cache hit (cached_tokens: 0, or no prompt_tokens_details at all).