Prompt caching

Reuse a stable prompt prefix across calls to cut both cost and time-to-first-token. Caching works on every surface and needs no special endpoint.

How caching works

When consecutive requests share an identical leading prefix — a long system prompt, a tool catalog, a document you keep asking about — that prefix can be served from a cache instead of reprocessed. The first request writes the cache; later requests read it, paying a fraction of the input cost and starting to respond sooner. Caches are short-lived and scoped to your account.

Exact-match prefix, not semantic

Caching matches an identical leading prefix — the same tokens in the same order from the start of the request. It is not semantic or vector caching: reordering content or paraphrasing the same idea will not hit the cache. Keep the cached prefix byte-for-byte stable across calls and put the part that changes (the user’s question) last.

Minimum prefix size

Caching activates once the stable prefix is large enough to be worth storing — as a rule of thumb, expect it to start helping around a thousand or more tokens of shared leading content. Very short prefixes may not cache at all, so there is no benefit to marking a one-line system prompt. The exact threshold is handled for you; you do not need to tune it.

Cache lifetime (TTL)

Caches are short-lived. You can pass a TTL hint (for example "ttl": "1h") alongside cache_control to ask for a longer-lived cache; it is honored where the serving model supports it and otherwise ignored. Treat the TTL as a hint, not a guarantee — design your usage so a cache miss is always correct, just slower and a little more expensive.

Isolated per account

Your cache is keyed to your account — another customer’s identical prompt never reads it, and yours never reads theirs. Isolation is automatic; there is nothing to configure.

Marking content to cache

Add a cache_control breakpoint to the end of the content you want cached. Everything up to that point becomes the cacheable prefix. The Anthropic surface takes it natively; the OpenAI-compatible surface accepts it on content parts too.

Python

from anthropic import Anthropic

client = Anthropic(api_key="llm_live_...", base_url="https://api.directinference.com/di")

msg = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=512,
    system=[
        {
            "type": "text",
            "text": LONG_STABLE_INSTRUCTIONS,        # reused on every call
            "cache_control": {"type": "ephemeral"},  # mark the prefix as cacheable
        },
    ],
    messages=[{"role": "user", "content": "What changed in section 4?"}],
)

print(msg.usage)   # cache_creation_input_tokens / cache_read_input_tokens

TypeScript

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ apiKey: "llm_live_...", baseURL: "https://api.directinference.com/di" });

const msg = await client.messages.create({
  model: "claude-sonnet-4-6",
  max_tokens: 512,
  system: [
    {
      type: "text",
      text: LONG_STABLE_INSTRUCTIONS,        // reused on every call
      cache_control: { type: "ephemeral" }, // mark the prefix as cacheable
    },
  ],
  messages: [{ role: "user", content: "What changed in section 4?" }],
});

console.log(msg.usage); // cache_creation_input_tokens / cache_read_input_tokens

curl

# cache_control is also accepted on the OpenAI-compatible surface.
curl https://api.directinference.com/di/v1/chat/completions \
  -H "Authorization: Bearer llm_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.5-mini",
    "messages": [
      { "role": "system", "content": [
        { "type": "text", "text": "<long stable instructions>",
          "cache_control": { "type": "ephemeral" } }
      ]},
      { "role": "user", "content": "What changed in section 4?" }
    ]
  }'

HTTPie

# Same request via HTTPie; cache_control rides on the OpenAI-compatible surface.
echo '{
  "model": "gpt-5.5-mini",
  "messages": [
    { "role": "system", "content": [
      { "type": "text", "text": "<long stable instructions>",
        "cache_control": { "type": "ephemeral" } }
    ]},
    { "role": "user", "content": "What changed in section 4?" }
  ]
}' | https POST api.directinference.com/di/v1/chat/completions \
  Authorization:'Bearer llm_live_...'

Order matters: stable content first

Put the parts that never change (instructions, schemas, documents) at the front and mark the breakpoint after them; keep the volatile user turn last. A prefix only hits the cache when it is byte-for-byte identical to a previous request, so anything that varies per call must come after the breakpoint.

Reading cache stats

Every response reports what the cache did, in the native shape of the surface you called. On the OpenAI-compatible surface, usage.prompt_tokens_details.cached_tokens and cache_write_tokens carry the read/write split. On the Anthropic surface, that split is cache_read_input_tokens and cache_creation_input_tokens.

How caching is billed

A cached request is billed in four buckets. Reads are far cheaper than fresh input, which is where the savings come from; a write costs a little more than input once, then pays for itself across subsequent reads.

Bucket	What it is
Uncached input	Input tokens read fresh — your normal input rate.
Cache read	Input tokens served from a prior write — billed at a steep discount to input.
Cache write	The one-time cost of storing a new prefix, slightly above the input rate.
Output	Generated tokens — your normal output rate, never cached.

Cache savings in the dashboard

You do not have to add up tokens by hand. The portal surfaces cache activity and the money it saved per request and in aggregate.

Where to look

The Overview shows a cache savings strip and timeline when there is cache activity, and each row in Usage & analytics breaks down cached vs. uncached tokens and the savings.