Caching prompt

Otu ọ na-arụ ọrụ

• Prefix, n'usoro — A na-agụ prompt n'usoro: system prompt, tool definitions, mgbe ahụ ka a gụọ messages. Cache na-eme match site na mmalize sequence ahụ ruo na token mbunye nke dị iche.
• Ihe a na-agụ dịka hit — Request nke prompt ya malitere na content nke gaghara — nke na-abụkarị turn mbunye nke conversation ahụ na messages ọhụrụ a gbakwunyele. Prefix a match-rị bụ cached input; ihe niile ndị gbasoro ya bụ regular input.
• Ọkwa nkewa — Cache na-ejide prompt n'ime blocks nke tokens 1,568, yabụ a naghị echekwa prompt dị mkpụmkpụ karịa tokens 1,500 ma ọ bụ ihe ruru ya. Ọnụọgụ cached n'ime azịza bụ ọnụọgụ input gị nke e jiri òkè cached nke prompt mụbaa, wedata ala. Ọ bụghị mgbe niile ka ọ bụ ọtụtụ nke nha block.
• Na-enweghị hit — A na-akwụ ụgwọ request nke mmalite ya na-adịghị na cache n'ọnụego input nkịtị. A naghị ebipụta oge ndụ maka prompts echekwara ma hit abụghị ihe e kwere nkwa: gụọ usage ka ị hụ ihe request weere na cache.
• Enweghị switch — Request anaghị abanye, enweghịkwa field na-agbanyụ caching.
• Mbee models — Every hosted open-weight id. GET /v1/models na-akọwa capabilities.prompt_caching: true na pricing.cached_input_per_million_usd maka ha. Shannon models na-akwụ ụgwọ ọnụgoเดียว.

Hụ cache hit n'ime azịza

Zipu requests abụọ malitere na otu system prompt ogologo ma bipụta usage nke ọ bụla. Nọmba mbụ bụ input nke request, nke abụọ bụ akụkụ ya e sitere na cache gụọ.

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.shannon-ai.com/v1")

handbook = open("handbook.txt").read()  # a long text that stays the same


def ask(question):
    response = client.chat.completions.create(
        model="Kimi-K3-3BIT-REAP",
        messages=[
            {"role": "system", "content": handbook},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    print(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens)


ask("What is the refund policy?")
ask("Who approves travel?")  # same start: read the second number

import { readFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "YOUR_API_KEY", baseURL: "https://api.shannon-ai.com/v1" });

const handbook = readFileSync("handbook.txt", "utf8"); // a long text that stays the same

async function ask(question) {
  const response = await client.chat.completions.create({
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      { role: "system", content: handbook },
      { role: "user", content: question },
    ],
  });
  const usage = response.usage;
  console.log(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens);
}

await ask("What is the refund policy?");
await ask("Who approves travel?"); // same start: read the second number

# handbook.txt is a long text that stays the same. jq builds the JSON body from it
# and prints the usage object of the reply. Run it twice with different questions.
jq -Rs '{
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      {role: "system", content: .},
      {role: "user", content: "What is the refund policy?"}
    ]
  }' handbook.txt \
  | curl -s https://api.shannon-ai.com/v1/chat/completions \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d @- \
  | jq .usage

Ugolu

Cached input tokens na-akwụ ụgwọ na 25% nke ọnụgo input nke model ahụ, a gbatie ya ruo $0.001 per 1M. Idebe ihe na cache anaghị akwụ ụgwọ extra, output akwụkwụsịla dịka ọmụma. Ọnụgo cached nke id ọ bụla nọ na Models & pricing table. Models & ọnụahịa

A na-akwụ ụgwọ input nke oku dị ka (input − cached) × input rate + cached × cached rate. Ọnụọgụ cached anaghị agafe ọnụọgụ input.

Usage log na-edepụta cached input nke oku ọ bụla. Tokens e kwụrụ ụgwọ ya na ọnụahịa ya agụnyela cached rate. Keys & usage

Mpụzụ ojiji

{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 812,
    "total_tokens": 20812,
    "prompt_tokens_details": {
      "cached_tokens": 18000
    },
    "completion_tokens_details": {
      "reasoning_tokens": 604
    }
  }
}

{
  "usage": {
    "input_tokens": 20000,
    "input_tokens_details": {
      "cached_tokens": 18000
    },
    "output_tokens": 812,
    "output_tokens_details": {
      "reasoning_tokens": 604
    },
    "total_tokens": 20812
  }
}

{
  "usage": {
    "input_tokens": 2000,
    "cache_read_input_tokens": 18000,
    "cache_creation_input_tokens": 0,
    "output_tokens": 812
  }
}

Azịza a na-eme streaming na-ebu otu fields n'ime usage ikpeazụ ya. Ọ dịghị mkpa ka ị rịọ ya:

Otu esi nweta cache hits ndịzi

• Mechie system prompt na tool definitions ka ha ghọzie byte-for-byte n'ime calls. Debe values nke call kachaa dịka timestamps ma ọ bụ request ids n'ikpeazụ nke message kachaa, ọ bụghị n'ime system prompt.
• Gbakwunye naanị na history. Ime editing, trimming ma ọ bụ summarising na turns mbunye na-agbanwe prefix, own ihe niile gbasoro mgbanwe mbunye ahụ a na-akwụ ụgwọ ya dịka regular input.
• Atụla tools, messages ma ọ bụ content blocks n'usoro iche n'etiti calls, ma mee serialise JSON (tool schemas, tool arguments na results) n'otu ụzọ mgbe niile.
• Nọgide na otu model id maka mkparịta ụka, ma zipu oku na-esote n'oge na-adịghị anya mgbe nke gara aga gasịrị.

API na-edebe mmalite mkparịta ụka kwụsiri ike n'ikpe ndị a:

• Ozi system ma ọ bụ developer e zigara mgbe e mesịrị n'ime mkparịta ụka na-anọgide n'ọnọdụ ya. Ọ naghị agbanwe mmalite nke prompt, yabụ turns tupu ya na-anọgide n'ime cache.
• A na-atụnyere arguments nke oku tool n'ime assistant turns gara aga site na uru. Usoro key na oghere nke JSON ahụ adịghị eme ihe.
• Endpoints atọ ahụ na-agụ mkparịta ụka n'otu ụzọ. Mkparịta ụka e jiri n'endpoint ọzọ na-ejigide shared prefix ya mgbe ọdịnaya bụ otu.

Fields a rịọrọ

A nabatara prompt_cache_key (Chat Completions na Responses) na cache_control na Messages content blocks, n'ihi nke ahụ, koodu client dịbe anya agaghị agbanwe. Ọ bụghịkwa na ha dị mkpa: caching abụrụla automatik ma na-arụ ọrụ nke ahụ n'enweghị ha.

Ịgụnume tokens

Endpoints abụọ n'efu, POST /v1/tokenize na POST /v1/messages/count_tokens, na-agụ tokens nke text ma ọ bụ nke request dum maka hosted open-weight models tupu i zipu ya. Ha nwere peeji nke ha: Ịgụ tokens