Prompt caching

ከመ’ይ ይ-Cሉ

• Prefix፣ ብቅደም — Prompt ብቅደም ይረክብ፡ system prompt፣ tool definitions፣ ድሕሪኡ መልእኽታት። እቲ ካሽ ካ’ብቲ መጅመርታ ናብቲ ቀዳማይ ዝተፈላለየ token ይ-Cሉ (matches) እዩ።
• እንታይ ከም hit ይቑጸር — ሓደ ት’ላብቲ እ’ቲ prompt ኡ ከም ር’Cቲ ር’Cቲ ዝነበረ ት’ላብቲ ብምጅማር — ብ’ብ’-ብ’-ብ’ ብ-Cቲ ሓድሽ መልእኽታትን ተ-Cሊሆም ዝ-append ተ-Cሊሆም ዝነበሩ መልእኽታትን። እቲ ተጋራርዩ ዝተረክበ prefix ካሽድ ኣተዋብ እዩ፤ ኩሉ ካብኡ ጻብና ዝመጸ ግን ር’Cቲ ኣተዋብ እዩ።
• ዝርዝርነት (Granularity) — እቲ cache prompt ብ blocks ናይ 1,568 tokens ይሕዞ፣ ስለዚ ካብ ኣስታት 1,500 tokens ዝሓጸረ prompt ኣይኽዘንን። እቲ ቁጽሪ cached ኣብ ምላሽ ቁጽሪ ኣተዋዕዎካ ብክፍሊ cached ናይ prompt ዝተዋሕለለ፣ ናብ ታሕቲ ዝተጠጋጋዐ እዩ። ናይ block ዓቐን ዝበዝሑ ክኸውን ኣየድልን።
• ብዘይ hit — መጀመርታኡ ኣብ cache ዘየለ ጠለብ ብስሩዕ ዋጋ ኣተዋዕዎ ይኽፈል። ንcached prompts ዕድመ ኣይታወጀን ከምኡውን hit ኣይተረጋገጸን፦ ጠለብ ካብ cache እንታይ ከም ዝወሰደ ንምርኣይ usage ኣንብብ።
• መቐያየሪ የለን — ጠለብ ኣይመዝገብን፣ ከምኡውን ሓደ field caching ኣይዓጽዎን።
• Cናይ ር’Cቲ ሞዴላት — ኩሉ hosted open-weight id። GET /v1/models capabilities.prompt_caching: trueን pricing.cached_input_per_million_usdን ይርክብ። Shannon models ሓደ ጽኑይ ር’Cቲ ይክፍሉ።

ኣብ ምላሽ cache hit ርአ

ብተመሳሳሊ ነዊሕ system prompt ዝጅምሩ ክልተ ጠለባት ልኣኽ እሞ usage ናይ ነፍሲ ወከፍ ሕተም። እቲ ቀዳማይ ቁጽሪ ኣተዋዕዎ ናይ ጠለብ እዩ፣ እቲ ካልኣይ ድማ ካብ cache ዝተነበበ ክፍሊኡ።

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.shannon-ai.com/v1")

handbook = open("handbook.txt").read()  # a long text that stays the same


def ask(question):
    response = client.chat.completions.create(
        model="Kimi-K3-3BIT-REAP",
        messages=[
            {"role": "system", "content": handbook},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    print(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens)


ask("What is the refund policy?")
ask("Who approves travel?")  # same start: read the second number

import { readFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "YOUR_API_KEY", baseURL: "https://api.shannon-ai.com/v1" });

const handbook = readFileSync("handbook.txt", "utf8"); // a long text that stays the same

async function ask(question) {
  const response = await client.chat.completions.create({
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      { role: "system", content: handbook },
      { role: "user", content: question },
    ],
  });
  const usage = response.usage;
  console.log(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens);
}

await ask("What is the refund policy?");
await ask("Who approves travel?"); // same start: read the second number

# handbook.txt is a long text that stays the same. jq builds the JSON body from it
# and prints the usage object of the reply. Run it twice with different questions.
jq -Rs '{
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      {role: "system", content: .},
      {role: "user", content: "What is the refund policy?"}
    ]
  }' handbook.txt \
  | curl -s https://api.shannon-ai.com/v1/chat/completions \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d @- \
  | jq .usage

ዋጋ

Cached input tokens ብ 25% ናይቲ ሞዴል ኣተዋብ ር’Cቲ ይክፍሉ፣ ናብ $0.001 ን 1M ይ-Cሉ (rounded)። ናብ ካሽ ምጽሓፍ ዝ-Cሊሆም ተ-Cሊሆም የለን፣ ውጽዋትን ከምቲ ዝ ዝ’Cሉ ይክፈሉን። ናይ ር’ሻ-id ካሽድ ር’Cቲ ኣብቲ Models & pricing table ይርከብ። ሞዴላትን ዋጋን

ኣተዋዕዎ ናይ ጥሪ ከም (ኣተዋዕዎ − cached) × ዋጋ ኣተዋዕዎ + cached × ዋጋ cached ይኽፈል። ቁጽሪ cached ካብ ቁጽሪ ኣተዋዕዎ ፈጺሙ ኣይዓቢን።

እቲ usage log cached ኣተዋዕዎ ናይ ነፍሲ ወከፍ ጥሪ ይዘርዝር። ዝተኸፍሉ tokens ከምኡውን ዋጋኡ ዋጋ cached ኣቀሚጦም ኣለዉ። ቁልፍታትን ኣጠቓቕማን

ናይ ኣ-’ውሳ-ት 字段ታት

{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 812,
    "total_tokens": 20812,
    "prompt_tokens_details": {
      "cached_tokens": 18000
    },
    "completion_tokens_details": {
      "reasoning_tokens": 604
    }
  }
}

{
  "usage": {
    "input_tokens": 20000,
    "input_tokens_details": {
      "cached_tokens": 18000
    },
    "output_tokens": 812,
    "output_tokens_details": {
      "reasoning_tokens": 604
    },
    "total_tokens": 20812
  }
}

{
  "usage": {
    "input_tokens": 2000,
    "cache_read_input_tokens": 18000,
    "cache_creation_input_tokens": 0,
    "output_tokens": 812
  }
}

ናይ stream ምላሽ ኣብ መወዳእታ usage ተመሳሳሊ fields ይሓዝ። ክትሓትቶ ኣየድልን፦

ተ-Cሊሆም cache hits ንምርካብ

• System promptን tool definitionsን ኣብ ር’Cቲ ት’ላብቲ ብ-Cቲ-ባይት (byte-for-byte) stable ግብሮም። ተለዋዋጥ ዋጋታት ከም timestamps ወይ request ids ኣብ መወዳ-Cቲ ዝ’Cሉ መልእኽቲ ኣ-Cሉ፣ ኣብ system prompt ኣይ-Cሉን።
• ናብ history ብ-Cቲ-ማቲካሊ append ግብር። ቀዳማይ ዙርያትን (turns) ም-edit-ን፣ ም-trim-ን ወይ ም-summarise-ን እቲ prefix ይቕይሮ፣ ድሕሪቲ ቕላይ ዝመጸ ኩሉ ር’Cቲ ኣተዋብ ኮይኑ ይክፈል።
• ኣብ መንጎ ት’ላብቲ tools፣ መልእኽታትን content blocksን ኣይ-Cሉን፣ JSON (tool schemas, tool arguments and results) ድማ ኩሉ ግዜ ብሓደ መሳሰሊያ ሰርዮም።
• ንሓደ ዘተ ኣብ ሓደ ሞዴል id ጸንሕ፣ እቲ ቀጻሊ ጥሪ ድማ ድሕሪ እቲ ቀዳማይ ቀልጢፍካ ልኣኽ።

እቲ API መጀመርታ ናይ ዘተ ኣብዚ ኩነታት ርጉእ ይሕዞ፦

• ኣብ ዘተ ደሓር ዝተላእከ system ወይ developer መልእኽቲ ኣብ ቦታኡ ይተርፍ። መጀመርታ ናይ prompt ኣይቕይርን፣ ስለዚ ቅድሚኡ ዘለዉ turns cached ይተርፉ።
• Arguments ናይ tool ጥሪታት ኣብ ዝሓለፉ assistant turns ብዋጋ ይነጻጸሩ። ትእዛዝ key ከምኡውን ክፍተት ናይ እቲ JSON ኣየገድስን።
• እቶም ሰለስተ endpoints ዘተ ብተመሳሳሊ ኣገባብ የንብቡ። ኣብ ካልእ endpoint ዝቐጸለ ዘተ ትሕዝቶኡ ሓደ እንተ ኾይኑ ሓቢሩ ዘለዎ prefix ይሕዝ።

ናይ ትሕቲ ጥሪ ፊልድታት

prompt_cache_key (Chat Completionsን Responsesን) ከምኡ’ን cache_control ኣብ content blocks’ን ተቐቢሎም እዮም፣ ስለዚ ዝርከብ ናይ client ኮድ ብዘይብሉ ለውጢ ይሰርሕ። ክልቲሎም ኣይላዘሙን እዮም፦ caching ብኣውቶማቲክ ይስራሕን ብዘይኣምላኽኦም ብዘ saman ይቕጽልን እዩ።

ናይ tokens ቆጻሪ

ክልተ ነጻ endpoints፣ POST /v1/tokenize ከምኡውን POST /v1/messages/count_tokens፣ ቅድሚ ምልኣኽካ ንዝተሃንደሱ open-weight ሞዴላት tokens ናይ ጽሑፍ ወይ ምሉእ ጠለብ ይቝጽሩ። ናቶም ገጽ ኣለዎም፦ ምቑጻር Token