Kaydinta Prompt-ka

Sida ay u shaqayso

• Prefix, sida ay u kala horrayn — Prompt-ka waxaa loo akhriyaa sidaan: system prompt, qeexidda tools-ka, ka dibna fariimaha. Cache-ku wuxuu is-waafajiyaa laga bilaabo bilowga taxanahaas ilaa token-ka ugu horreeya ee isbeddelka yimaado.
• Maxaa loo aqoonsadaa hit — Codsi uu prompt-kiisu ku bilaabo content isku mid ah codsi dhowaan dhacay — sida caadiga ah waa wejigii hore ee wada sheekaysiga oo fariimo cusub lagu daray. Prefix-ka is-waafaqay waa cached input; wax kasta oo ka dambeeya waa input caadi ah.
• Kala-yabasho — Cache-ku wuxuu prompt-ka ku hayaa blocks ah 1,568 token, sidaas darteed prompt ka gaaban qiyaastii 1,500 token lama cache-gareeyo. Tirada cached ee jawaabta waa tirada input-kaaga oo lagu dhufto qaybta cached ee prompt-ka, hoos loo wareejiyay. Lagama maarmaan ma aha inay noqoto isku-dhufte cabbirka block-ga.
• Hit la'aan — Codsi bilowgiisu cache-ka ku jirin waxaa lagu dallacaa qiimaha input-ka caadiga ah. Cimri lama daabaco prompts-ka cached, hit-na lama dammaanad qaado: akhri usage si aad u aragto waxa codsigu cache-ka ka qaaday.
• Furfur ma jiro — Codsi ma geli doono, field-na ma damiyo caching.
• Model-yada ay khuseyso — Id kasta oo hosted open-weight ah. GET /v1/models waxay soo sheegaysaa capabilities.prompt_caching: true iyo pricing.cached_input_per_million_usd. Shannon models-ku waxay bixiyaan hal heer oo go'an.

Arag cache hit jawaab dhexdeeda

Dir laba codsi oo ku bilaabma isla system prompt dheer oo daabac usage mid kasta. Lambarka ugu horreeya waa input-ka codsiga, kan labaad waa qaybta laga akhriyay cache.

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.shannon-ai.com/v1")

handbook = open("handbook.txt").read()  # a long text that stays the same


def ask(question):
    response = client.chat.completions.create(
        model="Kimi-K3-3BIT-REAP",
        messages=[
            {"role": "system", "content": handbook},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    print(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens)


ask("What is the refund policy?")
ask("Who approves travel?")  # same start: read the second number

import { readFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "YOUR_API_KEY", baseURL: "https://api.shannon-ai.com/v1" });

const handbook = readFileSync("handbook.txt", "utf8"); // a long text that stays the same

async function ask(question) {
  const response = await client.chat.completions.create({
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      { role: "system", content: handbook },
      { role: "user", content: question },
    ],
  });
  const usage = response.usage;
  console.log(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens);
}

await ask("What is the refund policy?");
await ask("Who approves travel?"); // same start: read the second number

# handbook.txt is a long text that stays the same. jq builds the JSON body from it
# and prints the usage object of the reply. Run it twice with different questions.
jq -Rs '{
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      {role: "system", content: .},
      {role: "user", content: "What is the refund policy?"}
    ]
  }' handbook.txt \
  | curl -s https://api.shannon-ai.com/v1/chat/completions \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d @- \
  | jq .usage

Qiimaha

Cached input tokens-ka waxaa lagu xisaabiyaa 25% heerka input-ka model-ka, waxaana loo gooyaa $0.001 per 1M. Qoraalka cache-ka wax kharash dheeraad ah ma leh, output-kana waxaa loo xisaabiyaa sida caadiga ah. Heerka cached-ka ee id kasta wuxuu ku jiraa jadwalka Models & pricing. Model-yo & qiimo

Input-ka wicitaan waxaa lagu dallacaa sida (input − cached) × qiimaha input + cached × qiimaha cached. Tirada cached marnaba kama weyn tirada input-ka.

Diiwaanka isticmaalka wuxuu liis garaynayaa input-ka cached ee wicitaan kasta. Token-yada la dallacay iyo kharashkiisa horey ayay qiimaha cached u ku jiraan. Fure & isticmaal

Goobaha isticmaalka

{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 812,
    "total_tokens": 20812,
    "prompt_tokens_details": {
      "cached_tokens": 18000
    },
    "completion_tokens_details": {
      "reasoning_tokens": 604
    }
  }
}

{
  "usage": {
    "input_tokens": 20000,
    "input_tokens_details": {
      "cached_tokens": 18000
    },
    "output_tokens": 812,
    "output_tokens_details": {
      "reasoning_tokens": 604
    },
    "total_tokens": 20812
  }
}

{
  "usage": {
    "input_tokens": 2000,
    "cache_read_input_tokens": 18000,
    "cache_creation_input_tokens": 0,
    "output_tokens": 812
  }
}

Jawaab stream ah waxay ku sidataa isla field-yadaas usage-keeda ugu dambeeya. Uma baahnid inaad codsato:

Sida loo helo cache hits badan

• Ka dhig system prompt-ka iyo qeexidda tools-ka kuwa isku mid ah (byte-for-byte) inta u dhaxaysa codsiyada. Qiimayaasha codsi kastaa leeyahay sida timestamps ama request ids geli dhamaadka fariinta ugu dambeysa, ha gelin system prompt-ka.
• Kaliya ku dar (append) taariikhda. Haddii aad wax ka beddesho, goo-gooyso ama soo koobto wejiyada hore, waxay beddelaysaa prefix-ka, wax kasta oo ka dambeeya isbeddelka ugu horreeya waxaa loo xisaabiyaa input caadi ah.
• Ha beddelin kala horreynta tools-ka, fariimaha ama content blocks-ka inta u dhaxaysa codsiyada, oo u serialize-garee JSON (tool schemas, tool arguments iyo results) si isku mid ah mar kasta.
• Wada-hadal ku hay hal id model, oo wicitaanka xiga dir muddo gaaban ka dib kii hore.

API-gu wuxuu bilowga wada-hadalka ku hayaa mid joogto ah kiisaskan:

• Fariin system ama developer oo wada-hadalka dhexdiisa dib loogu diray waxay joogtaa meesheeda. Ma beddesho bilowga prompt-ka, sidaas darteed wareegyada ka horreeya way cached ahaadaan.
• Arguments-ka wicitaannada tool ee wareegyada assistant ee hore waxaa lagu barbar dhigaa qiime. Kala horreynta furayaasha iyo meelaha bannaan ee JSON-kaas muhiim ma aha.
• Saddexda endpoint waxay wada-hadalka u akhriyaan si isku mid ah. Wada-hadal lagu sii waday endpoint kale wuxuu hayaa prefix-kiisa la wadaago marka nuxurku isku mid yahay.

Field-yada codsiga

prompt_cache_key (Chat Completions and Responses) iyo cache_control ee content blocks-ka Messages waa laga aqbalaa, sidaas darteed koodhka client-ka ee jiraa wuxuu u shaqaynayaa isbeddel la'aan. Midna ma aha qasab: caching-u waa otomaatik ah, waana isku mid xitaa haddii aysan jirin.

Tirinta tokens-ka

Laba endpoint oo bilaash ah, POST /v1/tokenize iyo POST /v1/messages/count_tokens, waxay tiriyaan token-yada qoraal ama codsi dhan ee model-yada open-weight ee la martigeliyo ka hor inta aadan dirin. Waxay leeyihiin bog u gaar ah: Tirinta token-yada