Adana prompt

Yadda yake aiki

• Prefix, bisa tsari — Ana karanta prompt ɗin bisa tsari: system prompt, tool definitions, sannan saƙonni. Cache ɗin yana dacewa daga farkon wannan tsarin har zuwa token na farko wanda ya bambanta.
• Me ake kira hit — Request wanda prompt ɗinsa ya fara da abu ɗan ɗaya da wani request na baya — yawanci zagaye na baya na wannan tattaunawa tare da ƙarin saƙonni sababbi. Matching prefix ɗin shine cached input; duk abin da yake bayansa shine regular input.
• Karin bayani — Cache yana riƙe prompt a blocks na tokens 1,568, don haka prompt da bai kai kusan tokens 1,500 ba ba a yi masa cache. Adadin cached a amsa shi ne adadin input ɗinku wanda aka ninka da kason prompt da aka yi cache, an zagaye zuwa ƙasa. Ba lallai ba ne ya zama ninki na girman block.
• Ba tare da hit ba — Ana lissafin request wanda farkonsa ba ya cikin cache a regular input rate. Ba a wallafa tsawon rayuwar prompts da aka yi cache ba kuma hit ba tabbas ba ne: ku karanta usage don ganin abin da request ya ɗauka daga cache.
• Babu maɓallin kunnawa — Request ba ya shiga da kansa, kuma babu field da ke kashe caching.
• Wane models — Kowane hosted open-weight id. GET /v1/models yana bayyana capabilities.prompt_caching: true da pricing.cached_input_per_million_usd gare su. Shannon models suna lissafa farashi guda ɗaya.

Ganin cache hit a amsa

Ku aika requests biyu da suka fara da system prompt mai tsawo iri ɗaya, ku buga usage na kowanne. Lamba ta farko ita ce input na request, ta biyu kuma ɓangaren da aka karanta daga cache.

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.shannon-ai.com/v1")

handbook = open("handbook.txt").read()  # a long text that stays the same


def ask(question):
    response = client.chat.completions.create(
        model="Kimi-K3-3BIT-REAP",
        messages=[
            {"role": "system", "content": handbook},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    print(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens)


ask("What is the refund policy?")
ask("Who approves travel?")  # same start: read the second number

import { readFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "YOUR_API_KEY", baseURL: "https://api.shannon-ai.com/v1" });

const handbook = readFileSync("handbook.txt", "utf8"); // a long text that stays the same

async function ask(question) {
  const response = await client.chat.completions.create({
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      { role: "system", content: handbook },
      { role: "user", content: question },
    ],
  });
  const usage = response.usage;
  console.log(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens);
}

await ask("What is the refund policy?");
await ask("Who approves travel?"); // same start: read the second number

# handbook.txt is a long text that stays the same. jq builds the JSON body from it
# and prints the usage object of the reply. Run it twice with different questions.
jq -Rs '{
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      {role: "system", content: .},
      {role: "user", content: "What is the refund policy?"}
    ]
  }' handbook.txt \
  | curl -s https://api.shannon-ai.com/v1/chat/completions \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d @- \
  | jq .usage

Farashi

Cached input tokens ana lissafarsu da 25% na farashin input na model ɗin, wanda aka zagaya zuwa $0.001 kowace 1M. Rubutu zuwa cache ba ya buƙatar ƙarin kuɗi, kuma output ana lissafinsa kamar yadda aka saba. Farashin cached na kowane id yana cikin teburin Models & pricing. Models & farashi

Ana cajin input na kira kamar (input − cached) × input rate + cached × cached rate. Adadin cached ba ya taɓa wuce adadin input.

Usage log yana lissafa cached input na kowane kira. Tokens da aka caje da kuɗin sun riga sun haɗa da cached rate. Maɓallai & amfani

Hanyoyin amfani

{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 812,
    "total_tokens": 20812,
    "prompt_tokens_details": {
      "cached_tokens": 18000
    },
    "completion_tokens_details": {
      "reasoning_tokens": 604
    }
  }
}

{
  "usage": {
    "input_tokens": 20000,
    "input_tokens_details": {
      "cached_tokens": 18000
    },
    "output_tokens": 812,
    "output_tokens_details": {
      "reasoning_tokens": 604
    },
    "total_tokens": 20812
  }
}

{
  "usage": {
    "input_tokens": 2000,
    "cache_read_input_tokens": 18000,
    "cache_creation_input_tokens": 0,
    "output_tokens": 812
  }
}

Amsar da aka yi streaming tana ɗauke da fields iri ɗaya a usage ɗinta na ƙarshe. Ba sai kun nema ba:

Yadda za a samu cache hits da yawa

• Ka tabbatar system prompt da tool definitions suna da tsayayyen bayani a kowane kira. Sanya darajojin kowane kira kamar timestamps ko request ids a ƙarshen saƙon na ƙarshe, ba a cikin system prompt ba.
• Karin bayani kawai a tarihin (history). Gyara, rage ko taƙa tattaunawa ta baya yana canza prefix ɗin, kuma duk abin da yake bayan canjin na farko ana lissafinsa a matsayin regular input.
• Kada ka canza tsarin tools, saƙonni ko content blocks tsakanin kira, kuma yi serialise na JSON (tool schemas, tool arguments da results) a hanya ɗa ɗaya a kowane lokaci.
• Ku tsaya a model id ɗaya a duk tattaunawa, kuma ku aika kira na gaba jim kaɗan bayan na baya.

API yana riƙe farkon tattaunawa a tsaye a waɗannan yanayi:

• Saƙon system ko developer da aka aika daga baya a tattaunawa yana nan a wurinsa. Ba ya canza farkon prompt, don haka turns ɗin da suka riga shi suna nan a cache.
• Ana kwatanta arguments na kiran tool a turns na assistant na baya ta ƙima. Tsarin keys da tazarar wannan JSON ba su da muhimmanci.
• Endpoints uku suna karanta tattaunawa iri ɗaya. Tattaunawar da aka ci gaba a wani endpoint tana riƙe prefix ɗinta na tarayya idan abun ciki iri ɗaya ne.

Kayan request

Ana amfani da prompt_cache_key (Chat Completions da Responses) da cache_control a block ɗin content na Messages, don haka lambobin client na yanzu suna gudanarwa ba tare da canji ba. Babu wanda ake bukata: caching na automatic ne kuma yana aiki a haka koda babu su.

Lissafin tokens

Endpoints biyu na kyauta, POST /v1/tokenize da POST /v1/messages/count_tokens, suna ƙirga tokens na rubutu ko na dukan request ga hosted open-weight models kafin ku aika. Suna da shafinsu: Ƙidayar tokens