Kusunga prompt

Momwe imagwirire

• Prefix, mwa dongosolo — Prompt imawerengedwa mwa dongosolo: system prompt, tool definitions, kenako messages. Cache imagwirizana kuyambira ku chiyambi cha sequence imeneyo mpaka ku token yoyamba yomwe ndiyo yosiyana.
• Chimene chimawerengedwa kuti ndi hit — Request yomwe prompt yake iyambe ndi zomwe zili m'request yomwe inachitika posachedwa — nthawi zambiri ndi turn yapitayi ya conversation imodzi pomwe messages zatsopano zawonjezawa. Prefix yomwe imagwirizana ndi cached input; chilichonse chomwe chili pambuyo pake ndi regular input.
• Kugawikana — Cache imasunga prompt mu ma block a ma token 1,568, kotero prompt yochepera pafupifupi ma token 1,500 siisungidwa. Chiwerengero cha cached mu yankho ndi chiwerengero chanu cha input chochulukitsidwa ndi gawo la prompt lomwe lasungidwa, chozungulitsidwa pansi. Sichiyenera kukhala chochulukitsa cha kukula kwa block.
• Popanda hit — Pempho lomwe poyambira pake palibe mu cache limaperekedwa mtengo wa input wamba. Palibe nthawi yosunga ya ma prompt osungidwa yofalitsidwa ndipo hit siitsimikizika: werengani usage kuti muone zomwe pempho latenga ku cache.
• Palibe switch — Pempho silisankha kulowa, ndipo palibe field yomwe imazimitsa caching.
• Models uti — Hosted open-weight id iliyonse. GET /v1/models imapereka malipoti a capabilities.prompt_caching: true ndi pricing.cached_input_per_million_usd kwa iwo. Shannon models zimabilidwa mtengo umodzindo.

Onani cache hit mu yankho

Tumizani ma pempho awiri oyamba ndi system prompt yayitali yofanana ndipo musindikize usage ya lililonse. Nambala yoyamba ndi input ya pempho, yachiwiri ndi gawo lake lomwe linawerengedwa kuchokera ku cache.

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.shannon-ai.com/v1")

handbook = open("handbook.txt").read()  # a long text that stays the same


def ask(question):
    response = client.chat.completions.create(
        model="Kimi-K3-3BIT-REAP",
        messages=[
            {"role": "system", "content": handbook},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    print(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens)


ask("What is the refund policy?")
ask("Who approves travel?")  # same start: read the second number

import { readFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "YOUR_API_KEY", baseURL: "https://api.shannon-ai.com/v1" });

const handbook = readFileSync("handbook.txt", "utf8"); // a long text that stays the same

async function ask(question) {
  const response = await client.chat.completions.create({
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      { role: "system", content: handbook },
      { role: "user", content: question },
    ],
  });
  const usage = response.usage;
  console.log(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens);
}

await ask("What is the refund policy?");
await ask("Who approves travel?"); // same start: read the second number

# handbook.txt is a long text that stays the same. jq builds the JSON body from it
# and prints the usage object of the reply. Run it twice with different questions.
jq -Rs '{
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      {role: "system", content: .},
      {role: "user", content: "What is the refund policy?"}
    ]
  }' handbook.txt \
  | curl -s https://api.shannon-ai.com/v1/chat/completions \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d @- \
  | jq .usage

Mitengo

Cached input tokens zimabilidwa pa 25% ya mtengo wa input wa model, yoyikidwa ku $0.001 per 1M. Kulemba ku cache sikudya chilichonse chowonjezereka, ndipo output imabilidwa monga wamba. Cached rate ya id iliyense ili m'tebulo la Models & pricing. Ma model ndi mitengo

Input ya call imaperekedwa mtengo motere: (input − cached) × input rate + cached × cached rate. Chiwerengero cha cached sichipitirira chiwerengero cha input.

Usage log imalemba input yosungidwa ya call iliyonse. Ma token ake operekedwa mtengo ndi mtengo wake zili kale ndi cached rate. Keys & usage

Magawo ogwiritsira ntchito

{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 812,
    "total_tokens": 20812,
    "prompt_tokens_details": {
      "cached_tokens": 18000
    },
    "completion_tokens_details": {
      "reasoning_tokens": 604
    }
  }
}

{
  "usage": {
    "input_tokens": 20000,
    "input_tokens_details": {
      "cached_tokens": 18000
    },
    "output_tokens": 812,
    "output_tokens_details": {
      "reasoning_tokens": 604
    },
    "total_tokens": 20812
  }
}

{
  "usage": {
    "input_tokens": 2000,
    "cache_read_input_tokens": 18000,
    "cache_creation_input_tokens": 0,
    "output_tokens": 812
  }
}

Yankho la stream limakhala ndi ma field omwewo mu usage yake yomaliza. Simuyenera kuipempha:

Kukonza kuti mupeze cache hits zochuluka

• Sungani system prompt ndi tool definitions kuti zikhale zomwezo pakupita ndi kupita pa calls. Ikani zinthu zosintha monga timestamps kapena request ids kumaliza kwa message yomaliza, osati mu system prompt.
• Onjezani zinthu ku history okha. Kusintha, kukometsa kapena kuchititsa summaries za turns zakale kusintha prefix, ndipo chilichonse chomwe chili pambuyo pa kusintha kumeneku chimabilidwa monga regular input.
• Musasintha dongosolo la tools, messages kapena content blocks pakati pa calls, ndipo serialise JSON (tool schemas, tool arguments ndi results) m'njira imodzi nthawi zonse.
• Khalani pa model id imodzi pa zokambirana, ndipo tumizani call yotsatira posachedwa pambuyo pa yapitayo.

API imasunga poyambira pa zokambirana mosasintha mu izi:

• Message ya system kapena developer yotumizidwa pambuyo pake pa zokambirana imakhala pamalo ake. Siisintha poyambira pa prompt, kotero ma turn omwe ali patsogolo pake amakhalabe osungidwa.
• Ma argument a ma call a tool mu ma turn akale a assistant amafananizidwa ndi mtengo. Dongosolo la ma key ndi ma space a JSON imeneyo sizimakhudza.
• Ma endpoint atatuwa amawerenga zokambirana mofanana. Zokambirana zopitirizidwa pa endpoint ina zimasunga prefix yogawana pamene zomwe zili ndi zofanana.

Fields za request

prompt_cache_key (Chat Completions and Responses) ndi cache_control pa Messages content blocks zimlandiridwa, choncho code ya client yomwe ilipo imagwira ntchito popanda kusintha. Aliyense sakuphikira: caching imachitika okha ndipo imagwira ntchito mofanana popanda iwo.

Kuwerengera tokens

Ma endpoint awiri aulere, POST /v1/tokenize ndi POST /v1/messages/count_tokens, amawerenga ma token a text kapena a pempho lonse la hosted open-weight model musanatumize. Ali ndi tsamba lawo: Kuwerenga ma token