Kuchenza kwe-prompt

Inoshanda sei

• Prefix, maererano — Prompt inoverengwa maererano: system prompt, tool definitions, apo zvinotevera messages. Cache inobvumirana kubva kutanga kwe-sequence iyoyo kusvika pa-token yekutanga inosiyana.
• Zvinonzi hit — Request ine prompt inotanga ne-content yakafanana ne-request yakaitwa kare — kazhinji imapages ekupedzisira kwemashandiso emu-conversation imwe chete vane messages idzva dzakabatirwa. Matching prefix i-cached input; zvese zvinotevera izvo i-regular input.
• Kugadzirwa — Cache inochengeta prompt mu-blocks dze-1,568 tokens, saka prompt pfupi kupfuura 1,500 tokens inenge haina kuchenzwa. Cached count mumhinduro i-input count yako inowedzerwa nechikamu che-prompt chakachenzwa, chakadzikiswa. Hazvirevi kuti iri multiple ye-block size.
• Pasina hit — Request ine chikamu chekutanga chiri kunze kwe-cache inobhadharwa pa-regular input rate. Hapana nguva inoburitswa yekuchengetwa kwe-prompts dzakachenzwa uye hit haivimbisirwi: verenga usage kuti uone zvakatorwa ne-request kubva ku-cache.
• Hapana switch — Request haipinde, uye hapana field inodzima caching.
• Ndeipi models — Every hosted open-weight id. GET /v1/models inopa reports capabilities.prompt_caching: true ne-pricing.cached_input_per_million_usd kwadzo. Shannon models inobhadharisa rudele rumwe chete.

Ona cache hit mumhinduro

Tumira ma-request maviri anotanga ne-system prompt imwechete refu uye priinta usage ye-imwe neimwe. Nhamba yekutanga i-input ye-request, yechipiri ndicho chikamu chayo chakaverengwa kubva ku-cache.

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.shannon-ai.com/v1")

handbook = open("handbook.txt").read()  # a long text that stays the same


def ask(question):
    response = client.chat.completions.create(
        model="Kimi-K3-3BIT-REAP",
        messages=[
            {"role": "system", "content": handbook},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    print(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens)


ask("What is the refund policy?")
ask("Who approves travel?")  # same start: read the second number

import { readFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "YOUR_API_KEY", baseURL: "https://api.shannon-ai.com/v1" });

const handbook = readFileSync("handbook.txt", "utf8"); // a long text that stays the same

async function ask(question) {
  const response = await client.chat.completions.create({
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      { role: "system", content: handbook },
      { role: "user", content: question },
    ],
  });
  const usage = response.usage;
  console.log(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens);
}

await ask("What is the refund policy?");
await ask("Who approves travel?"); // same start: read the second number

# handbook.txt is a long text that stays the same. jq builds the JSON body from it
# and prints the usage object of the reply. Run it twice with different questions.
jq -Rs '{
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      {role: "system", content: .},
      {role: "user", content: "What is the refund policy?"}
    ]
  }' handbook.txt \
  | curl -s https://api.shannon-ai.com/v1/chat/completions \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d @- \
  | jq .usage

Mutengo

Cached input tokens dzinobhadhariswa ne25% yerudele re-input remudelyo, yakaksetwa pa $0.001 per 1M. Kunyora mu-cache hakuna kudhura zvimwe, uye output inobhadhariswa sezvawaka acostumbrada. Cached rate ye-id imwe neimwe iri mu-Models & pricing table. Ma-model nemitengo

Input ye-call inobhadharwa se (input − cached) × input rate + cached × cached rate. Cached count haipfuuri input count.

Usage log inoratidza cached input ye-call imwe neimwe. Tokens dzayo dzakabhadharwa nemutengo zvatoisa cached rate. Keys & usage

Zvik-usage

{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 812,
    "total_tokens": 20812,
    "prompt_tokens_details": {
      "cached_tokens": 18000
    },
    "completion_tokens_details": {
      "reasoning_tokens": 604
    }
  }
}

{
  "usage": {
    "input_tokens": 20000,
    "input_tokens_details": {
      "cached_tokens": 18000
    },
    "output_tokens": 812,
    "output_tokens_details": {
      "reasoning_tokens": 604
    },
    "total_tokens": 20812
  }
}

{
  "usage": {
    "input_tokens": 2000,
    "cache_read_input_tokens": 18000,
    "cache_creation_input_tokens": 0,
    "output_tokens": 812
  }
}

Mhinduro ye-stream inotakura ma-field akafanana mu-usage yayo yekupedzisira. Hausi kufanira kuikumbira:

Kuwana cache hits dzakawanda

• Chengetedza system prompt ne-tool definitions kuti dzive stable byte-for-byte pakati pe-calls. Isa values dze-per-call senyaya ye-timestamps kana request ids ekupedzisira kwe-message yekupedzisira, kwete mu-system prompt.
• Append chete ku-history. Kuchinja, kudika kana ku-summarise turns dzekare kunochinja prefix, uye zvese zvinotevera kuchinjiwa zvobhadhariswa se-regular input.
• Usachinja order ye-tools, messages kana content blocks pakati pe-calls, uye serialise JSON (tool schemas, tool arguments ne-results) nzira imwe chete nguva dzose.
• Gara pa-model id imwechete muhurukuro, uye utumire call inotevera nekukurumidza mushure meyakapfuura.

API inochengeta kutanga kwehurukuro kusingachinji mune idzi nyaya:

• system kana developer message inotumirwa gare gare muhurukuro inogara payakaiswa. Haichinji kutanga kwe-prompt, saka ma-turn akaiva pamberi payo anoramba akachenzwa.
• Arguments dze-tool calls mu-assistant turns dzekare dzinenzaniswa nevalue. Order ye-keys ne-spacing ye-JSON iyoyo hazvina basa.
• Ma-endpoint matatu anoverenga hurukuro nenzira imwechete. Hurukuro inoenderera pa-endpoint imwe inochengeta shared prefix yayo kana content yakafanana.

Zvikamu zvekukumbira

prompt_cache_key (Chat Completions and Responses) uye cache_control paMessages content blocks zvinogamukanwa, saka existing client code inoshanda pasina kushandura. Hapana chinomanikidzwa: caching yakazvigadzira uye inoshanda zvakafanana pasina izvi.

Kuverenga tokens

Ma-endpoint maviri asingabhadharwi, POST /v1/tokenize ne-POST /v1/messages/count_tokens, anoverenga tokens dze-text kana dze-request yose ye-hosted open-weight models usati watumira. Dzine peji radzo: Kuverenga tokens