Kukesha kwa prompt

Inavyofanya kazi

• Prefix, kwa mpangilio — Prompt inasomwa kwa mpangilio: system prompt, tool definitions, kisha ujumbe. Cache inalingana kuanzia mwanzo wa mlolongo huo hadi token ya kwanza inayotofautiana.
• Nini huhesabiwa kama hit — Ombi ambalo prompt yake inaanza na maudhui sawa na ombi la hivi karibuni — kwa kawaida ni mzunguko uliopita wa mazungumzo hayo hayo ambapo ujumbe mpya umeongezwa. Prefix inayolingana ni cached input; kila kitu baada ya hapo ni regular input.
• Usahihi — Cache inahifadhi prompt kwa blocks za tokens 1,568, kwa hiyo prompt fupi kuliko takriban tokens 1,500 haikeshwi. Hesabu ya cached kwenye jibu ni hesabu yako ya input iliyozidishwa na sehemu iliyokeshwa ya prompt, ikipunguzwa kwenda chini. Si lazima iwe kigawe cha ukubwa wa block.
• Bila hit — Ombi ambalo mwanzo wake haumo kwenye cache hutozwa kwa bei ya kawaida ya input. Hakuna muda wa kuishi uliochapishwa kwa prompts zilizokeshwa na hit haihakikishwi: soma usage kuona kile ombi lilichukua kutoka cache.
• Hakuna swichi — Ombi halijiunge kwa hiari, na hakuna field inayozima caching.
• Model zipi — Kila hosted open-weight id. GET /v1/models inaripoti capabilities.prompt_caching: true na pricing.cached_input_per_million_usd kwa ajili yao. Shannon models zinatoza kiwango kimoja cha flat rate.

Angalia cache hit kwenye jibu

Tuma maombi mawili yanayoanza na system prompt ndefu ile ile na uchapishe usage ya kila moja. Namba ya kwanza ni input ya ombi, ya pili ni sehemu yake iliyosomwa kutoka cache.

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.shannon-ai.com/v1")

handbook = open("handbook.txt").read()  # a long text that stays the same


def ask(question):
    response = client.chat.completions.create(
        model="Kimi-K3-3BIT-REAP",
        messages=[
            {"role": "system", "content": handbook},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    print(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens)


ask("What is the refund policy?")
ask("Who approves travel?")  # same start: read the second number

import { readFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "YOUR_API_KEY", baseURL: "https://api.shannon-ai.com/v1" });

const handbook = readFileSync("handbook.txt", "utf8"); // a long text that stays the same

async function ask(question) {
  const response = await client.chat.completions.create({
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      { role: "system", content: handbook },
      { role: "user", content: question },
    ],
  });
  const usage = response.usage;
  console.log(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens);
}

await ask("What is the refund policy?");
await ask("Who approves travel?"); // same start: read the second number

# handbook.txt is a long text that stays the same. jq builds the JSON body from it
# and prints the usage object of the reply. Run it twice with different questions.
jq -Rs '{
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      {role: "system", content: .},
      {role: "user", content: "What is the refund policy?"}
    ]
  }' handbook.txt \
  | curl -s https://api.shannon-ai.com/v1/chat/completions \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d @- \
  | jq .usage

Bei

Cached input tokens zinatozwa 25% ya kiwango cha input cha model, ikizungushwa hadi $0.001 kwa 1M. Kuandika kwenye cache hakugharimu chochote cha ziada, na output inatozwa kama kawaida. Kiwango cha cached cha kila id kiko kwenye jedwali la Models & pricing. Model na bei

Input ya wito hutozwa kama (input − cached) × bei ya input + cached × bei ya cached. Hesabu ya cached haizidi kamwe hesabu ya input.

Usage log inaorodhesha input iliyokeshwa ya kila wito. Tokens zake zilizotozwa na gharama tayari vinajumuisha bei ya cached. Keys na matumizi

Field za matumizi

{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 812,
    "total_tokens": 20812,
    "prompt_tokens_details": {
      "cached_tokens": 18000
    },
    "completion_tokens_details": {
      "reasoning_tokens": 604
    }
  }
}

{
  "usage": {
    "input_tokens": 20000,
    "input_tokens_details": {
      "cached_tokens": 18000
    },
    "output_tokens": 812,
    "output_tokens_details": {
      "reasoning_tokens": 604
    },
    "total_tokens": 20812
  }
}

{
  "usage": {
    "input_tokens": 2000,
    "cache_read_input_tokens": 18000,
    "cache_creation_input_tokens": 0,
    "output_tokens": 812
  }
}

Jibu la stream hubeba fields zile zile kwenye usage yake ya mwisho. Huhitaji kuiomba:

Kupata cache hits zaidi

• Weka system prompt na tool definitions kuwa stable (byte-for-byte) kwenye wito zote. Weka thamani za kila wito kama timestamps au request ids mwishoni mwa ujumbe wa mwisho, si kwenye system prompt.
• Ongeza tu kwenye historia. Kuhariri, kupunguza au kufupisha mzunguko wa awali hubadilisha prefix, na kila kitu baada ya mabadiliko ya kwanza kinatozwa kama regular input.
• Usibadilishe mpangilio wa tools, ujumbe au content blocks kati ya wito, na fanya serialise ya JSON (tool schemas, tool arguments na results) kwa njia sawa kila wakati.
• Kaa kwenye model id moja kwa mazungumzo, na utume wito unaofuata muda mfupi baada ya ule wa kabla yake.

API huweka mwanzo wa mazungumzo thabiti katika visa hivi:

• Ujumbe wa system au developer unaotumwa baadaye kwenye mazungumzo hubaki mahali pake. Haubadilishi mwanzo wa prompt, kwa hiyo zamu zilizo kabla yake zinabaki zimekeshwa.
• Arguments za wito wa tools katika zamu za awali za assistant hulinganishwa kwa thamani. Mpangilio wa keys na nafasi za JSON hiyo hazijalishi.
• Endpoint tatu husoma mazungumzo kwa njia ile ile. Mazungumzo yanayoendelezwa kwenye endpoint nyingine huhifadhi prefix yake ya pamoja wakati maudhui ni yale yale.

Fields za ombi

prompt_cache_key (Chat Completions na Responses) na cache_control kwenye content blocks za Messages zinakubaliwa, hivyo code ya client ya sasa inafanya kazi bila mabadiliko. Vyote viwili si lazima: caching ni ya kiotomatiki na inafanya kazi vivyo hivyo bila hivyo.

Kuhesabu tokens

Endpoint mbili za bure, POST /v1/tokenize na POST /v1/messages/count_tokens, huhesabu tokens za maandishi au za ombi zima kwa model za open-weight zinazopangishwa kabla ya kutuma. Zina ukurasa wake: Kuhesabu tokens