Cachatio prompti

Quomodo operatur

• Prefix, in ordine — Prompt legitur in ordine: system prompt, definitiones tool, deinde nuntii. Cache concordat ab initio sequenti usque ad primum token quem differt.
• Quid computatur ut hit — Request cuius prompt incipit eodem contentu quam request recentior — typice conversatio praecedens cum novis nuntiis additis. Prefix concordans est input cached; omnia post id sunt input regularis.
• Granularitas — Cache promptum in blocis 1,568 tokenorum tenet, itaque promptum brevius quam circiter 1,500 tokens non cachatur. Numerus cached in responso est numerus inputi tui per partem cached prompti multiplicatus, deorsum rotundatus. Non necessario multiplum magnitudinis bloci est.
• Sine hit — Petitio cuius initium in cache non est mercede inputi regulari computatur. Nulla vita promptorum cachatorum publicatur et hit non spondetur: usage lege ut videas quid petitio e cache sumpserit.
• Nullus interruptor — Petitio non eligit, et nullus campus cachationem exstinguit.
• Quae modelli — Omne hosted open-weight id. GET /v1/models nuntiat capabilities.prompt_caching: true et pricing.cached_input_per_million_usd pro eis. Shannon models unum pretium fixum computant.

Cache hit in responso vide

Duas petitiones mitte quae ab eodem system prompt longo incipiunt et usum utriusque imprime. Primus numerus est input petitionis, secundus est pars eius e cache lecta.

from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY", base_url="https://api.shannon-ai.com/v1")

handbook = open("handbook.txt").read()  # a long text that stays the same


def ask(question):
    response = client.chat.completions.create(
        model="Kimi-K3-3BIT-REAP",
        messages=[
            {"role": "system", "content": handbook},
            {"role": "user", "content": question},
        ],
    )
    usage = response.usage
    print(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens)


ask("What is the refund policy?")
ask("Who approves travel?")  # same start: read the second number

import { readFileSync } from "node:fs";
import OpenAI from "openai";

const client = new OpenAI({ apiKey: "YOUR_API_KEY", baseURL: "https://api.shannon-ai.com/v1" });

const handbook = readFileSync("handbook.txt", "utf8"); // a long text that stays the same

async function ask(question) {
  const response = await client.chat.completions.create({
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      { role: "system", content: handbook },
      { role: "user", content: question },
    ],
  });
  const usage = response.usage;
  console.log(usage.prompt_tokens, usage.prompt_tokens_details.cached_tokens);
}

await ask("What is the refund policy?");
await ask("Who approves travel?"); // same start: read the second number

# handbook.txt is a long text that stays the same. jq builds the JSON body from it
# and prints the usage object of the reply. Run it twice with different questions.
jq -Rs '{
    model: "Kimi-K3-3BIT-REAP",
    messages: [
      {role: "system", content: .},
      {role: "user", content: "What is the refund policy?"}
    ]
  }' handbook.txt \
  | curl -s https://api.shannon-ai.com/v1/chat/completions \
      -H "Authorization: Bearer YOUR_API_KEY" \
      -H "Content-Type: application/json" \
      -d @- \
  | jq .usage

Pretia

Cached input tokens computantur ad 25% pretii input modelli, rotundati ad $0.001 per 1M. Scriptura in cache nihil extra costat, et output computatur ut usuale. Pretium cached cuiusque id in tabula Models & pricing est. Modella et pretia

Input vocationis computatur ut (input − cached) × merces inputi + cached × merces cached. Numerus cached numquam maior est quam numerus inputi.

Index usus input cached cuiusque vocationis enumerat. Tokens computati et sumptus eius mercedem cached iam includunt. Claves et usus

Campi usus

{
  "usage": {
    "prompt_tokens": 20000,
    "completion_tokens": 812,
    "total_tokens": 20812,
    "prompt_tokens_details": {
      "cached_tokens": 18000
    },
    "completion_tokens_details": {
      "reasoning_tokens": 604
    }
  }
}

{
  "usage": {
    "input_tokens": 20000,
    "input_tokens_details": {
      "cached_tokens": 18000
    },
    "output_tokens": 812,
    "output_tokens_details": {
      "reasoning_tokens": 604
    },
    "total_tokens": 20812
  }
}

{
  "usage": {
    "input_tokens": 2000,
    "cache_read_input_tokens": 18000,
    "cache_creation_input_tokens": 0,
    "output_tokens": 812
  }
}

Responsum per fluxum missum eosdem campos in usu finali fert. Eum petere non debes:

Plures cache hits obtinendi

• System prompt et definitiones tool byte-for-byte stabiles serva inter vocationes. Valores per-call, ut timestamps vel request ids, pone in fine ultimi nuntii, non in system prompt.
• Tantum ad historiam addere. Editio, trimmatio vel summarium turnum prioris mutat prefix, et omnia post primam mutationem computantur ut input regularis.
• Noli reordinare tool, nuntios vel content blocks inter vocationes, et serialise JSON (tool schemas, tool arguments et results) eodem modo semper.
• In uno id modelli pro colloquio mane, et vocationem sequentem mox post praecedentem mitte.

API initium colloquii his casibus stabile servat:

• Nuntius system vel developer postea in colloquio missus loco suo manet. Initium prompti non mutat, itaque vices ante eum cachatae manent.
• Argumenta vocationum instrumentorum in vicibus assistentis prioribus per valorem comparantur. Ordo clavium et spatia illius JSON non refert.
• Tria endpoints colloquium eodem modo legunt. Colloquium in alio endpoint continuatum prefixum commune retinet cum contentum idem est.

Campi petitionis

prompt_cache_key (Chat Completions et Responses) et cache_control in content blocks nuntiorum acceptantur, ut codex clientis existens immutabilis maneat. Nullum ex his requiritur: caching automaticum est et eodem modo operatur sine his.

Numeratio tokens

Duo endpoints gratuita, POST /v1/tokenize et POST /v1/messages/count_tokens, tokens textus vel totius petitionis pro modellis open-weight hospitatis computant antequam mittis. Paginam propriam habent: Computatio tokenum