Kuwerenga ma token

Werengani ma token a text kapena a pempho lonse musanalitumize.

POST https://api.shannon-ai.com/v1/tokenize

POST https://api.shannon-ai.com/v1/messages/count_tokens

Ma endpoint onse awiri amawerenga ndi tokenizer ya model yomwe mwatchula, ndipo palibe model yomwe imayenda. Amakhudza hosted open-weight model. /v1/tokenize imalandira text yosavuta kapena zokambirana za Chat Completions. /v1/messages/count_tokens imalandira pempho mu mtundu wa Anthropic Messages, womwe ndi call yomwe Anthropic SDK ndi Claude Code zimapanga.

Kuwerenga ndi kwaulere. Call imafuna API key yanu, siitenga kanthu pa balance yanu ndipo siioneka mu usage log yanu.

Werengani text

Tumizani model ndi text. Text imawerengedwa monga ilili, popanda chat formatting yozungulira.

import requests

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
        "text": "Hello, world",
    },
)
print(response.json()["tokens"])

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    text: "Hello, world",
  }),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "text": "Hello, world"
  }'

200 Yankho

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 3
}

Manambala a mu mayankho a patsamba lino ndi zitsanzo. Text yomweyo imapereka kuwerenga kosiyana pa model ina.

Werengani pempho la chat

Tumizani model ndi messages, ndi tools pamene pempho lili nazo, monga momwe mungatumizire ku /v1/chat/completions. Yankho ndi kukula kwa input yonse.

import requests

request = {
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is the weather in Paris?"},
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
}

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json=request,
)
print(response.json()["tokens"])

const request = {
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  messages: [
    { role: "system", content: "You are a concise assistant." },
    { role: "user", content: "What is the weather in Paris?" },
  ],
  tools: [
    {
      type: "function",
      function: {
        name: "get_weather",
        description: "Current weather for a city",
        parameters: {
          type: "object",
          properties: { city: { type: "string" } },
          required: ["city"],
        },
      },
    },
  ],
};

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify(request),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
          }
        }
      }
    ]
  }'

200 Yankho

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 164
}

Ma field a /v1/tokenize

Field	Mtundu	Kufotokozera
`model`	`string`	Yofunika. Id ya hosted open-weight model. Zilembo zazikulu ndi zazing'ono zimatengedwa mofanana.
`text`	`string`	Text yoyenera kuwerengedwa monga ilili, popanda chat formatting. Mpaka ma byte 4,000,000. Tumizani `text` kapena `messages`; zonse ziwiri zikakhalapo, `text` ndiyo imawerengedwa.
`messages`	`array`	Ma chat message mu mtundu wa Chat Completions. Amawerengedwa ngati input yonse ya pempho: message iliyonse ndi formatting yomwe chat template ya model imayiika mozungulira.
`tools`	`array`	Matanthauzo a ma tool oti aphatikizidwe mu kuwerenga. Amagwiritsidwa ntchito pamodzi ndi `messages`.

Yankho ndi JSON object yokhala ndi ma field awa:

Field	Mtundu	Kufotokozera
`model`	`string`	Model id yomwe kuwerenga kunachitidwira, mu kalembedwe kake kofalitsidwa.
`tokens`	`integer`	Ndi `text`: ma token a text. Ndi `messages`: ma token a input yonse, kuphatikiza zithunzi.

Werengani pempho la Messages

Tumizani body yomwe mungatumize ku /v1/messages: model, messages, ndi system ndi tools mukamazigwiritsa ntchito. Ma Anthropic SDK ovomerezeka amaitana endpoint iyi kudzera mu messages.count_tokens.

import anthropic

client = anthropic.Anthropic(
    api_key="YOUR_API_KEY",
    base_url="https://api.shannon-ai.com",
)

count = client.messages.count_tokens(
    model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    system="You are a concise assistant.",
    messages=[
        {"role": "user", "content": "Summarise the attached report."}
    ],
)
print(count.input_tokens)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: "YOUR_API_KEY",
  baseURL: "https://api.shannon-ai.com",
});

const count = await client.messages.countTokens({
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  system: "You are a concise assistant.",
  messages: [
    { role: "user", content: "Summarise the attached report." },
  ],
});
console.log(count.input_tokens);

curl https://api.shannon-ai.com/v1/messages/count_tokens \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "system": "You are a concise assistant.",
    "messages": [
      {"role": "user", "content": "Summarise the attached report."}
    ]
  }'

200 Yankho

{
  "input_tokens": 21
}

Ma field a /v1/messages/count_tokens

Field	Mtundu	Kufotokozera
`model`	`string`	Yofunika. Id ya hosted open-weight model.
`messages`	`array`	Yofunika. Ma message mu mtundu wa Anthropic Messages. Ma block a `text`, `image`, `tool_use` ndi `tool_result` amawerengedwa.
`system`	`string \| array`	System prompt: string kapena array ya ma text block.
`tools`	`array`	Matanthauzo a ma tool okhala ndi `name`, `description` ndi `input_schema`.

Zimalandiridwa pofuna kugwirizana, popanda kukhudza kuwerenga: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Mutha kupereka body ya pempho lenileni popanda kusintha.

Yankho ndi JSON object yokhala ndi ma field awa:

Field	Mtundu	Kufotokozera
`input_tokens`	`integer`	Ma token a input yonse: system prompt, ma message, ma tool ndi zithunzi.

Ma model othandizidwa

Ma endpoint onse awiri amawerenga pa hosted open-weight model. GET /v1/models imalemba /v1/tokenize ndi /v1/messages/count_tokens mu endpoints ya model iliyonse yomwe imazithandizira. Mtengo wina uliwonse wa model, kuphatikiza ma id a Shannon, umayankhidwa ndi 400.

DeepSeek-V4-Pro-0813-3BIT-REAP
GLM-5.2-3BIT-REAP
Kimi-K3-3BIT-REAP
Nemotron3Ultra-3BIT-REAP
MiniMax-M3-3BIT-REAP
DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP
Kimi-K2.6-W4A16-AUTOROUND-REAP
Laguna-S-2.1-W4A16-AUTOROUND-REAP
inkling-W4A16-AUTOROUND-REAP
MiMo-V2.5-Pro-W8A16
MiMo-V2.5-W8A16
Hy3-W8A16

Pa model ya Shannon, werengani chiwerengero cha ma token kuchokera ku object ya usage ya yankho.

Momwe kuwerenga kumachitikira

Model iliyonse imawerengedwa ndi tokenizer yake ndi chat template yake. Palibe kuyerekezera kochokera ku zilembo kapena mawu komwe kumagwiritsidwa ntchito.

Zomwe zimawerengedwa	Lamulo
Text	Ma token a string monga yatumizidwa. String yopanda kanthu imawerengedwa 0.
Messages	Ma message ndi ma tool amakonzedwa ndi chat template ya model yokha, mpaka pomwe yankho limayambira, ndipo prompt yonseyo imawerengedwa.
Ma role	Ma message a `system`, `user`, `assistant` ndi `tool` amawerengedwa. `developer` imawerengedwa ngati `system`. Message yopanda zomwe zili mkati komanso yopanda call ya tool siiwonjezera kanthu.
Ma call a tool ndi zotsatira	Ma call a tool a ma turn akale a assistant ndi zotsatira zake ndi gawo la kuwerenga, pa ma endpoint onse awiri.
Zithunzi	Chithunzi chotumizidwa mkati mwa body (base64 kapena `data:` URL) chimawonjezera token imodzi pa patch iliyonse ya ma pixel 28 × 28: `ceil(width / 28) × ceil(height / 28)`. Chithunzi choperekedwa ngati `http(s)` URL sichitsitsidwa ndi ma endpoint awa ndipo chimawerengedwa 1,024.

Chitsanzo: chithunzi cha ma pixel 1,024 × 768 chimawerengedwa ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = ma token 1,036.

Kuwerenga ndi zomwe pempho limaperekedwa mtengo

Kuwerenga kwa pempho lonse kumachitika mofanana ndi kuwerenga kwa input ya pempho lenileni lokhala ndi model, ma message ndi ma tool ofanana. Yankho limanena nambala imeneyo ngati usage.prompt_tokens pa Chat Completions, ngati usage.input_tokens pa Responses, komanso ngati usage.input_tokens kuphatikiza usage.cache_read_input_tokens pa Messages.

Kuwerenga ndi input isanachotsedwe kuchotsera kwa input yosungidwa. Pempho lenileni likhoza kuwerenga gawo la input imeneyo kuchokera ku cache ndikuliperekera mtengo wa cached rate. Kusunga prompt
Chithunzi choperekedwa ngati http(s) URL chimawerengedwa 1,024 pano. Pempho lenileni limatsitsa chithunzicho ndikuchiwerenga kuchokera ku kukula kwake mu ma pixel, kotero manambala awiriwo akhoza kusiyana. Tumizani chithunzi ngati base64 kuti mupeze nambala yofanana.
Output siili gawo la kuwerenga. Yankho la pempho lenileni limaperekedwa mtengo ngati ma output token pamwamba pake, kuphatikiza reasoning.
Kuwerenga kwa text kulibe chat formatting. Gwiritsani ntchito kuyeza chikalata kapena gawo la prompt, ndipo mtundu wa messages kuyeza pempho.

Kuti musinthe kuwerenga kukhala mtengo, chulukitsani ndi mtengo wa input wa model pa ma token 1M. Ma model ndi mitengo

Malire

Malire	Mtengo	Pamwamba pake
Kutalika kwa `text`	Ma byte 4,000,000 (UTF-8)	`413` ndi message `text too long`
Body ya pempho	32 MiB	`413`
Pa pempho lililonse	Text imodzi kapena zokambirana chimodzi	Tumizani pempho limodzi pa text iliyonse kuti muwerenge ma text angapo.

Ma call owerengera siawerengedwa mu malire a ma pempho 120 pa mphindi. Malire ndi balance

Zolakwika

Status	Type	Message	Liti
`400`	`invalid_request_error`	`tokenize is available for the hosted open models; unknown model: <model>`	`/v1/tokenize` ndi `model` yomwe si id ya hosted open-weight.
`400`	`invalid_request_error`	`count_tokens is available for the hosted open models; unknown model: <model>`	`/v1/messages/count_tokens` ndi `model` yomwe si id ya hosted open-weight, kapena popanda `model`.
`400`	`invalid_request_error`	send `text` or `messages`	`/v1/tokenize` popanda `text` kapena `messages`.
`401`	`authentication_error`	`Missing authentication / Invalid API key`	Kiyi sinatumizidwe, kapena kiyi siyovomerezeka.
`413`	`invalid_request_error`	`text too long`	`text` ndi yayitali kuposa ma byte 4,000,000. Body yoposa 32 MiB imayankhidwanso ndi `413`.
`415`	`invalid_request_error`	Expected request with `Content-Type: application/json`	Pempho lilibe JSON content type.
`422`	`invalid_request_error`	`Failed to deserialize the JSON body into the target type: …`	Field yofunika ikusowa (`model` pa `/v1/tokenize`, `messages` pa `/v1/messages/count_tokens`) kapena field ili ndi type yolakwika.
`503`	`api_error`	`token counting is temporarily unavailable for this model`	Kuwerenga sikungachitike pa model iyi pakali pano. Yesaninso pambuyo pake.

/v1/tokenize imabweza zolakwika mu mawonekedwe a OpenAI. Pa /v1/messages/count_tokens zolakwika za endpoint yokha (400 ya model, 503) zimabwera mu mawonekedwe a Anthropic, ndipo 401, 413, 415 ndi 422 zimabwera mu mawonekedwe a OpenAI. Werengani status code poyamba, kenako error.type ndi error.message, zomwe zimakhalapo mu mawonekedwe onse awiri.

400 /v1/tokenize

{
  "error": {
    "type": "invalid_request_error",
    "message": "tokenize is available for the hosted open models; unknown model: shannon-3"
  }
}

400 /v1/messages/count_tokens

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
  }
}