Tirinta token-yada

Tiri token-yada qoraal ama codsi dhan ka hor inta aadan dirin.

POST https://api.shannon-ai.com/v1/tokenize

POST https://api.shannon-ai.com/v1/messages/count_tokens

Labada endpoint waxay ku tiriyaan tokenizer-ka model-ka aad magacawdo, model-na ma orodo. Waxay daboolaan model-yada open-weight ee la martigeliyo. /v1/tokenize wuxuu qaataa qoraal cad ama wada-hadal Chat Completions. /v1/messages/count_tokens wuxuu qaataa codsi qaabka Anthropic Messages, kaas oo ah wicitaanka Anthropic SDK iyo Claude Code sameeyaan.

Tirintu waa bilaash. Wicitaanku wuxuu u baahan yahay furahaaga API, haraaggaaga waxba kama qaado kumana muuqdo diiwaanka isticmaalkaaga.

Tiri qoraal

Dir model iyo text. Qoraalka waxaa la tiriyaa sida uu yahay, qaabayn chat la'aan.

import requests

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
        "text": "Hello, world",
    },
)
print(response.json()["tokens"])

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    text: "Hello, world",
  }),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "text": "Hello, world"
  }'

200 Jawaab

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 3
}

Tirooyinka jawaabaha bogga waa tusaalayaal. Isla qoraalka wuxuu siinayaa tirin kale model kale.

Tiri codsi chat

Dir model iyo messages, iyo tools marka codsigu leeyahay, sida aad u diri lahayd /v1/chat/completions. Jawaabtu waa cabbirka input-ka oo dhan.

import requests

request = {
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is the weather in Paris?"},
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
}

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json=request,
)
print(response.json()["tokens"])

const request = {
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  messages: [
    { role: "system", content: "You are a concise assistant." },
    { role: "user", content: "What is the weather in Paris?" },
  ],
  tools: [
    {
      type: "function",
      function: {
        name: "get_weather",
        description: "Current weather for a city",
        parameters: {
          type: "object",
          properties: { city: { type: "string" } },
          required: ["city"],
        },
      },
    },
  ],
};

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify(request),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
          }
        }
      }
    ]
  }'

200 Jawaab

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 164
}

Field-yada /v1/tokenize

Field	Nooc	Sharaxaad
`model`	`string`	Loo baahan yahay. Id model open-weight oo la martigeliyay. Xarfaha waaweyn iyo yaryar si isku mid ah ayaa loo maareeyaa.
`text`	`string`	Qoraal la tiriyo sida uu yahay, qaabayn chat la'aan. Ilaa 4,000,000 byte. Dir `text` ama `messages`; marka labaduba jiraan, `text` ayaa la tiriyaa.
`messages`	`array`	Fariimo chat qaabka Chat Completions. Waxaa la tiriyaa sida input-ka oo dhan ee codsi: fariin kasta iyo qaabaynta chat template-ka model-ku agteeda geliyo.
`tools`	`array`	Qeexidaha tool ee tirinta lagu darayo. Waxaa lagu isticmaalaa `messages` la socda.

Jawaabtu waa shay JSON ah oo leh field-yadan:

Field	Nooc	Sharaxaad
`model`	`string`	Id-ga model-ka tirintu loo sameeyay, higgaadiisa la daabacay.
`tokens`	`integer`	`text` la socda: token-yada qoraalka. `messages` la socda: token-yada input-ka oo dhan, sawirro ku jira.

Tiri codsi Messages

Dir jidhka aad u diri lahayd /v1/messages: model, messages, iyo system iyo tools marka aad isticmaasho. SDK-yada Anthropic ee rasmiga ah endpoint-kan waxay ku wacaan messages.count_tokens.

import anthropic

client = anthropic.Anthropic(
    api_key="YOUR_API_KEY",
    base_url="https://api.shannon-ai.com",
)

count = client.messages.count_tokens(
    model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    system="You are a concise assistant.",
    messages=[
        {"role": "user", "content": "Summarise the attached report."}
    ],
)
print(count.input_tokens)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: "YOUR_API_KEY",
  baseURL: "https://api.shannon-ai.com",
});

const count = await client.messages.countTokens({
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  system: "You are a concise assistant.",
  messages: [
    { role: "user", content: "Summarise the attached report." },
  ],
});
console.log(count.input_tokens);

curl https://api.shannon-ai.com/v1/messages/count_tokens \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "system": "You are a concise assistant.",
    "messages": [
      {"role": "user", "content": "Summarise the attached report."}
    ]
  }'

200 Jawaab

{
  "input_tokens": 21
}

Field-yada /v1/messages/count_tokens

Field	Nooc	Sharaxaad
`model`	`string`	Loo baahan yahay. Id model open-weight oo la martigeliyay.
`messages`	`array`	Loo baahan yahay. Fariimaha qaabka Anthropic Messages. Blocks-ka `text`, `image`, `tool_use` iyo `tool_result` waa la tiriyaa.
`system`	`string \| array`	System prompt-ka: string ama array ah blocks qoraal ah.
`tools`	`array`	Qeexidaha tool oo leh `name`, `description` iyo `input_schema`.

La aqbalay u waafaqsanaan, saamayn la'aan tirinta: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Waxaad gudbin kartaa jidhka codsi runta ah isbeddel la'aan.

Jawaabtu waa shay JSON ah oo leh field-yadan:

Field	Nooc	Sharaxaad
`input_tokens`	`integer`	Token-yada input-ka oo dhan: system prompt, fariimo, tools iyo sawirro.

Model-yada la taageero

Labada endpoint waxay tiriyaan model-yada open-weight ee la martigeliyo. GET /v1/models wuxuu liis garaynayaa /v1/tokenize iyo /v1/messages/count_tokens endpoints ee model kasta oo taageera. Qiime model kale kasta, id-yada Shannon ku jira, waxaa lagu jawaabaa 400.

DeepSeek-V4-Pro-0813-3BIT-REAP
GLM-5.2-3BIT-REAP
Kimi-K3-3BIT-REAP
Nemotron3Ultra-3BIT-REAP
MiniMax-M3-3BIT-REAP
DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP
Kimi-K2.6-W4A16-AUTOROUND-REAP
Laguna-S-2.1-W4A16-AUTOROUND-REAP
inkling-W4A16-AUTOROUND-REAP
MiMo-V2.5-Pro-W8A16
MiMo-V2.5-W8A16
Hy3-W8A16

Model Shannon, tirooyinka token-yada ka akhri shayga usage ee jawaab.

Sida tirinta loo sameeyo

Model kasta waxaa lagu tiriyaa tokenizer-kiisa iyo chat template-kiisa. Qiyaas xaraf ama eray ku salaysan ma jirto.

Waxa la tiriyo	Xeer
Qoraal	Token-yada string-ka sida loo diray. String madhan wuxuu tirsan yahay 0.
Fariimo	Fariimaha iyo tools-ka waxaa lagu habeeyaa chat template-ka model-ka laftiisa, ilaa meesha jawaabtu ka bilaabanto, prompt-kaas oo dhanna waa la tiriyaa.
Doorarka	Fariimaha `system`, `user`, `assistant` iyo `tool` waa la tiriyaa. `developer` waxaa loo tiriyaa sida `system`. Fariin aan lahayn nuxur iyo wicitaan tool midna wax ma ku darto.
Wicitaanno iyo natiijooyin tool	Wicitaannada tool ee wareegyada assistant ee hore iyo natiijooyinkooda waxay qayb ka yihiin tirinta, labada endpoint.
Sawirro	Sawir jidhka gudihiisa loo diray (base64 ama `data:` URL) wuxuu ku daraa hal token patch kasta oo 28 × 28 pixel ah: `ceil(width / 28) × ceil(height / 28)`. Sawir loo bixiyay sida `http(s)` URL endpoint-yadani ma soo dejiyaan wuxuuna tirsan yahay 1,024.

Tusaale: sawir 1,024 × 768 pixel ah wuxuu tirsan yahay ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 token.

Tirinta iyo waxa codsi lagu dallaco

Tirinta codsi dhan waxaa loo sameeyaa si isku mid ah tirada input-ka codsi runta ah oo leh isla model, fariimo iyo tools. Jawaab waxay lambarkaas sheegtaa sida usage.prompt_tokens Chat Completions, sida usage.input_tokens Responses, iyo sida usage.input_tokens oo lagu daray usage.cache_read_input_tokens Messages.

Tirintu waa input-ka ka hor dhimista input-ka cached. Codsi runta ah wuxuu qayb ka mid ah input-kaas ka akhrin karaa cache oo qaybtaas ku dallaci karaa qiimaha cached. Kaydinta Prompt-ka
Sawir loo bixiyay sida http(s) URL halkan wuxuu tirsan yahay 1,024. Codsi runta ah sawirka wuu soo dejiyaa wuxuuna ka tiriyaa cabbirkiisa pixel, sidaas darteed labada lambar way kala duwanaan karaan. Sawirka u dir sida base64 si aad u hesho isla lambarka.
Output tirinta kuma jiro. Jawaabta codsi runta ah waxaa loogu dallacaa sida output token-yo kor ah, reasoning ku jira.
Tirinta text ma laha qaabayn chat. Ku isticmaal cabbirka dukumenti ama qayb prompt ah, qaabka messages-na cabbirka codsi.

Si aad tirin ugu beddesho kharash, ku dhufo qiimaha input-ka ee model-ka halkii 1M token. Model-yo & qiimo

Xadad

Xad	Qiime	Ka sarreeya
Dhererka `text`	4,000,000 byte (UTF-8)	`413` oo leh fariinta `text too long`
Jidhka codsiga	32 MiB	`413`
Codsi kasta	Hal qoraal ama hal wada-hadal	Si aad u tirisid dhowr qoraal, hal codsi qoraal kasta dir.

Wicitaannada tirintu kuma jiraan xadka 120 codsi daqiiqad kasta. Xadad iyo haraag

Khaladaad

Xaalad	Nooc	Fariin	Goorta
`400`	`invalid_request_error`	`tokenize is available for the hosted open models; unknown model: <model>`	`/v1/tokenize` oo leh `model` aan ahayn id open-weight oo la martigeliyay.
`400`	`invalid_request_error`	`count_tokens is available for the hosted open models; unknown model: <model>`	`/v1/messages/count_tokens` oo leh `model` aan ahayn id open-weight oo la martigeliyay, ama `model` la'aan.
`400`	`invalid_request_error`	send `text` or `messages`	`/v1/tokenize` oo aan lahayn `text` iyo `messages` midna.
`401`	`authentication_error`	`Missing authentication / Invalid API key`	Fure lama dirin, ama furuhu sax ma aha.
`413`	`invalid_request_error`	`text too long`	`text` wuu ka dheer yahay 4,000,000 byte. Jidh ka weyn 32 MiB sidoo kale waxaa lagu jawaabaa `413`.
`415`	`invalid_request_error`	Expected request with `Content-Type: application/json`	Codsigu nooc nuxur JSON ah ma laha.
`422`	`invalid_request_error`	`Failed to deserialize the JSON body into the target type: …`	Field loo baahan yahay waa maqan yahay (`model` `/v1/tokenize`, `messages` `/v1/messages/count_tokens`) ama field wuxuu leeyahay nooc khaldan.
`503`	`api_error`	`token counting is temporarily unavailable for this model`	Tirinta lagama samayn karo model-kan hadda. Dambe isku day.

/v1/tokenize wuxuu khaladaadka ku soo celiyaa qaabka OpenAI. /v1/messages/count_tokens khaladaadka endpoint-ka laftiisa (400 model, 503) waxay u yimaadaan qaabka Anthropic, 401, 413, 415 iyo 422 -na waxay u yimaadaan qaabka OpenAI. Marka hore akhri koodhka xaaladda, kaddib error.type iyo error.message, kuwaas oo ku jira labada qaab.

400 /v1/tokenize

{
  "error": {
    "type": "invalid_request_error",
    "message": "tokenize is available for the hosted open models; unknown model: shannon-3"
  }
}

400 /v1/messages/count_tokens

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
  }
}