Kuverenga tokens
Verenga tokens dze-text kana dze-request yose usati wayitumira.
POST https://api.shannon-ai.com/v1/tokenize
POST https://api.shannon-ai.com/v1/messages/count_tokens
Ma-endpoint ese maviri anoverenga ne-tokenizer ye-model yaunotumidza, uye hapana model inomhanya. Anofukidza hosted open-weight models. /v1/tokenize inotora text chaiyo kana hurukuro ye-Chat Completions. /v1/messages/count_tokens inotora request mu-Anthropic Messages format, ndiyo call inoitwa ne-Anthropic SDK ne-Claude Code.
Kuverenga hakubhadharwi. Call inoda API key yako, haitore chinhu mu-balance yako uye haioneki mu-usage log yako.
Verenga text
Tumira model ne-text. Text inoverengwa sezvairi, pasina chat formatting yakaipoteredza.
import requests
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world",
},
)
print(response.json()["tokens"]) const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
text: "Hello, world",
}),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world"
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 3
} Nhamba dziri mumhinduro dzepeji rino i-mienzaniso. Text imwechete inopa count yakasiyana pa-model yakasiyana.
Verenga chat request
Tumira model ne-messages, pamwe ne-tools kana request ichiine, chaizvo sezvaungatumira ku-/v1/chat/completions. Mhinduro i-size ye-input yose.
import requests
request = {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"},
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}
],
}
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json=request,
)
print(response.json()["tokens"]) const request = {
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
messages: [
{ role: "system", content: "You are a concise assistant." },
{ role: "user", content: "What is the weather in Paris?" },
],
tools: [
{
type: "function",
function: {
name: "get_weather",
description: "Current weather for a city",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
],
};
const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify(request),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}
}
]
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 164
} Fields dze-/v1/tokenize
| Field | Mhando | Tsananguro |
|---|---|---|
model | string | Inodiwa. Hosted open-weight model id. Mavara makuru nemadiki anotorwa zvakafanana. |
text | string | Text yekuverenga sezvairi, pasina chat formatting. Kusvika ku-4,000,000 bytes. Tumira text kana messages; kana dzese dziripo, text ndiyo inoverengwa. |
messages | array | Chat messages mu-Chat Completions format. Dzinoverengwa se-input yakazara ye-request: message imwe neimwe ine formatting inoiswa ne-chat template ye-model yakaipoteredza. |
tools | array | Tool definitions dzekusanganisa mu-count. Dzinoshandiswa pamwe chete ne-messages. |
Mhinduro i-JSON object ine ma-field aya:
| Field | Mhando | Tsananguro |
|---|---|---|
model | string | Model id yakaitirwa count, nemanyorerwo ayo akaburitswa. |
tokens | integer | Ne-text: tokens dze-text. Ne-messages: tokens dze-input yose, mifananidzo yakaverengwa. |
Verenga Messages request
Tumira body yaungatumira ku-/v1/messages: model, messages, ne-system ne-tools kana uchizvishandisa. Official Anthropic SDKs dzinodanidzira endpoint iyi kuburikidza ne-messages.count_tokens.
import anthropic
client = anthropic.Anthropic(
api_key="YOUR_API_KEY",
base_url="https://api.shannon-ai.com",
)
count = client.messages.count_tokens(
model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system="You are a concise assistant.",
messages=[
{"role": "user", "content": "Summarise the attached report."}
],
)
print(count.input_tokens) import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
apiKey: "YOUR_API_KEY",
baseURL: "https://api.shannon-ai.com",
});
const count = await client.messages.countTokens({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system: "You are a concise assistant.",
messages: [
{ role: "user", content: "Summarise the attached report." },
],
});
console.log(count.input_tokens); curl https://api.shannon-ai.com/v1/messages/count_tokens \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"system": "You are a concise assistant.",
"messages": [
{"role": "user", "content": "Summarise the attached report."}
]
}' {
"input_tokens": 21
} Fields dze-/v1/messages/count_tokens
| Field | Mhando | Tsananguro |
|---|---|---|
model | string | Inodiwa. Hosted open-weight model id. |
messages | array | Inodiwa. Messages mu-Anthropic Messages format. text, image, tool_use ne-tool_result blocks dzinoverengwa. |
system | string | array | System prompt: string kana array ye-text blocks. |
tools | array | Tool definitions dzine name, description ne-input_schema. |
Dzinogamuchirwa kuti zvienderane, pasina mhedzisiro pa-count: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Unogona kupfuudza body ye-request chaiyo isina kuchinjwa.
Mhinduro i-JSON object ine ma-field aya:
| Field | Mhando | Tsananguro |
|---|---|---|
input_tokens | integer | Tokens dze-input yose: system prompt, messages, tools nemifananidzo. |
Ma-model anotsigirwa
Ma-endpoint ese maviri anoverenga ye-hosted open-weight models. GET /v1/models inoratidza /v1/tokenize ne-/v1/messages/count_tokens mu-endpoints ye-model imwe neimwe inodzitsigira. Chero model value imwe, Shannon ids dzakaverengwa, inopindurwa ne-400.
DeepSeek-V4-Pro-0813-3BIT-REAPGLM-5.2-3BIT-REAPKimi-K3-3BIT-REAPNemotron3Ultra-3BIT-REAPMiniMax-M3-3BIT-REAPDeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAPKimi-K2.6-W4A16-AUTOROUND-REAPLaguna-S-2.1-W4A16-AUTOROUND-REAPinkling-W4A16-AUTOROUND-REAPMiMo-V2.5-Pro-W8A16MiMo-V2.5-W8A16Hy3-W8A16
Pa-Shannon model, verenga huwandu hwe-tokens kubva ku-usage object yemhinduro.
Kuti count inoitwa sei
Model imwe neimwe inoverengwa ne-tokenizer yayo pachayo ne-chat template yayo pachayo. Hapana estimate inobva pa-characters kana mazwi inoshandiswa.
| Zvinoverengwa | Mutemo |
|---|---|
| Text | Tokens dze-string sezvayakatumirwa. String isina chinhu inoverengwa 0. |
| Messages | Messages ne-tools zvinorongwa ne-chat template ye-model pachayo, kusvika panotanga mhinduro, uye prompt iyoyo yose inoverengwa. |
| Roles | system, user, assistant ne-tool messages dzinoverengwa. developer inoverengwa se-system. Message isina content uye isina tool call haiwedzere chinhu. |
| Tool calls nemhedzisiro | Tool calls dze-assistant turns dzekare nemhedzisiro yadzo zviri chikamu che-count, pa-endpoints dzese mbiri. |
| Mifananidzo | Mufananidzo watumirwa mukati me-body (base64 kana data: URL) unowedzera token imwe pa-28 × 28 pixel patch imwe neimwe: ceil(width / 28) × ceil(height / 28). Mufananidzo wakapiwa se-http(s) URL haudhaunirodhwi ne-endpoints idzi uye unoverengwa 1,024. |
Muenzaniso: mufananidzo we-1,024 × 768 pixel unoverengwa ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 tokens.
Count uye zvinobhadharwa ne-request
Kuverenga kwe-request yose kunoitwa nenzira imwechete ne-input count ye-request chaiyo ine model, messages ne-tools zvakafanana. Mhinduro inoratidza nhamba iyoyo se-usage.prompt_tokens pa-Chat Completions, se-usage.input_tokens pa-Responses, uye se-usage.input_tokens pamwe ne-usage.cache_read_input_tokens pa-Messages.
- Count ndiyo input isati yawana cached-input discount. Request chaiyo inogona kuverenga chikamu che-input iyoyo kubva ku-cache ndokuibhadhara pa-cached rate. Kuchenza kwe-prompt
- Mufananidzo wakapiwa se-
http(s)URL unoverengwa 1,024 pano. Request chaiyo inodhaunirodha mufananidzo ndokuuverenga kubva pa-size yawo mu-pixels, saka nhamba mbiri dzinogona kusiyana. Tumira mufananidzo se-base64 kuti uwane nhamba imwechete. - Output haisi chikamu che-count. Mhinduro ye-request chaiyo inobhadharwa se-output tokens pamusoro, reasoning yakaverengwa.
textcount haina chat formatting. Iishandise kuyera document kana chikamu che-prompt, uyemessagesform kuyera request.
Kushandura count kuita mutengo, ichiwanza ne-input price ye-model pa1M tokens. Ma-model nemitengo
Limits
| Limit | Value | Pamusoro payo |
|---|---|---|
Kureba kwe-text | 4,000,000 bytes (UTF-8) | 413 ine message text too long |
| Request body | 32 MiB | 413 |
| Pa-request | Text imwe kana hurukuro imwe | Tumira request imwe pa-text imwe kuverenga ma-text akawanda. |
Counting calls hadziverengwi mu-limit ye-120 requests paminiti imwe. Limits ne-balance
Errors
| Status | Mhando | Message | Rinhi |
|---|---|---|---|
400 | invalid_request_error | tokenize is available for the hosted open models; unknown model: <model> | /v1/tokenize ine model isiri hosted open-weight id. |
400 | invalid_request_error | count_tokens is available for the hosted open models; unknown model: <model> | /v1/messages/count_tokens ine model isiri hosted open-weight id, kana isina model. |
400 | invalid_request_error | send `text` or `messages` | /v1/tokenize isina text kana messages. |
401 | authentication_error | Missing authentication / Invalid API key | Hapana key yakatumirwa, kana key haina kunaka. |
413 | invalid_request_error | text too long | text yakareba kupfuura 4,000,000 bytes. Body inopfuura 32 MiB inopindurwawo ne-413. |
415 | invalid_request_error | Expected request with `Content-Type: application/json` | Request haina JSON content type. |
422 | invalid_request_error | Failed to deserialize the JSON body into the target type: … | Field inodiwa haisipo (model pa-/v1/tokenize, messages pa-/v1/messages/count_tokens) kana field ine type isiriyo. |
503 | api_error | token counting is temporarily unavailable for this model | Count haigoni kuitwa ye-model iyi panguva ino. Edza zvakare gare gare. |
/v1/tokenize inodzosa errors mu-OpenAI shape. Pa-/v1/messages/count_tokens errors dze-endpoint pachayo (400 ye-model, 503) dzinouya mu-Anthropic shape, uye 401, 413, 415 ne-422 dzinouya mu-OpenAI shape. Verenga status code kutanga, wozoverenga error.type ne-error.message, dziripo mu-shapes dzese mbiri.
{
"error": {
"type": "invalid_request_error",
"message": "tokenize is available for the hosted open models; unknown model: shannon-3"
}
} {
"type": "error",
"error": {
"type": "invalid_request_error",
"message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
}
}