Tokenlarni sanash
Matn yoki butun so'rovning tokenlarini yuborishdan oldin sanang.
POST https://api.shannon-ai.com/v1/tokenize
POST https://api.shannon-ai.com/v1/messages/count_tokens
Ikkala endpoint ham siz ko'rsatgan modelning tokenizatori bilan sanaydi, hech qanday model ishga tushmaydi. Ular xostingdagi open-weight modellarni qamrab oladi. /v1/tokenize oddiy matnni yoki Chat Completions suhbatini qabul qiladi. /v1/messages/count_tokens Anthropic Messages formatidagi so'rovni qabul qiladi, Anthropic SDK va Claude Code aynan shu chaqiruvni qiladi.
Sanash bepul. Chaqiruv API kalitingizni talab qiladi, balansingizdan hech narsa olmaydi va usage jurnalida ko'rinmaydi.
Matnni sanash
model va text yuboring. Matn atrofida chat formatlashsiz, qanday bo'lsa shundayligicha sanaladi.
import requests
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world",
},
)
print(response.json()["tokens"]) const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
text: "Hello, world",
}),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world"
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 3
} Bu sahifadagi javoblardagi sonlar misollar. Xuddi shu matn boshqa modelda boshqa son beradi.
Chat so'rovini sanash
model va messages ni, so'rovda bo'lsa tools bilan birga, /v1/chat/completions ga yuborganingizdek yuboring. Javob butun kirishning hajmi bo'ladi.
import requests
request = {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"},
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}
],
}
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json=request,
)
print(response.json()["tokens"]) const request = {
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
messages: [
{ role: "system", content: "You are a concise assistant." },
{ role: "user", content: "What is the weather in Paris?" },
],
tools: [
{
type: "function",
function: {
name: "get_weather",
description: "Current weather for a city",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
],
};
const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify(request),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}
}
]
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 164
} /v1/tokenize maydonlari
| Maydon | Tur | Tavsif |
|---|---|---|
model | string | Majburiy. Xostingdagi open-weight model id'si. Bosh va kichik harflar bir xil hisoblanadi. |
text | string | Chat formatlashsiz, qanday bo'lsa shundayligicha sanaladigan matn. 4,000,000 baytgacha. text yoki messages yuboring; ikkalasi ham bo'lsa, text sanaladi. |
messages | array | Chat Completions formatidagi chat xabarlari. Ular so'rovning to'liq kirishi sifatida sanaladi: har bir xabar modelning chat shabloni uni o'rab turadigan formatlash bilan birga. |
tools | array | Sanoqqa kiritiladigan tool ta'riflari. messages bilan birga ishlatiladi. |
Javob quyidagi maydonlarga ega JSON obyekti:
| Maydon | Tur | Tavsif |
|---|---|---|
model | string | Sanoq qaysi model uchun qilingan bo'lsa, shu model id'si, e'lon qilingan yozilishida. |
tokens | integer | text bilan: matnning tokenlari. messages bilan: rasmlar bilan birga butun kirishning tokenlari. |
Messages so'rovini sanash
/v1/messagesga yuboradigan tanangizni yuboring: model, messages va ishlatsangiz system hamda tools. Rasmiy Anthropic SDK'lar bu endpointni messages.count_tokens orqali chaqiradi.
import anthropic
client = anthropic.Anthropic(
api_key="YOUR_API_KEY",
base_url="https://api.shannon-ai.com",
)
count = client.messages.count_tokens(
model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system="You are a concise assistant.",
messages=[
{"role": "user", "content": "Summarise the attached report."}
],
)
print(count.input_tokens) import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
apiKey: "YOUR_API_KEY",
baseURL: "https://api.shannon-ai.com",
});
const count = await client.messages.countTokens({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system: "You are a concise assistant.",
messages: [
{ role: "user", content: "Summarise the attached report." },
],
});
console.log(count.input_tokens); curl https://api.shannon-ai.com/v1/messages/count_tokens \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"system": "You are a concise assistant.",
"messages": [
{"role": "user", "content": "Summarise the attached report."}
]
}' {
"input_tokens": 21
} /v1/messages/count_tokens maydonlari
| Maydon | Tur | Tavsif |
|---|---|---|
model | string | Majburiy. Xostingdagi open-weight model id'si. |
messages | array | Majburiy. Anthropic Messages formatidagi xabarlar. text, image, tool_use va tool_result bloklari sanaladi. |
system | string | array | System prompt: satr yoki matn bloklari massivi. |
tools | array | name, description va input_schemali tool ta'riflari. |
Moslik uchun qabul qilinadi, sanashga ta'sir qilmaydi: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Haqiqiy so'rov tanasini o'zgartirmay berishingiz mumkin.
Javob quyidagi maydonlarga ega JSON obyekti:
| Maydon | Tur | Tavsif |
|---|---|---|
input_tokens | integer | Butun kirish tokenlari: system prompt, xabarlar, tool'lar va rasmlar. |
Qo'llab-quvvatlanadigan modellar
Ikkala endpoint ham xostingdagi open-weight modellar uchun sanaydi. GET /v1/models /v1/tokenize va /v1/messages/count_tokens ni ularni qo'llab-quvvatlaydigan har bir modelning endpoints maydonida ko'rsatadi. Boshqa har qanday model qiymatiga, jumladan Shannon id'lariga, 400 bilan javob beriladi.
DeepSeek-V4-Pro-0813-3BIT-REAPGLM-5.2-3BIT-REAPKimi-K3-3BIT-REAPNemotron3Ultra-3BIT-REAPMiniMax-M3-3BIT-REAPDeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAPKimi-K2.6-W4A16-AUTOROUND-REAPLaguna-S-2.1-W4A16-AUTOROUND-REAPinkling-W4A16-AUTOROUND-REAPMiMo-V2.5-Pro-W8A16MiMo-V2.5-W8A16Hy3-W8A16
Shannon modeli uchun token sonlarini javobdagi usage obyektidan o'qing.
Sanash qanday bajariladi
Har bir model o'zining tokenizatori va o'zining chat shabloni bilan sanaladi. Belgilar yoki so'zlar bo'yicha taxmin ishlatilmaydi.
| Nima sanaladi | Qoida |
|---|---|
| Matn | Satrning yuborilgan holatdagi tokenlari. Bo'sh satr 0 deb sanaladi. |
| Xabarlar | Xabarlar va tool'lar modelning o'z chat shabloni bilan, javob boshlanadigan joygacha joylashtiriladi va shu butun prompt sanaladi. |
| Rollar | system, user, assistant va tool xabarlari sanaladi. developer system kabi sanaladi. Kontenti ham, tool chaqiruvi ham bo'lmagan xabar hech narsa qo'shmaydi. |
| Tool chaqiruvlari va natijalari | Oldingi assistant bosqichlarining tool chaqiruvlari va ularning natijalari ikkala endpointda ham sanoqqa kiradi. |
| Rasmlar | Tana ichida yuborilgan rasm (base64 yoki data: URL) har bir 28 × 28 piksellik bo'lak uchun bitta token qo'shadi: ceil(width / 28) × ceil(height / 28). http(s) URL sifatida berilgan rasmni bu endpointlar yuklab olmaydi va u 1,024 deb sanaladi. |
Misol: 1,024 × 768 piksellik rasm ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 token sanaladi.
Son va so'rov nima uchun hisoblanadi
Butun so'rovni sanash xuddi shu model, xabarlar va tool'lar bilan haqiqiy so'rovning kirish sonidek bajariladi. Javob bu sonni Chat Completions'da usage.prompt_tokens, Responses'da usage.input_tokens, Messages'da esa usage.input_tokens plyus usage.cache_read_input_tokens sifatida ko'rsatadi.
- Son keshdagi kirish chegirmasidan oldingi kirish. Haqiqiy so'rov shu kirishning bir qismini keshdan o'qishi va o'sha qismni keshdagi narxda hisoblashi mumkin. Prompt keshlash
http(s)URL sifatida berilgan rasm bu yerda 1,024 hisoblanadi. Haqiqiy so'rov rasmni yuklab oladi va uni piksellardagi o'lchamidan sanaydi, shuning uchun ikki son farq qilishi mumkin. Bir xil son olish uchun rasmni base64 sifatida yuboring.- Chiqish sanashga kirmaydi. Haqiqiy so'rov javobi qo'shimcha ravishda chiqish tokenlari sifatida, mulohaza bilan birga hisoblanadi.
textsanash chat formatlashiga ega emas. Undan hujjat yoki prompt qismini o'lchash uchun, so'rovni o'lchash uchun esamessagesshaklidan foydalaning.
Sonni narxga aylantirish uchun uni modelning 1M token uchun kirish narxiga ko'paytiring. Modellar va narxlar
Limitlar
| Limit | Qiymat | Undan oshganda |
|---|---|---|
text uzunligi | 4,000,000 bayt (UTF-8) | 413 va text too long xabari |
| So'rov tanasi | 32 MiB | 413 |
| Har bir so'rovda | Bitta matn yoki bitta suhbat | Bir nechta matnni sanash uchun har bir matn uchun alohida so'rov yuboring. |
Sanash chaqiruvlari daqiqasiga 120 ta so'rov limitiga hisoblanmaydi. Limitlar va balans
Xatolar
| Holat | Tur | Xabar | Qachon |
|---|---|---|---|
400 | invalid_request_error | tokenize is available for the hosted open models; unknown model: <model> | Xostingdagi open-weight id bo'lmagan model bilan /v1/tokenize. |
400 | invalid_request_error | count_tokens is available for the hosted open models; unknown model: <model> | Xostingdagi open-weight id bo'lmagan model bilan yoki modelsiz /v1/messages/count_tokens. |
400 | invalid_request_error | send `text` or `messages` | text ham, messages ham bo'lmagan /v1/tokenize. |
401 | authentication_error | Missing authentication / Invalid API key | Kalit yuborilmagan yoki kalit yaroqsiz. |
413 | invalid_request_error | text too long | text 4,000,000 baytdan uzun. 32 MiB dan katta tanaga ham 413 bilan javob beriladi. |
415 | invalid_request_error | Expected request with `Content-Type: application/json` | So'rovda JSON kontent turi yo'q. |
422 | invalid_request_error | Failed to deserialize the JSON body into the target type: … | Majburiy maydon yo'q (/v1/tokenize da model, /v1/messages/count_tokens da messages) yoki maydon noto'g'ri turda. |
503 | api_error | token counting is temporarily unavailable for this model | Sanashni bu model uchun hozir bajarib bo'lmaydi. Keyinroq qayta urinib ko'ring. |
/v1/tokenize xatolarni OpenAI shaklida qaytaradi. /v1/messages/count_tokens da endpointning o'z xatolari (model uchun 400, 503) Anthropic shaklida, 401, 413, 415 va 422 esa OpenAI shaklida keladi. Avval holat kodini, keyin ikkala shaklda ham bor error.type va error.messageni o'qing.
{
"error": {
"type": "invalid_request_error",
"message": "tokenize is available for the hosted open models; unknown model: shannon-3"
}
} {
"type": "error",
"error": {
"type": "invalid_request_error",
"message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
}
}