Ukubala ii-token
Bala ii-token zombhalo okanye zesicelo esipheleleyo phambi kokuba usithumele.
POST https://api.shannon-ai.com/v1/tokenize
POST https://api.shannon-ai.com/v1/messages/count_tokens
Zozibini ii-endpoint zibala nge-tokenizer yemodeli oyichazayo, kwaye akukho modeli iqhubayo. Zigubungela iimodeli ze-open-weight ezisingathiweyo. I-/v1/tokenize yamkela umbhalo olula okanye incoko ye-Chat Completions. I-/v1/messages/count_tokens yamkela isicelo kwifomathi ye-Anthropic Messages, nobu bubizo obwenziwa yi-Anthropic SDK kunye ne-Claude Code.
Ukubala kusimahla. Ubizo lufuna i-API key yakho, ayithathi nto kwibhalansi yakho kwaye ayivelanga kwi-log yakho ye-usage.
Bala umbhalo
Thumela i-model kunye ne-text. Umbhalo ubalwa njengoko unjalo, ngaphandle kwefomathi yencoko ejikelezileyo.
import requests
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world",
},
)
print(response.json()["tokens"]) const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
text: "Hello, world",
}),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world"
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 3
} Amanani kwiimpendulo zeli phepha ngemizekelo. Umbhalo ofanayo unika inani elahlukileyo kwimodeli eyahlukileyo.
Bala isicelo sencoko
Thumela i-model kunye ne-messages, kunye ne-tools xa isicelo sinazo, ngokufanayo nanjengoko ungazithumela kwi-/v1/chat/completions. Impendulo bubungakanani begalelo lonke.
import requests
request = {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"},
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}
],
}
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json=request,
)
print(response.json()["tokens"]) const request = {
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
messages: [
{ role: "system", content: "You are a concise assistant." },
{ role: "user", content: "What is the weather in Paris?" },
],
tools: [
{
type: "function",
function: {
name: "get_weather",
description: "Current weather for a city",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
],
};
const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify(request),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}
}
]
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 164
} Iintsimi ze-/v1/tokenize
| Intsimi | Uhlobo | Inkcazelo |
|---|---|---|
model | string | Iyafuneka. I-id yemodeli ye-open-weight esingathiweyo. Oonobumba abakhulu nabancinci bathathwa ngokufanayo. |
text | string | Umbhalo wokubala njengoko unjalo, ngaphandle kwefomathi yencoko. Ukuya kuthi ga kwi-4,000,000 yeebhayithi. Thumela i-text okanye i-messages; xa zozibini zikhona, i-text iyabalwa. |
messages | array | Imiyalezo yencoko kwifomathi ye-Chat Completions. Ibalwa njengegalelo elipheleleyo lesicelo: umyalezo ngamnye kunye nefomathi etemplate yencoko yemodeli iyibeka ngeenxa zonke. |
tools | array | Uchazo lwee-tool oluza kubandakanywa kwinani. Lusetyenziswa kunye ne-messages. |
Impendulo yi-JSON object enezi ntsimi:
| Intsimi | Uhlobo | Inkcazelo |
|---|---|---|
model | string | I-id yemodeli enikwe inani, ngendlela yayo epapashiweyo yokubhala. |
tokens | integer | Kunye ne-text: ii-token zombhalo. Kunye ne-messages: ii-token zegalelo lonke, kubandakanywa nemifanekiso. |
Bala isicelo se-Messages
Thumela umzimba obuya kuwuthumela kwi-/v1/messages: i-model, i-messages, kunye ne-system ne-tools xa uzisebenzisa. Ii-Anthropic SDK ezisemthethweni zibiza le endpoint nge-messages.count_tokens.
import anthropic
client = anthropic.Anthropic(
api_key="YOUR_API_KEY",
base_url="https://api.shannon-ai.com",
)
count = client.messages.count_tokens(
model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system="You are a concise assistant.",
messages=[
{"role": "user", "content": "Summarise the attached report."}
],
)
print(count.input_tokens) import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
apiKey: "YOUR_API_KEY",
baseURL: "https://api.shannon-ai.com",
});
const count = await client.messages.countTokens({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system: "You are a concise assistant.",
messages: [
{ role: "user", content: "Summarise the attached report." },
],
});
console.log(count.input_tokens); curl https://api.shannon-ai.com/v1/messages/count_tokens \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"system": "You are a concise assistant.",
"messages": [
{"role": "user", "content": "Summarise the attached report."}
]
}' {
"input_tokens": 21
} Iintsimi ze-/v1/messages/count_tokens
| Intsimi | Uhlobo | Inkcazelo |
|---|---|---|
model | string | Iyafuneka. I-id yemodeli ye-open-weight esingathiweyo. |
messages | array | Iyafuneka. Imiyalezo kwifomathi ye-Anthropic Messages. Iibhloko ze-text, image, tool_use kunye ne-tool_result ziyabalwa. |
system | string | array | I-system prompt: umtya okanye uluhlu lweebhloko zombhalo. |
tools | array | Uchazo lwee-tool kunye ne-name, i-description kunye ne-input_schema. |
Ziyamkelwa ukuze kuhambelane, azinampembelelo kwinani: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Unokudlulisa umzimba wesicelo sokwenene ngaphandle kotshintsho.
Impendulo yi-JSON object enezi ntsimi:
| Intsimi | Uhlobo | Inkcazelo |
|---|---|---|
input_tokens | integer | Ii-token zegalelo lonke: i-system prompt, imiyalezo, ii-tool kunye nemifanekiso. |
Iimodeli ezixhaswayo
Zozibini ii-endpoint zibala iimodeli ze-open-weight ezisingathiweyo. I-GET /v1/models idwelisa i-/v1/tokenize kunye ne-/v1/messages/count_tokens kwi-endpoints yemodeli nganye ezixhasayo. Naliphi na elinye ixabiso le-model, kubandakanywa nee-id zeShannon, liphendulwa nge-400.
DeepSeek-V4-Pro-0813-3BIT-REAPGLM-5.2-3BIT-REAPKimi-K3-3BIT-REAPNemotron3Ultra-3BIT-REAPMiniMax-M3-3BIT-REAPDeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAPKimi-K2.6-W4A16-AUTOROUND-REAPLaguna-S-2.1-W4A16-AUTOROUND-REAPinkling-W4A16-AUTOROUND-REAPMiMo-V2.5-Pro-W8A16MiMo-V2.5-W8A16Hy3-W8A16
Kwimodeli yeShannon, funda amanani ee-token kwi-object ye-usage yempendulo.
Indlela inani elenziwa ngayo
Imodeli nganye ibalwa nge-tokenizer yayo kunye netemplate yayo yencoko. Akusetyenziswa ngqikelelo esekelwe kwinani lohlobo okanye lamagama.
| Okubalwayo | Umthetho |
|---|---|
| Umbhalo | Ii-token zomtya njengoko uthunyelwe. Umtya ongenanto ubalwa njenge-0. |
| Imiyalezo | Imiyalezo nee-tool zicwangciswa ngetemplate yencoko yemodeli ngokwayo, ukuya kuthi ga kwindawo apho impendulo iqala khona, kwaye lonke olo prompt luyabalwa. |
| Iindima | Imiyalezo ye-system, user, assistant kunye ne-tool iyabalwa. I-developer ibalwa njenge-system. Umyalezo ongenamxholo kwaye ongenabizo lwe-tool awongezi nto. |
| Ubizo lwee-tool neziphumo | Ubizo lwee-tool lwemijikelo yangaphambili yesincedisi kunye neziphumo zazo ziyinxalenye yenani, kuzo zozibini ii-endpoint. |
| Imifanekiso | Umfanekiso othunyelwe ngaphakathi kumzimba (i-base64 okanye i-URL ye-data:) wongeza i-token enye kwiqhekeza ngalinye leepikseli ezingama-28 × 28: ceil(width / 28) × ceil(height / 28). Umfanekiso onikwe njenge-URL ye-http(s) awulandwa zezi endpoint kwaye ubalwa njenge-1,024. |
Umzekelo: umfanekiso oyi-1,024 × 768 yeepikseli ubalwa njenge-ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 yee-token.
Inani kunye nokuhlawulwa kwesicelo
Ukubalwa kwesicelo esipheleleyo kwenziwa ngendlela efanayo nokubalwa kwegalelo lesicelo sokwenene kunye nemodeli, imiyalezo kunye nee-tool ezifanayo. Impendulo ibika eli nani njenge-usage.prompt_tokens kwi-Chat Completions, njenge-usage.input_tokens kwi-Responses, nanjenge-usage.input_tokens kunye ne-usage.cache_read_input_tokens kwi-Messages.
- Inani lilixabiso legalelo phambi kwesaphulelo segalelo le-cache. Isicelo sokwenene sinokufunda inxalenye yegalelo kwi-cache sihlawule loo nxalenye ngexabiso le-cache. I-Prompt caching
- Umfanekiso onikwe njenge-URL ye-
http(s)ubalwa njenge-1,024 apha. Isicelo sokwenene silanda umfanekiso sibale ngobungakanani bawo kwiipikseli, ngoko amanani amabini anokwahluka. Thumela umfanekiso njenge-base64 ukufumana inani elifanayo. - Okuphumayo akuyonxalenye yenani. Impendulo yesicelo sokwenene ihlawulwa njengee-token zokuphuma ngaphezulu, kubandakanywa ukuqiqa.
- Inani le-
textalinalo uhlobo lwencoko. Lisebenzise ukulinganisa uxwebhu okanye inxalenye ye-prompt, kwaye uhlobo lwe-messagesukulinganisa isicelo.
Ukuguqula inani libe yindleko, lilinganise ngexabiso legalelo lemodeli kwii-token ezingu-1M. Iimodeli namaxabiso
Imida
| Umda | Ixabiso | Ngaphezu kwawo |
|---|---|---|
Ubude be-text | 4,000,000 yeebhayithi (UTF-8) | 413 kunye nomyalezo othi text too long |
| Umzimba wesicelo | 32 MiB | 413 |
| Ngesicelo | Umbhalo omnye okanye incoko enye | Thumela isicelo esinye ngombhalo ukubala imibhalo emininzi. |
Iminxeba yokubala ayibalwa kumda wezicelo ezili-120 ngomzuzu. Imida nebhalansi
Iimpazamo
| Isimo | Uhlobo | Umyalezo | Xa |
|---|---|---|---|
400 | invalid_request_error | tokenize is available for the hosted open models; unknown model: <model> | I-/v1/tokenize kunye ne-model engeyiyo i-id ye-open-weight esingathiweyo. |
400 | invalid_request_error | count_tokens is available for the hosted open models; unknown model: <model> | I-/v1/messages/count_tokens kunye ne-model engeyiyo i-id ye-open-weight esingathiweyo, okanye ngaphandle kwe-model. |
400 | invalid_request_error | send `text` or `messages` | I-/v1/tokenize ngaphandle kwe-text okanye i-messages. |
401 | authentication_error | Missing authentication / Invalid API key | Akukho sitshixo sithunyelweyo, okanye isitshixo asisebenzi. |
413 | invalid_request_error | text too long | I-text ide kuneebhayithi ezingu-4,000,000. Umzimba ongaphezu kwe-32 MiB uphendulwa nge-413 nawo. |
415 | invalid_request_error | Expected request with `Content-Type: application/json` | Isicelo asinalo uhlobo lomxholo lwe-JSON. |
422 | invalid_request_error | Failed to deserialize the JSON body into the target type: … | Intsimi efunekayo ayikho (i-model kwi-/v1/tokenize, i-messages kwi-/v1/messages/count_tokens) okanye intsimi inohlobo olungalunganga. |
503 | api_error | token counting is temporarily unavailable for this model | Ukubala akunakwenziwa kule modeli okwangoku. Zama kwakhona kamva. |
I-/v1/tokenize ibuyisa iimpazamo ngesimo se-OpenAI. Kwi-/v1/messages/count_tokens iimpazamo ze-endpoint ngokwayo (i-400 yemodeli, i-503) zifika ngesimo se-Anthropic, kwaye i-401, i-413, i-415 kunye ne-422 zifika ngesimo se-OpenAI. Funda ikhowudi yesimo kuqala, emva koko i-error.type kunye ne-error.message, ezikhoyo kuzo zozibini izimo.
{
"error": {
"type": "invalid_request_error",
"message": "tokenize is available for the hosted open models; unknown model: shannon-3"
}
} {
"type": "error",
"error": {
"type": "invalid_request_error",
"message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
}
}