Te tatau token
Tatauria ngā token o tētahi kuputuhi, o tētahi tono katoa rānei i mua i te tuku.
POST https://api.shannon-ai.com/v1/tokenize
POST https://api.shannon-ai.com/v1/messages/count_tokens
Ka tatau ngā endpoint e rua ki te tokenizer o te model ka whakaingoatia e koe, ā, kāore he model e rere. Ka kapi i ngā model open-weight hosted. Ka whakaae te /v1/tokenize ki tētahi kuputuhi noa, ki tētahi kōrero Chat Completions rānei. Ka whakaae te /v1/messages/count_tokens ki tētahi tono i te hōputu Anthropic Messages, ko te karanga tēnei ka mahia e te SDK Anthropic me Claude Code.
He kore utu te tatau. E hiahia ana tētahi karanga ki tō API key, kāore e tango i tētahi mea i tō toenga, ā, kāore e puta i tō log whakamahinga.
Tatau i tētahi kuputuhi
Tukuna te model me te text. Ka tatauria te kuputuhi pēnei i a ia, me te kore whakahōputu chat huri noa.
import requests
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world",
},
)
print(response.json()["tokens"]) const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
text: "Hello, world",
}),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world"
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 3
} He tauira ngā tau i ngā whakautu o tēnei whārangi. Ka hoatu te kuputuhi ōrite i tētahi tatau rerekē i runga i tētahi model rerekē.
Tatau i tētahi tono chat
Tukuna te model me te messages, me te tools ina he tā te tono, pēnei tonu i tā koe tuku ki te /v1/chat/completions. Ko te whakautu te rahi o te input katoa.
import requests
request = {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"},
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}
],
}
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json=request,
)
print(response.json()["tokens"]) const request = {
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
messages: [
{ role: "system", content: "You are a concise assistant." },
{ role: "user", content: "What is the weather in Paris?" },
],
tools: [
{
type: "function",
function: {
name: "get_weather",
description: "Current weather for a city",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
],
};
const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify(request),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}
}
]
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 164
} Ngā āpure o te /v1/tokenize
| Āpure | Momo | Whakaahuatanga |
|---|---|---|
model | string | E hiahiatia ana. He id model open-weight hosted. Ka whakaarohia te pū nui me te pū iti hei mea ōrite. |
text | string | He kuputuhi hei tatau pēnei i a ia, me te kore whakahōputu chat. Tae atu ki te 4,000,000 paita. Tukuna te text te messages rānei; ina kei reira ngā mea e rua, ka tatauria te text. |
messages | array | Ngā karere chat i te hōputu Chat Completions. Ka tatauria hei input katoa o tētahi tono: ia karere me te whakahōputu ka whakanōhia e te tauira chat a te model huri noa i a ia. |
tools | array | Ngā whakamārama tool hei uru ki te tatau. Ka whakamahia tahi me te messages. |
He ahanoa JSON te whakautu me ēnei āpure:
| Āpure | Momo | Whakaahuatanga |
|---|---|---|
model | string | Te id model i mahia te tatau mōna, i tōna tuhinga kua whakaputaina. |
tokens | integer | Me te text: ngā token o te kuputuhi. Me te messages: ngā token o te input katoa, tae atu ki ngā whakaahua. |
Tatau i tētahi tono Messages
Tukuna te tinana ka tukuna e koe ki te /v1/messages: model, messages, me te system me te tools ina whakamahi koe i a rāua. Ka karanga ngā SDK Anthropic ōkawa i tēnei endpoint mā te messages.count_tokens.
import anthropic
client = anthropic.Anthropic(
api_key="YOUR_API_KEY",
base_url="https://api.shannon-ai.com",
)
count = client.messages.count_tokens(
model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system="You are a concise assistant.",
messages=[
{"role": "user", "content": "Summarise the attached report."}
],
)
print(count.input_tokens) import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
apiKey: "YOUR_API_KEY",
baseURL: "https://api.shannon-ai.com",
});
const count = await client.messages.countTokens({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system: "You are a concise assistant.",
messages: [
{ role: "user", content: "Summarise the attached report." },
],
});
console.log(count.input_tokens); curl https://api.shannon-ai.com/v1/messages/count_tokens \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"system": "You are a concise assistant.",
"messages": [
{"role": "user", "content": "Summarise the attached report."}
]
}' {
"input_tokens": 21
} Ngā āpure o te /v1/messages/count_tokens
| Āpure | Momo | Whakaahuatanga |
|---|---|---|
model | string | E hiahiatia ana. He id model open-weight hosted. |
messages | array | E hiahiatia ana. Ngā karere i te hōputu Anthropic Messages. Ka tatauria ngā poraka text, image, tool_use me tool_result. |
system | string | array | Te system prompt: he aho, he array poraka kuputuhi rānei. |
tools | array | Ngā whakamārama tool me te name, description me te input_schema. |
E whakaaetia ana mō te hototahi, kāore he pānga ki te tatau: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Ka taea e koe te tuku i te tinana o tētahi tono tūturu kāore i huri.
He ahanoa JSON te whakautu me ēnei āpure:
| Āpure | Momo | Whakaahuatanga |
|---|---|---|
input_tokens | integer | Ngā token o te input katoa: system prompt, ngā karere, ngā tool me ngā whakaahua. |
Ngā model e tautokohia ana
Ka tatau ngā endpoint e rua mō ngā model open-weight hosted. Ka rārangi te GET /v1/models i te /v1/tokenize me te /v1/messages/count_tokens i ngā endpoints o ia model e tautoko ana i a rāua. Ka whakautua tētahi atu uara model, tae atu ki ngā id Shannon, ki te 400.
DeepSeek-V4-Pro-0813-3BIT-REAPGLM-5.2-3BIT-REAPKimi-K3-3BIT-REAPNemotron3Ultra-3BIT-REAPMiniMax-M3-3BIT-REAPDeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAPKimi-K2.6-W4A16-AUTOROUND-REAPLaguna-S-2.1-W4A16-AUTOROUND-REAPinkling-W4A16-AUTOROUND-REAPMiMo-V2.5-Pro-W8A16MiMo-V2.5-W8A16Hy3-W8A16
Mō tētahi model Shannon, pānuitia ngā tatau token mai i te ahanoa usage o tētahi whakautu.
Te āhua o te tatau
Ka tatauria ia model ki tōna ake tokenizer me tōna ake tauira chat. Kāore e whakamahia he whakatau mai i ngā tohu, i ngā kupu rānei.
| Te mea e tatauria ana | Ture |
|---|---|
| He kuputuhi | Ngā token o te aho pēnei i tō tuku. Ka tatau tētahi aho kau hei 0. |
| Ngā karere | Ka whakatakotoria ngā karere me ngā tool ki te tauira chat ake a te model, tae noa ki te wāhi e tīmata ai te whakautu, ā, ka tatauria taua prompt katoa. |
| Ngā tūranga | Ka tatauria ngā karere system, user, assistant me tool. Ka tatauria te developer hei system. Kāore tētahi karere kāore he content kāore he karanga tool e tāpiri i tētahi mea. |
| Ngā karanga tool me ngā hua | He wāhanga o te tatau ngā karanga tool o ngā hurihanga assistant o mua me ō rātou hua, i ngā endpoint e rua. |
| Ngā whakaahua | Ka tāpiri tētahi whakaahua i tukuna i roto i te tinana (base64 he URL data: rānei) i te token kotahi mō ia kōpaki pika 28 × 28: ceil(width / 28) × ceil(height / 28). Kāore ēnei endpoint e tikiake i tētahi whakaahua i hoatu hei URL http(s), ā, ka tatau hei 1,024. |
Tauira: ka tatau tētahi whakaahua 1,024 × 768 pika hei ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 token.
Te tatau me te mea e utua ana e tētahi tono
Ka mahia te tatau o tētahi tono katoa i te ara ōrite ki te tatau input o tētahi tono tūturu me te model, ngā karere me ngā tool ōrite. Ka pūrongo tētahi whakautu i taua tau hei usage.prompt_tokens i Chat Completions, hei usage.input_tokens i Responses, hei usage.input_tokens me te usage.cache_read_input_tokens i Messages.
- Ko te tatau te input i mua i te whakaitinga input-cached. Ka taea e tētahi tono tūturu te pānui i tētahi wāhanga o taua input mai i te cache, ā, ka whakarite i taua wāhanga ki te reiti cached. Tūhono caching
- Ka tatau tētahi whakaahua i hoatu hei URL
http(s)hei 1,024 i konei. Ka tikiake tētahi tono tūturu i te whakaahua, ā, ka tatau mai i tōna rahi pika, nō reira ka taea te rerekē o ngā tau e rua. Tukuna te whakaahua hei base64 kia whiwhi koe i te tau ōrite. - Kāore te output e uru ki te tatau. Ka whakaritea te whakautu o tētahi tono tūturu hei token output i runga ake, tae atu ki te reasoning.
- Kāore he whakahōputu chat o tētahi tatau
text. Whakamahia hei ine i tētahi tuhinga, i tētahi wāhanga prompt rānei, ā, te āhuamessageshei ine i tētahi tono.
Hei huri i tētahi tatau hei utu, whakareatia ki te utu input a te model mō ia 1M token. Ngā model me ngā utu
Ngā tepe
| Tepe | Uara | Kei runga ake |
|---|---|---|
Te roa o te text | 4,000,000 paita (UTF-8) | 413 me te karere text too long |
| Tinana tono | 32 MiB | 413 |
| Mō ia tono | Kotahi kuputuhi, kotahi kōrero rānei | Tukuna tētahi tono mō ia kuputuhi hei tatau i ētahi kuputuhi maha. |
Kāore ngā karanga tatau e tatau ki te tepe o te 120 tono ia meneti. Ngā tepe me te toenga
Ngā hapa
| Tūnga | Momo | Karere | Āhea |
|---|---|---|---|
400 | invalid_request_error | tokenize is available for the hosted open models; unknown model: <model> | Te /v1/tokenize me tētahi model ehara i tētahi id open-weight hosted. |
400 | invalid_request_error | count_tokens is available for the hosted open models; unknown model: <model> | Te /v1/messages/count_tokens me tētahi model ehara i tētahi id open-weight hosted, kāore rānei he model. |
400 | invalid_request_error | send `text` or `messages` | Te /v1/tokenize kāore he text kāore hoki he messages. |
401 | authentication_error | Missing authentication / Invalid API key | Kāore he key i tukuna, ehara rānei te key i te mea tika. |
413 | invalid_request_error | text too long | Roa ake te text i te 4,000,000 paita. Ka whakautua hoki tētahi tinana neke atu i te 32 MiB ki te 413. |
415 | invalid_request_error | Expected request with `Content-Type: application/json` | Kāore he momo content JSON o te tono. |
422 | invalid_request_error | Failed to deserialize the JSON body into the target type: … | Kei te ngaro tētahi āpure e hiahiatia ana (model i te /v1/tokenize, messages i te /v1/messages/count_tokens) he hē rānei te momo o tētahi āpure. |
503 | api_error | token counting is temporarily unavailable for this model | Kāore e taea te tatau mō tēnei model i tēnei wā. Whakamātauria anō a muri ake nei. |
Ka whakahoki te /v1/tokenize i ngā hapa i te āhua OpenAI. I runga i te /v1/messages/count_tokens ka tae mai ngā hapa o te endpoint ake (400 mō te model, 503) i te āhua Anthropic, ā, ka tae mai te 401, 413, 415 me te 422 i te āhua OpenAI. Pānuitia te waehere tūnga i te tuatahi, kātahi te error.type me te error.message, kei ngā āhua e rua.
{
"error": {
"type": "invalid_request_error",
"message": "tokenize is available for the hosted open models; unknown model: shannon-3"
}
} {
"type": "error",
"error": {
"type": "invalid_request_error",
"message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
}
}