Computatio tokenum
Computa tokens textus vel totius petitionis antequam eam mittis.
POST https://api.shannon-ai.com/v1/tokenize
POST https://api.shannon-ai.com/v1/messages/count_tokens
Ambo endpoints cum tokenizer modelli quod nominas computant, et nullum modellum currit. Modellos open-weight hosted comprehendunt. /v1/tokenize textum simplicem vel conversationem Chat Completions accipit. /v1/messages/count_tokens petitionem in forma Anthropic Messages accipit, quae est vocatio quam SDK Anthropic et Claude Code faciunt.
Computatio gratis est. Vocatio clavem API tuam requirit, nihil de saldo tuo trahit et in log usus tui non apparet.
Textum computare
Mitte model et text. Textus ut est computatur, sine formatione chat circa eum.
import requests
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world",
},
)
print(response.json()["tokens"]) const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
text: "Hello, world",
}),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"text": "Hello, world"
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 3
} Numeri in responsionibus huius paginae exempla sunt. Idem textus in alio modello alium numerum dat.
Petitionem chat computare
Mitte model et messages, cum tools si petitio eas habet, prorsus ut ad /v1/chat/completions mitteres. Responsio est magnitudo totius inputi.
import requests
request = {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"},
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"],
},
},
}
],
}
response = requests.post(
"https://api.shannon-ai.com/v1/tokenize",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json=request,
)
print(response.json()["tokens"]) const request = {
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
messages: [
{ role: "system", content: "You are a concise assistant." },
{ role: "user", content: "What is the weather in Paris?" },
],
tools: [
{
type: "function",
function: {
name: "get_weather",
description: "Current weather for a city",
parameters: {
type: "object",
properties: { city: { type: "string" } },
required: ["city"],
},
},
},
],
};
const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
method: "POST",
headers: {
Authorization: "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
body: JSON.stringify(request),
});
const { tokens } = await response.json();
console.log(tokens); curl https://api.shannon-ai.com/v1/tokenize \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"messages": [
{"role": "system", "content": "You are a concise assistant."},
{"role": "user", "content": "What is the weather in Paris?"}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "Current weather for a city",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}
}
]
}' {
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"tokens": 164
} Campi /v1/tokenize
| Campus | Genus | Descriptio |
|---|---|---|
model | string | Necessarium. Id modelli open-weight hosted. Litterae maiusculae et minusculae eodem modo tractantur. |
text | string | Textus ut est computandus, sine formatione chat. Ad 4,000,000 bytes. Mitte text vel messages; cum ambo adsunt, text computatur. |
messages | array | Nuntii chat in forma Chat Completions. Computantur ut input plenum petitionis: quisque nuntius cum formatione quam chat template modelli circa eum ponit. |
tools | array | Definitiones tool in computationem includendae. Una cum messages adhibentur. |
Responsio est objectum JSON cum his campis:
| Campus | Genus | Descriptio |
|---|---|---|
model | string | Id modelli pro quo computatio facta est, in scriptura sua publicata. |
tokens | integer | Cum text: tokens textus. Cum messages: tokens totius inputi, imaginibus inclusis. |
Petitionem Messages computare
Mitte corpus quod ad /v1/messages mitteres: model, messages, et system et tools cum eis uteris. SDK Anthropic officiales hoc endpoint per messages.count_tokens vocant.
import anthropic
client = anthropic.Anthropic(
api_key="YOUR_API_KEY",
base_url="https://api.shannon-ai.com",
)
count = client.messages.count_tokens(
model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system="You are a concise assistant.",
messages=[
{"role": "user", "content": "Summarise the attached report."}
],
)
print(count.input_tokens) import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic({
apiKey: "YOUR_API_KEY",
baseURL: "https://api.shannon-ai.com",
});
const count = await client.messages.countTokens({
model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
system: "You are a concise assistant.",
messages: [
{ role: "user", content: "Summarise the attached report." },
],
});
console.log(count.input_tokens); curl https://api.shannon-ai.com/v1/messages/count_tokens \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
"system": "You are a concise assistant.",
"messages": [
{"role": "user", "content": "Summarise the attached report."}
]
}' {
"input_tokens": 21
} Campi /v1/messages/count_tokens
| Campus | Genus | Descriptio |
|---|---|---|
model | string | Necessarium. Id modelli open-weight hosted. |
messages | array | Necessarium. Nuntii in forma Anthropic Messages. Blocks text, image, tool_use et tool_result computantur. |
system | string | array | System prompt: string aut array text blocks. |
tools | array | Definitiones tool cum name, description et input_schema. |
Propter compatibilitatem accepta, sine effectu in computationem: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Corpus petitionis verae sine mutatione dare potes.
Responsio est objectum JSON cum his campis:
| Campus | Genus | Descriptio |
|---|---|---|
input_tokens | integer | Tokens totius inputi: system prompt, nuntii, tools et imagines. |
Modelli sustenti
Ambo endpoints pro modellis open-weight hosted computant. GET /v1/models /v1/tokenize et /v1/messages/count_tokens in endpoints cuiusque modelli quod ea sustinet enumerat. Quivis alius valor model, ids Shannon inclusis, cum 400 respondetur.
DeepSeek-V4-Pro-0813-3BIT-REAPGLM-5.2-3BIT-REAPKimi-K3-3BIT-REAPNemotron3Ultra-3BIT-REAPMiniMax-M3-3BIT-REAPDeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAPKimi-K2.6-W4A16-AUTOROUND-REAPLaguna-S-2.1-W4A16-AUTOROUND-REAPinkling-W4A16-AUTOROUND-REAPMiMo-V2.5-Pro-W8A16MiMo-V2.5-W8A16Hy3-W8A16
Pro modello Shannon, lege numeros tokenum ex objecto usage responsionis.
Quomodo computatio fiat
Quodque modellum cum tokenizer suo et suo chat template computatur. Nulla aestimatio ex litteris vel verbis adhibetur.
| Quid computetur | Regula |
|---|---|
| Textus | Tokens string ut missi. String vacuus 0 computat. |
| Nuntii | Nuntii et tools cum chat template proprio modelli disponuntur, usque ad punctum ubi responsio incipit, et totum illud prompt computatur. |
| Partes (roles) | Nuntii system, user, assistant et tool computantur. developer ut system computatur. Nuntius sine contento et sine vocatione tool nihil addit. |
| Vocationes tool et eventus | Vocationes tool turnorum assistentis priorum et eventus earum pars computationis sunt, in utroque endpoint. |
| Imagines | Imago intra corpus missa (base64 vel data: URL) unum token per patch 28 × 28 pixelorum addit: ceil(width / 28) × ceil(height / 28). Imago ut URL http(s) data ab his endpoints non capitur et 1,024 computat. |
Exemplum: imago 1,024 × 768 pixelorum computat ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 tokens.
Computatio et quanti petitio constet
Computatio totius petitionis eodem modo fit ac computatio inputi petitionis verae cum eodem modello, nuntiis et tools. Responsio eum numerum nuntiat ut usage.prompt_tokens in Chat Completions, ut usage.input_tokens in Responses, et ut usage.input_tokens plus usage.cache_read_input_tokens in Messages.
- Computatio est input ante deminutionem pretii inputi cached. Petitio vera partem illius inputi e cache legere potest et eam partem ad rate cached computare. Cachatio prompti
- Imago ut URL
http(s)data hic 1,024 computat. Petitio vera imaginem capit et ex magnitudine eius in pixels computat, itaque duo numeri differre possunt. Mitte imaginem ut base64 ut eundem numerum accipias. - Output non est pars computationis. Responsio petitionis verae insuper ut tokens output computatur, ratiocinatione inclusa.
- Computatio
textnullam formationem chat habet. Utere ea ad documentum vel partem prompti metiendam, et formamessagesad petitionem metiendam.
Ut computationem in sumptum vertas, multiplica eam per pretium input modelli per 1M tokenum. Modella et pretia
Limites
| Limes | Valor | Supra eum |
|---|---|---|
Longitudo text | 4,000,000 bytes (UTF-8) | 413 cum nuntio text too long |
| Corpus petitionis | 32 MiB | 413 |
| Per petitionem | Unus textus vel una conversatio | Mitte unam petitionem per textum ut plures textus computes. |
Vocationes computationis ad limitem 120 petitionum per minutum non computantur. Limites et saldum
Errores
| Status | Type | Nuntius | Quando |
|---|---|---|---|
400 | invalid_request_error | tokenize is available for the hosted open models; unknown model: <model> | /v1/tokenize cum model quod non est id open-weight hosted. |
400 | invalid_request_error | count_tokens is available for the hosted open models; unknown model: <model> | /v1/messages/count_tokens cum model quod non est id open-weight hosted, aut sine model. |
400 | invalid_request_error | send `text` or `messages` | /v1/tokenize neque cum text neque cum messages. |
401 | authentication_error | Missing authentication / Invalid API key | Nulla clavis missa est, aut clavis valida non est. |
413 | invalid_request_error | text too long | text longius est quam 4,000,000 bytes. Corpus supra 32 MiB etiam cum 413 respondetur. |
415 | invalid_request_error | Expected request with `Content-Type: application/json` | Petitio content type JSON non habet. |
422 | invalid_request_error | Failed to deserialize the JSON body into the target type: … | Campus necessarius deest (model in /v1/tokenize, messages in /v1/messages/count_tokens) aut campus typum falsum habet. |
503 | api_error | token counting is temporarily unavailable for this model | Computatio pro hoc modello hoc momento fieri non potest. Tenta iterum postea. |
/v1/tokenize errores in forma OpenAI reddit. In /v1/messages/count_tokens errores ipsius endpoint (400 pro modello, 503) in forma Anthropic veniunt, et 401, 413, 415 et 422 in forma OpenAI veniunt. Lege primum codicem status, deinde error.type et error.message, qui in utraque forma adsunt.
{
"error": {
"type": "invalid_request_error",
"message": "tokenize is available for the hosted open models; unknown model: shannon-3"
}
} {
"type": "error",
"error": {
"type": "invalid_request_error",
"message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
}
}