Ịgụ tokens

Gụọ tokens nke text ma ọ bụ nke request dum tupu ị zipụ ya.

POST https://api.shannon-ai.com/v1/tokenize

POST https://api.shannon-ai.com/v1/messages/count_tokens

Endpoints abụọ ahụ na-agụ site na tokenizer nke model i kpọrọ aha, ọ dịghịkwa model na-agba. Ha na-ekpuchi hosted open-weight models. /v1/tokenize na-anara plain text ma ọ bụ Chat Completions conversation. /v1/messages/count_tokens na-anara request n'Anthropic Messages format, nke bụ call Anthropic SDK na Claude Code na-eme.

Ịgụ bụ n'efu. Call chọrọ API key gị, anaghị ewepụ ihe ọ bụla na balance gị ma anaghị apụta na usage log gị.

Gụọ text

Zipụ model na text. A na-agụ text dị ka ọ dị, na-enweghị chat formatting gburugburu ya.

import requests

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
        "text": "Hello, world",
    },
)
print(response.json()["tokens"])

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    text: "Hello, world",
  }),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "text": "Hello, world"
  }'

200 Azịza

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 3
}

Ọnụọgụ dị n'azịza na peeji a bụ ihe atụ. Otu ederede na-enye ọnụọgụ dị iche na model dị iche.

Gụọ chat request

Zipụ model na messages, ya na tools mgbe request nwere ha, kpọmkwem dị ka ị ga-ezipu ha na /v1/chat/completions. Azịza bụ nha input dum.

import requests

request = {
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is the weather in Paris?"},
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
}

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json=request,
)
print(response.json()["tokens"])

const request = {
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  messages: [
    { role: "system", content: "You are a concise assistant." },
    { role: "user", content: "What is the weather in Paris?" },
  ],
  tools: [
    {
      type: "function",
      function: {
        name: "get_weather",
        description: "Current weather for a city",
        parameters: {
          type: "object",
          properties: { city: { type: "string" } },
          required: ["city"],
        },
      },
    },
  ],
};

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify(request),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
          }
        }
      }
    ]
  }'

200 Azịza

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 164
}

Fields nke /v1/tokenize

Field	Ụdị	Nkọwa
`model`	`string`	Achọrọ. Hosted open-weight model id. A na-ewere mkpụrụedemede nta na nnukwu dị ka otu.
`text`	`string`	Text a ga-agụ dị ka ọ dị, na-enweghị chat formatting. Ruo bytes 4,000,000. Zipụ `text` ma ọ bụ `messages`; mgbe ha abụọ dị, a na-agụ `text`.
`messages`	`array`	Chat messages n'Chat Completions format. A na-agụ ha dị ka input zuru ezu nke request: message ọ bụla na formatting chat template nke model na-etinye gburugburu ya.
`tools`	`array`	Tool definitions a ga-etinye n'ọnụọgụ. A na-eji ya na `messages`.

Azịza bụ JSON object nwere fields ndị a:

Field	Ụdị	Nkọwa
`model`	`string`	Model id e mere ọnụọgụ maka ya, n'ụdị ederede e bipụtara.
`tokens`	`integer`	Na `text`: tokens nke text. Na `messages`: tokens nke input dum, gụnyere images.

Gụọ Messages request

Zipụ body ị ga-ezipu na /v1/messages: model, messages, na system na tools mgbe ị na-eji ha. Anthropic SDKs gọọmentị na-akpọ endpoint a site na messages.count_tokens.

import anthropic

client = anthropic.Anthropic(
    api_key="YOUR_API_KEY",
    base_url="https://api.shannon-ai.com",
)

count = client.messages.count_tokens(
    model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    system="You are a concise assistant.",
    messages=[
        {"role": "user", "content": "Summarise the attached report."}
    ],
)
print(count.input_tokens)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: "YOUR_API_KEY",
  baseURL: "https://api.shannon-ai.com",
});

const count = await client.messages.countTokens({
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  system: "You are a concise assistant.",
  messages: [
    { role: "user", content: "Summarise the attached report." },
  ],
});
console.log(count.input_tokens);

curl https://api.shannon-ai.com/v1/messages/count_tokens \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "system": "You are a concise assistant.",
    "messages": [
      {"role": "user", "content": "Summarise the attached report."}
    ]
  }'

200 Azịza

{
  "input_tokens": 21
}

Fields nke /v1/messages/count_tokens

Field	Ụdị	Nkọwa
`model`	`string`	Achọrọ. Hosted open-weight model id.
`messages`	`array`	Achọrọ. Messages n'Anthropic Messages format. A na-agụ blocks `text`, `image`, `tool_use` na `tool_result`.
`system`	`string \| array`	System prompt: string ma ọ bụ array nke text blocks.
`tools`	`array`	Tool definitions nwere `name`, `description` na `input_schema`.

Anabatara maka ndakọrịta, na-enweghị mmetụta na ọnụọgụ: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Ị nwere ike inye body nke request n'ezie na-enweghị mgbanwe.

Azịza bụ JSON object nwere fields ndị a:

Field	Ụdị	Nkọwa
`input_tokens`	`integer`	Tokens nke input dum: system prompt, messages, tools na images.

Models akwadoro

Endpoints abụọ ahụ na-agụ maka hosted open-weight models. GET /v1/models na-depụta /v1/tokenize na /v1/messages/count_tokens n'ime endpoints nke model ọ bụla na-akwado ha. A na-aza uru model ọ bụla ọzọ, gụnyere Shannon ids, na 400.

DeepSeek-V4-Pro-0813-3BIT-REAP
GLM-5.2-3BIT-REAP
Kimi-K3-3BIT-REAP
Nemotron3Ultra-3BIT-REAP
MiniMax-M3-3BIT-REAP
DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP
Kimi-K2.6-W4A16-AUTOROUND-REAP
Laguna-S-2.1-W4A16-AUTOROUND-REAP
inkling-W4A16-AUTOROUND-REAP
MiMo-V2.5-Pro-W8A16
MiMo-V2.5-W8A16
Hy3-W8A16

Maka Shannon model, gụọ ọnụọgụ tokens site na object usage nke azịza.

Otu e si eme ọnụọgụ

A na-agụ model ọ bụla site na tokenizer nke ya na chat template nke ya. A naghị eji atụmatụ si na mkpụrụedemede ma ọ bụ okwu.

Ihe a na-agụ	Iwu
Text	Tokens nke string dị ka e zipụrụ ya. String efu na-agụ 0.
Messages	A na-edozi messages na tools site na chat template nke model n'onwe ya, ruo ebe azịza na-amalite, a na-agụkwa prompt ahụ dum.
Roles	A na-agụ messages `system`, `user`, `assistant` na `tool`. A na-agụ `developer` dị ka `system`. Message na-enweghị content na-enweghị tool call anaghị agbakwunye ihe ọ bụla.
Tool calls na results	Tool calls nke assistant turns gara aga na results ha bụ akụkụ nke ọnụọgụ, na endpoints abụọ ahụ.
Images	Image e zipụrụ n'ime body (base64 ma ọ bụ `data:` URL) na-agbakwunye otu token kwa patch pixel 28 × 28: `ceil(width / 28) × ceil(height / 28)`. Endpoints ndị a anaghị ebubata image enyere dị ka `http(s)` URL, ma na-agụ ya 1,024.

Ihe atụ: image 1,024 × 768 pixel na-agụ ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = tokens 1,036.

Ọnụọgụ na ihe a na-akwụ ụgwọ maka request

A na-eme ọnụọgụ nke request dum n'otu ụzọ ahụ dị ka input count nke request n'ezie nwere otu model, messages na tools. Azịza na-akọ ọnụọgụ ahụ dị ka usage.prompt_tokens na Chat Completions, dị ka usage.input_tokens na Responses, na dị ka usage.input_tokens gbakwunyere usage.cache_read_input_tokens na Messages.

Ọnụọgụ bụ input tupu cached-input discount. Request n'ezie nwere ike ịgụ akụkụ nke input ahụ site na cache ma na-akwụ ụgwọ akụkụ ahụ na cached rate. Caching prompt
Image enyere dị ka http(s) URL na-agụ 1,024 ebe a. Request n'ezie na-ebubata image ma na-agụ ya site na nha ya na pixels, ya mere ọnụọgụ abụọ ahụ nwere ike ịdị iche. Zipụ image dị ka base64 iji nweta otu ọnụọgụ.
Output abụghị akụkụ nke ọnụọgụ. A na-akwụ ụgwọ azịza nke request n'ezie dị ka output tokens n'elu, gụnyere reasoning.
Ọnụọgụ text enweghị chat formatting. Jiri ya tụọ akwụkwọ ma ọ bụ akụkụ prompt, na ụdị messages iji tụọ request.

Iji mee ọnụọgụ ka ọ bụrụ ọnụahịa, jiri input price nke model kwa tokens 1M mụbaa ya. Models & ọnụahịa

Limits

Oke	Uru	Karịa ya
Ogologo nke `text`	Bytes 4,000,000 (UTF-8)	`413` nwere ozi `text too long`
Request body	32 MiB	`413`
Kwa request	Otu text ma ọ bụ otu mkparịta ụka	Zipụ otu request maka text ọ bụla iji gụọ texts dị iche iche.

A naghị agụ counting calls n'ime oke nke requests 120 n'otu nkeji. Limits na balance

Njehie

Status	Ụdị	Ozi	Mgbe
`400`	`invalid_request_error`	`tokenize is available for the hosted open models; unknown model: <model>`	`/v1/tokenize` nwere `model` abụghị hosted open-weight id.
`400`	`invalid_request_error`	`count_tokens is available for the hosted open models; unknown model: <model>`	`/v1/messages/count_tokens` nwere `model` abụghị hosted open-weight id, ma ọ bụ na-enweghị `model`.
`400`	`invalid_request_error`	send `text` or `messages`	`/v1/tokenize` na-enweghị `text` ma ọ bụ `messages`.
`401`	`authentication_error`	`Missing authentication / Invalid API key`	Ezipụghị key, ma ọ bụ key adịghị mma.
`413`	`invalid_request_error`	`text too long`	`text` ka ogologo karịa bytes 4,000,000. A na-aza body karịrị 32 MiB nkekwa na `413`.
`415`	`invalid_request_error`	Expected request with `Content-Type: application/json`	Request enweghị JSON content type.
`422`	`invalid_request_error`	`Failed to deserialize the JSON body into the target type: …`	Field chọrọ achọrọ adịghị (`model` na `/v1/tokenize`, `messages` na `/v1/messages/count_tokens`) ma ọ bụ field nwere type na-ezighi ezi.
`503`	`api_error`	`token counting is temporarily unavailable for this model`	Enweghị ike ime ọnụọgụ maka model a n'oge a. Nwaa ọzọ mgbe e mesịrị.

/v1/tokenize na-eweghachi njehie n'ụdị OpenAI. Na /v1/messages/count_tokens njehie nke endpoint n'onwe ya (400 maka model, 503) na-abịa n'ụdị Anthropic, 401, 413, 415 na 422 na-abịa n'ụdị OpenAI. Gụọ status code buru ụzọ, wee error.type na error.message, nke dị n'ụdị abụọ ahụ.

400 /v1/tokenize

{
  "error": {
    "type": "invalid_request_error",
    "message": "tokenize is available for the hosted open models; unknown model: shannon-3"
  }
}

400 /v1/messages/count_tokens

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
  }
}