Ho bala tokens

Bala tokens tsa mongolo kapa tsa kopo eohle pele u e romela.

POST https://api.shannon-ai.com/v1/tokenize

POST https://api.shannon-ai.com/v1/messages/count_tokens

Li-endpoint ka bobeli li bala ka tokenizer ea model eo u e rehang, mme ha ho model e sebetsang. Li koahela li-model tse hosted tsa open-weight. /v1/tokenize e amohela mongolo o hlakileng kapa moqoqo oa Chat Completions. /v1/messages/count_tokens e amohela kopo ka sebopeho sa Anthropic Messages, e leng kopo eo Anthropic SDK le Claude Code li e etsang.

Ho bala ke mahala. Kopo e hloka senotlolo sa hau sa API, ha e nke letho ho balance ea hau mme ha e hlahe ho usage log ea hau.

Bala mongolo

Romela model le text. Mongolo o baloa joalo ka ha o le, ntle le chat formatting e o pota-potileng.

import requests

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
        "text": "Hello, world",
    },
)
print(response.json()["tokens"])

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    text: "Hello, world",
  }),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "text": "Hello, world"
  }'

200 Karabo

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 3
}

Linomoro tse likarabong leqepheng lena ke mehlala. Mongolo o tšoanang o fana ka palo e fapaneng ho model e fapaneng.

Bala kopo ea chat

Romela model le messages, hammoho le tools ha kopo e na le tsona, hantle joalo ka ha u ka li romela ho /v1/chat/completions. Karabo ke boholo ba input eohle.

import requests

request = {
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is the weather in Paris?"},
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
}

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json=request,
)
print(response.json()["tokens"])

const request = {
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  messages: [
    { role: "system", content: "You are a concise assistant." },
    { role: "user", content: "What is the weather in Paris?" },
  ],
  tools: [
    {
      type: "function",
      function: {
        name: "get_weather",
        description: "Current weather for a city",
        parameters: {
          type: "object",
          properties: { city: { type: "string" } },
          required: ["city"],
        },
      },
    },
  ],
};

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify(request),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
          }
        }
      }
    ]
  }'

200 Karabo

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 164
}

Li-field tsa /v1/tokenize

Field	Mofuta	Tlhaloso
`model`	`string`	E hlokahala. Id ea model e hosted ea open-weight. Litlhaku tse kholo le tse nyane li nkoa li tšoana.
`text`	`string`	Mongolo oo o lokelang ho baloa joalo ka ha o le, ntle le chat formatting. Ho fihlela ho li-byte tse 4,000,000. Romela `text` kapa `messages`; ha tse peli li le teng, `text` e a baloa.
`messages`	`array`	Melaetsa ea chat ka sebopeho sa Chat Completions. E baloa e le input eohle ea kopo: molaetsa o mong le o mong o nang le formatting eo chat template ea model e e beha ho o pota-pota.
`tools`	`array`	Litlhaloso tsa tool tse lokelang ho kenyelletsoa palong. Li sebelisoa hammoho le `messages`.

Karabo ke ntho ea JSON e nang le li-field tsena:

Field	Mofuta	Tlhaloso
`model`	`string`	Id ea model eo palo e entsoeng bakeng sa eona, ka mongolo oa eona o phatlalalitsoeng.
`tokens`	`integer`	Ka `text`: tokens tsa mongolo. Ka `messages`: tokens tsa input eohle, ho kenyelletsa litšoantšo.

Bala kopo ea Messages

Romela body eo u ka e romelang ho /v1/messages: model, messages, le system le tools ha u li sebelisa. Li-SDK tsa semmuso tsa Anthropic li letsetsa endpoint ena ka messages.count_tokens.

import anthropic

client = anthropic.Anthropic(
    api_key="YOUR_API_KEY",
    base_url="https://api.shannon-ai.com",
)

count = client.messages.count_tokens(
    model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    system="You are a concise assistant.",
    messages=[
        {"role": "user", "content": "Summarise the attached report."}
    ],
)
print(count.input_tokens)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: "YOUR_API_KEY",
  baseURL: "https://api.shannon-ai.com",
});

const count = await client.messages.countTokens({
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  system: "You are a concise assistant.",
  messages: [
    { role: "user", content: "Summarise the attached report." },
  ],
});
console.log(count.input_tokens);

curl https://api.shannon-ai.com/v1/messages/count_tokens \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "system": "You are a concise assistant.",
    "messages": [
      {"role": "user", "content": "Summarise the attached report."}
    ]
  }'

200 Karabo

{
  "input_tokens": 21
}

Li-field tsa /v1/messages/count_tokens

Field	Mofuta	Tlhaloso
`model`	`string`	E hlokahala. Id ea model e hosted ea open-weight.
`messages`	`array`	E hlokahala. Melaetsa ka sebopeho sa Anthropic Messages. Li-block tsa `text`, `image`, `tool_use` le `tool_result` lia baloa.
`system`	`string \| array`	System prompt: string kapa array ea li-block tsa mongolo.
`tools`	`array`	Litlhaloso tsa tool tse nang le `name`, `description` le `input_schema`.

Li amohelwa bakeng sa compatibility, ntle le phello palong: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. U ka fetisa body ea kopo ea 'nete ntle le phetoho.

Karabo ke ntho ea JSON e nang le li-field tsena:

Field	Mofuta	Tlhaloso
`input_tokens`	`integer`	Tokens tsa input eohle: system prompt, melaetsa, tools le litšoantšo.

Li-model tse tšehetsoang

Li-endpoint ka bobeli li bala bakeng sa li-model tse hosted tsa open-weight. GET /v1/models e thathamisa /v1/tokenize le /v1/messages/count_tokens ho endpoints ea model e 'ngoe le e 'ngoe e li tšehetsang. Boleng bo bong bofe ba model, ho kenyelletsa li-id tsa Shannon, bo arajoa ka 400.

DeepSeek-V4-Pro-0813-3BIT-REAP
GLM-5.2-3BIT-REAP
Kimi-K3-3BIT-REAP
Nemotron3Ultra-3BIT-REAP
MiniMax-M3-3BIT-REAP
DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP
Kimi-K2.6-W4A16-AUTOROUND-REAP
Laguna-S-2.1-W4A16-AUTOROUND-REAP
inkling-W4A16-AUTOROUND-REAP
MiMo-V2.5-Pro-W8A16
MiMo-V2.5-W8A16
Hy3-W8A16

Bakeng sa model ea Shannon, bala palo ea tokens ho ntho ea usage ea karabo.

Hore na palo e etsoa joang

Model e 'ngoe le e 'ngoe e baloa ka tokenizer ea eona le chat template ea eona. Ha ho khakanyo ho tsoa litlhakung kapa mantsoeng e sebelisoang.

Se baloang	Molao
Mongolo	Tokens tsa string joalo ka ha e rometsoe. String e se nang letho e bala 0.
Melaetsa	Melaetsa le tools li hlophisoa ka chat template ea model ka boeona, ho fihlela moo karabo e qalang teng, mme prompt eo eohle e baloa.
Li-role	Melaetsa ea `system`, `user`, `assistant` le `tool` e a baloa. `developer` e baloa e le `system`. Molaetsa o se nang litaba le kopo ea tool ha o kenye letho.
Likopo tsa tool le liphetho	Likopo tsa tool tsa li-turn tse fetileng tsa assistant le liphetho tsa tsona ke karolo ea palo, ho li-endpoint ka bobeli.
Litšoantšo	Setšoantšo se rometsoeng ka hare ho body (base64 kapa URL ea `data:`) se eketsa token e le 'ngoe ka patch ea pixel tsa 28 × 28: `ceil(width / 28) × ceil(height / 28)`. Setšoantšo se fanoeng e le URL ea `http(s)` ha se download-oe ke li-endpoint tsena mme se bala 1,024.

Mohlala: setšoantšo sa pixel tse 1,024 × 768 se bala ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 tokens.

Palo le seo kopo e se lefisoang

Palo ea kopo eohle e etsoa ka tsela e tšoanang le palo ea input ea kopo ea 'nete e nang le model, melaetsa le tools tse tšoanang. Karabo e tlaleha palo eo e le usage.prompt_tokens ho Chat Completions, e le usage.input_tokens ho Responses, le e le usage.input_tokens hammoho le usage.cache_read_input_tokens ho Messages.

Palo ke input pele ho discount ea input e cached. Kopo ea 'nete e ka bala karolo ea input eo ho cache mme ea lefisa karolo eo ka cached rate. Caching ea Prompt
Setšoantšo se fanoeng e le URL ea http(s) se bala 1,024 mona. Kopo ea 'nete e download setšoantšo mme e se bala ho tloha boholong ba sona ka pixel, kahoo linomoro tse peli li ka fapana. Romela setšoantšo e le base64 ho fumana nomoro e tšoanang.
Output hase karolo ea palo. Karabo ea kopo ea 'nete e lefisoa e le output tokens ho tlatsetsa, ho kenyelletsa reasoning.
Palo ea text ha e na formatting ea chat. E sebelise ho lekanya tokomane kapa karolo ea prompt, le sebopeho sa messages ho lekanya kopo.

Ho fetolela palo ho theko, e atolose ka theko ea input ea model ka 1M tokens. Li-model le litheko

Meeli

Moeli	Boleng	Ka holimo ho oona
Bolelele ba `text`	Li-byte tse 4,000,000 (UTF-8)	`413` le molaetsa `text too long`
Body ea kopo	32 MiB	`413`
Ka kopo	Mongolo o le mong kapa moqoqo o le mong	Romela kopo e le 'ngoe ka mongolo ho bala mengolo e mengata.

Likopo tsa ho bala ha li baloe ho moeli oa likopo tse 120 ka motsotso. Meeli le balance

Liphoso

Boemo	Mofuta	Molaetsa	Neng
`400`	`invalid_request_error`	`tokenize is available for the hosted open models; unknown model: <model>`	`/v1/tokenize` le `model` e seng id ea model e hosted ea open-weight.
`400`	`invalid_request_error`	`count_tokens is available for the hosted open models; unknown model: <model>`	`/v1/messages/count_tokens` le `model` e seng id ea model e hosted ea open-weight, kapa ntle le `model`.
`400`	`invalid_request_error`	send `text` or `messages`	`/v1/tokenize` ntle le `text` kapa `messages`.
`401`	`authentication_error`	`Missing authentication / Invalid API key`	Ha ho senotlolo se rometsoeng, kapa senotlolo ha se nepahale.
`413`	`invalid_request_error`	`text too long`	`text` e telele ho feta li-byte tse 4,000,000. Body e fetang 32 MiB le eona e arajoa ka `413`.
`415`	`invalid_request_error`	Expected request with `Content-Type: application/json`	Kopo ha e na content type ea JSON.
`422`	`invalid_request_error`	`Failed to deserialize the JSON body into the target type: …`	Field e hlokahalang ha e eo (`model` ho `/v1/tokenize`, `messages` ho `/v1/messages/count_tokens`) kapa field e na le mofuta o fosahetseng.
`503`	`api_error`	`token counting is temporarily unavailable for this model`	Palo e ke ke ea etsoa bakeng sa model ena hajoale. Leka hape hamorao.

/v1/tokenize e khutlisa liphoso ka sebopeho sa OpenAI. Ho /v1/messages/count_tokens liphoso tsa endpoint ka boeona (400 bakeng sa model, 503) li fihla ka sebopeho sa Anthropic, mme 401, 413, 415 le 422 li fihla ka sebopeho sa OpenAI. Bala khoutu ea boemo pele, ebe error.type le error.message, tse teng mebopehong ka bobeli.

400 /v1/tokenize

{
  "error": {
    "type": "invalid_request_error",
    "message": "tokenize is available for the hosted open models; unknown model: shannon-3"
  }
}

400 /v1/messages/count_tokens

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
  }
}