Ƙidayar tokens

Ku ƙidaya tokens na rubutu ko na cikakken request kafin ku aika shi.

POST https://api.shannon-ai.com/v1/tokenize

POST https://api.shannon-ai.com/v1/messages/count_tokens

Endpoints biyun suna ƙidaya da tokenizer na model da kuka ambata, kuma babu model da ke gudana. Suna rufe hosted open-weight models. /v1/tokenize yana ɗaukar rubutu na fili ko tattaunawar Chat Completions. /v1/messages/count_tokens yana ɗaukar request a tsarin Anthropic Messages, wato kiran da Anthropic SDK da Claude Code ke yi.

Ƙidaya kyauta ce. Kira yana buƙatar API key ɗinku, ba ya cire komai daga balance ɗinku kuma ba ya bayyana a usage log ɗinku.

Ƙidaya rubutu

Ku aika model da text. Ana ƙidaya rubutun yadda yake, ba tare da tsarin chat a kewaye da shi ba.

import requests

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
        "text": "Hello, world",
    },
)
print(response.json()["tokens"])

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    text: "Hello, world",
  }),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "text": "Hello, world"
  }'

200 Amsa

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 3
}

Lambobin da ke cikin amsoshin wannan shafi misalai ne. Rubutu iri ɗaya yana ba da ƙidaya daban a model daban.

Ƙidaya request na chat

Ku aika model da messages, tare da tools idan request yana da su, daidai kamar yadda za ku aika su zuwa /v1/chat/completions. Amsar ita ce girman dukkan input.

import requests

request = {
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is the weather in Paris?"},
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
}

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json=request,
)
print(response.json()["tokens"])

const request = {
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  messages: [
    { role: "system", content: "You are a concise assistant." },
    { role: "user", content: "What is the weather in Paris?" },
  ],
  tools: [
    {
      type: "function",
      function: {
        name: "get_weather",
        description: "Current weather for a city",
        parameters: {
          type: "object",
          properties: { city: { type: "string" } },
          required: ["city"],
        },
      },
    },
  ],
};

const response = await fetch("https://api.shannon-ai.com/v1/tokenize", {
  method: "POST",
  headers: {
    Authorization: "Bearer YOUR_API_KEY",
    "Content-Type": "application/json",
  },
  body: JSON.stringify(request),
});

const { tokens } = await response.json();
console.log(tokens);

curl https://api.shannon-ai.com/v1/tokenize \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
      {"role": "system", "content": "You are a concise assistant."},
      {"role": "user", "content": "What is the weather in Paris?"}
    ],
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "Current weather for a city",
          "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"]
          }
        }
      }
    ]
  }'

200 Amsa

{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 164
}

Fields na /v1/tokenize

Field	Nau'i	Bayani
`model`	`string`	Ana buƙata. Id na hosted open-weight model. Ana ɗaukar manyan haruffa da ƙanana iri ɗaya.
`text`	`string`	Rubutu da za a ƙidaya yadda yake, ba tare da tsarin chat ba. Har zuwa bytes 4,000,000. Ku aika `text` ko `messages`; idan duka biyun suna nan, ana ƙidaya `text`.
`messages`	`array`	Saƙonnin chat a tsarin Chat Completions. Ana ƙidaya su a matsayin cikakken input na request: kowane saƙo tare da tsarin da chat template na model ke sa a kewaye da shi.
`tools`	`array`	Ma'anonin tools da za a haɗa a cikin ƙidayar. Ana amfani da su tare da `messages`.

Amsar object ce ta JSON mai waɗannan fields:

Field	Nau'i	Bayani
`model`	`string`	Id na model da aka yi ƙidayar masa, a rubutunsa da aka wallafa.
`tokens`	`integer`	Da `text`: tokens na rubutun. Da `messages`: tokens na dukkan input, tare da hotuna.

Ƙidaya request na Messages

Ku aika body da za ku aika zuwa /v1/messages: model, messages, da system da tools idan kuna amfani da su. Anthropic SDKs na hukuma suna kiran wannan endpoint ta messages.count_tokens.

import anthropic

client = anthropic.Anthropic(
    api_key="YOUR_API_KEY",
    base_url="https://api.shannon-ai.com",
)

count = client.messages.count_tokens(
    model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    system="You are a concise assistant.",
    messages=[
        {"role": "user", "content": "Summarise the attached report."}
    ],
)
print(count.input_tokens)

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({
  apiKey: "YOUR_API_KEY",
  baseURL: "https://api.shannon-ai.com",
});

const count = await client.messages.countTokens({
  model: "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  system: "You are a concise assistant.",
  messages: [
    { role: "user", content: "Summarise the attached report." },
  ],
});
console.log(count.input_tokens);

curl https://api.shannon-ai.com/v1/messages/count_tokens \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "system": "You are a concise assistant.",
    "messages": [
      {"role": "user", "content": "Summarise the attached report."}
    ]
  }'

200 Amsa

{
  "input_tokens": 21
}

Fields na /v1/messages/count_tokens

Field	Nau'i	Bayani
`model`	`string`	Ana buƙata. Id na hosted open-weight model.
`messages`	`array`	Ana buƙata. Messages a tsarin Anthropic Messages. Ana ƙidaya blocks na `text`, `image`, `tool_use` da `tool_result`.
`system`	`string \| array`	System prompt: string ko jerin text blocks.
`tools`	`array`	Ma'anonin tools masu `name`, `description` da `input_schema`.

Ana karɓa don dacewa, ba tare da tasiri a kan ƙidayar ba: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Kuna iya mika body na ainihin request ba tare da canji ba.

Amsar object ce ta JSON mai waɗannan fields:

Field	Nau'i	Bayani
`input_tokens`	`integer`	Tokens na dukkan input: system prompt, messages, tools da hotuna.

Models da ake tallafawa

Endpoints biyun suna ƙidaya don hosted open-weight models. GET /v1/models yana lissafa /v1/tokenize da /v1/messages/count_tokens a cikin endpoints na kowane model da ke tallafa musu. Duk wata ƙimar model ta daban, tare da ids na Shannon, ana amsa ta da 400.

DeepSeek-V4-Pro-0813-3BIT-REAP
GLM-5.2-3BIT-REAP
Kimi-K3-3BIT-REAP
Nemotron3Ultra-3BIT-REAP
MiniMax-M3-3BIT-REAP
DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP
Kimi-K2.6-W4A16-AUTOROUND-REAP
Laguna-S-2.1-W4A16-AUTOROUND-REAP
inkling-W4A16-AUTOROUND-REAP
MiMo-V2.5-Pro-W8A16
MiMo-V2.5-W8A16
Hy3-W8A16

Don model na Shannon, ku karanta ƙidayar tokens daga object na usage na amsa.

Yadda ake yin ƙidayar

Ana ƙidaya kowane model da tokenizer ɗinsa da chat template ɗinsa. Ba a amfani da ƙiyasi daga haruffa ko kalmomi.

Abin da ake ƙidaya	Doka
Rubutu	Tokens na string yadda aka aika shi. String mara komai yana ƙidayuwa 0.
Messages	Ana tsara messages da tools da chat template na model da kansa, har zuwa inda amsar ke farawa, kuma ana ƙidaya dukkan wannan prompt.
Roles	Ana ƙidaya saƙonnin `system`, `user`, `assistant` da `tool`. Ana ƙidaya `developer` a matsayin `system`. Saƙo ba tare da content ba kuma ba tare da kiran tool ba ba ya ƙara komai.
Kiran tools da sakamako	Kiran tools na turns na assistant na baya da sakamakonsu suna cikin ƙidayar, a endpoints biyun.
Hotuna	Hoto da aka aika a cikin body (base64 ko `data:` URL) yana ƙara token ɗaya ga kowane patch na pixels 28 × 28: `ceil(width / 28) × ceil(height / 28)`. Hoto da aka bayar a matsayin URL na `http(s)` waɗannan endpoints ba sa sauke shi kuma yana ƙidayuwa 1,024.

Misali: hoto mai girman 1,024 × 768 pixels yana ƙidayuwa ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = tokens 1,036.

Ƙidayar da abin da ake cajin request

Ana yin ƙidayar cikakken request iri ɗaya da ƙidayar input na ainihin request mai model, messages da tools iri ɗaya. Amsa tana bayar da wannan lamba a matsayin usage.prompt_tokens a Chat Completions, a matsayin usage.input_tokens a Responses, da kuma a matsayin usage.input_tokens tare da usage.cache_read_input_tokens a Messages.

Ƙidayar ita ce input kafin rangwamen cached-input. Ainihin request na iya karanta ɓangare na wannan input daga cache kuma ya caji wannan ɓangaren a farashin cached. Adana prompt
Hoton da aka bayar a matsayin URL na http(s) yana ƙidayuwa 1,024 a nan. Ainihin request yana sauke hoton kuma yana ƙidaya shi daga girmansa a pixels, don haka lambobin biyu na iya bambanta. Ku aika hoton a matsayin base64 don samun lamba iri ɗaya.
Output ba ya cikin ƙidayar. Ana cajin amsar ainihin request a matsayin output tokens a kai, tare da reasoning.
Ƙidayar text ba ta da tsarin chat. Ku yi amfani da ita don auna takarda ko ɓangaren prompt, da tsarin messages don auna request.

Don mayar da ƙidaya zuwa kuɗi, ku ninka ta da farashin input na model ga kowane tokens 1M. Models & farashi

Iyakoki

Iyaka	Ƙima	Idan ya wuce
Tsawon `text`	bytes 4,000,000 (UTF-8)	`413` da saƙon `text too long`
Body na request	32 MiB	`413`
A kowane request	Rubutu ɗaya ko tattaunawa ɗaya	Ku aika request ɗaya ga kowane rubutu don ƙidaya rubutu da yawa.

Ba a ƙidaya kiran ƙidaya cikin iyakar requests 120 a minti ɗaya. Iyakoki da balance

Kurakurai

Status	Nau'i	Saƙo	Yaushe
`400`	`invalid_request_error`	`tokenize is available for the hosted open models; unknown model: <model>`	`/v1/tokenize` da `model` wanda ba id na hosted open-weight ba ne.
`400`	`invalid_request_error`	`count_tokens is available for the hosted open models; unknown model: <model>`	`/v1/messages/count_tokens` da `model` wanda ba id na hosted open-weight ba ne, ko ba tare da `model` ba.
`400`	`invalid_request_error`	send `text` or `messages`	`/v1/tokenize` ba tare da `text` ko `messages` ba.
`401`	`authentication_error`	`Missing authentication / Invalid API key`	Ba a aika maɓalli ba, ko kuma maɓallin ba shi da inganci.
`413`	`invalid_request_error`	`text too long`	`text` ya fi bytes 4,000,000 tsawo. Ana kuma amsa body da ya wuce 32 MiB da `413`.
`415`	`invalid_request_error`	Expected request with `Content-Type: application/json`	Request ba shi da content type na JSON.
`422`	`invalid_request_error`	`Failed to deserialize the JSON body into the target type: …`	Field da ake buƙata ya ɓace (`model` a `/v1/tokenize`, `messages` a `/v1/messages/count_tokens`) ko kuma field yana da nau'i mara kyau.
`503`	`api_error`	`token counting is temporarily unavailable for this model`	Ba za a iya yin ƙidayar ga wannan model a halin yanzu ba. Ku sake gwadawa daga baya.

/v1/tokenize yana mayar da kurakurai a tsarin OpenAI. A /v1/messages/count_tokens kurakuran endpoint ɗin kansa (400 ga model, 503) suna zuwa a tsarin Anthropic, kuma 401, 413, 415 da 422 suna zuwa a tsarin OpenAI. Ku karanta status code da farko, sannan error.type da error.message, waɗanda suke a cikin duka tsare-tsaren biyu.

400 /v1/tokenize

{
  "error": {
    "type": "invalid_request_error",
    "message": "tokenize is available for the hosted open models; unknown model: shannon-3"
  }
}

400 /v1/messages/count_tokens

{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
  }
}