Simira kuzvirimo
Kuverenga tokens

Kuverenga tokens

Verenga tokens dze-text kana dze-request yose usati wayitumira.

POST https://api.shannon-ai.com/v1/tokenize

POST https://api.shannon-ai.com/v1/messages/count_tokens

Ma-endpoint ese maviri anoverenga ne-tokenizer ye-model yaunotumidza, uye hapana model inomhanya. Anofukidza hosted open-weight models. /v1/tokenize inotora text chaiyo kana hurukuro ye-Chat Completions. /v1/messages/count_tokens inotora request mu-Anthropic Messages format, ndiyo call inoitwa ne-Anthropic SDK ne-Claude Code.

Kuverenga hakubhadharwi. Call inoda API key yako, haitore chinhu mu-balance yako uye haioneki mu-usage log yako.

Verenga text

Tumira model ne-text. Text inoverengwa sezvairi, pasina chat formatting yakaipoteredza.

import requests

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
        "text": "Hello, world",
    },
)
print(response.json()["tokens"])
200 Reply
{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 3
}

Nhamba dziri mumhinduro dzepeji rino i-mienzaniso. Text imwechete inopa count yakasiyana pa-model yakasiyana.

Verenga chat request

Tumira model ne-messages, pamwe ne-tools kana request ichiine, chaizvo sezvaungatumira ku-/v1/chat/completions. Mhinduro i-size ye-input yose.

import requests

request = {
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is the weather in Paris?"},
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
}

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json=request,
)
print(response.json()["tokens"])
200 Reply
{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 164
}

Fields dze-/v1/tokenize

Field Mhando Tsananguro
model string Inodiwa. Hosted open-weight model id. Mavara makuru nemadiki anotorwa zvakafanana.
text string Text yekuverenga sezvairi, pasina chat formatting. Kusvika ku-4,000,000 bytes. Tumira text kana messages; kana dzese dziripo, text ndiyo inoverengwa.
messages array Chat messages mu-Chat Completions format. Dzinoverengwa se-input yakazara ye-request: message imwe neimwe ine formatting inoiswa ne-chat template ye-model yakaipoteredza.
tools array Tool definitions dzekusanganisa mu-count. Dzinoshandiswa pamwe chete ne-messages.

Mhinduro i-JSON object ine ma-field aya:

Field Mhando Tsananguro
model string Model id yakaitirwa count, nemanyorerwo ayo akaburitswa.
tokens integer Ne-text: tokens dze-text. Ne-messages: tokens dze-input yose, mifananidzo yakaverengwa.

Verenga Messages request

Tumira body yaungatumira ku-/v1/messages: model, messages, ne-system ne-tools kana uchizvishandisa. Official Anthropic SDKs dzinodanidzira endpoint iyi kuburikidza ne-messages.count_tokens.

import anthropic

client = anthropic.Anthropic(
    api_key="YOUR_API_KEY",
    base_url="https://api.shannon-ai.com",
)

count = client.messages.count_tokens(
    model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    system="You are a concise assistant.",
    messages=[
        {"role": "user", "content": "Summarise the attached report."}
    ],
)
print(count.input_tokens)
200 Reply
{
  "input_tokens": 21
}

Fields dze-/v1/messages/count_tokens

Field Mhando Tsananguro
model string Inodiwa. Hosted open-weight model id.
messages array Inodiwa. Messages mu-Anthropic Messages format. text, image, tool_use ne-tool_result blocks dzinoverengwa.
system string | array System prompt: string kana array ye-text blocks.
tools array Tool definitions dzine name, description ne-input_schema.

Dzinogamuchirwa kuti zvienderane, pasina mhedzisiro pa-count: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Unogona kupfuudza body ye-request chaiyo isina kuchinjwa.

Mhinduro i-JSON object ine ma-field aya:

Field Mhando Tsananguro
input_tokens integer Tokens dze-input yose: system prompt, messages, tools nemifananidzo.

Ma-model anotsigirwa

Ma-endpoint ese maviri anoverenga ye-hosted open-weight models. GET /v1/models inoratidza /v1/tokenize ne-/v1/messages/count_tokens mu-endpoints ye-model imwe neimwe inodzitsigira. Chero model value imwe, Shannon ids dzakaverengwa, inopindurwa ne-400.

  • DeepSeek-V4-Pro-0813-3BIT-REAP
  • GLM-5.2-3BIT-REAP
  • Kimi-K3-3BIT-REAP
  • Nemotron3Ultra-3BIT-REAP
  • MiniMax-M3-3BIT-REAP
  • DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP
  • Kimi-K2.6-W4A16-AUTOROUND-REAP
  • Laguna-S-2.1-W4A16-AUTOROUND-REAP
  • inkling-W4A16-AUTOROUND-REAP
  • MiMo-V2.5-Pro-W8A16
  • MiMo-V2.5-W8A16
  • Hy3-W8A16

Pa-Shannon model, verenga huwandu hwe-tokens kubva ku-usage object yemhinduro.

Kuti count inoitwa sei

Model imwe neimwe inoverengwa ne-tokenizer yayo pachayo ne-chat template yayo pachayo. Hapana estimate inobva pa-characters kana mazwi inoshandiswa.

Zvinoverengwa Mutemo
Text Tokens dze-string sezvayakatumirwa. String isina chinhu inoverengwa 0.
Messages Messages ne-tools zvinorongwa ne-chat template ye-model pachayo, kusvika panotanga mhinduro, uye prompt iyoyo yose inoverengwa.
Roles system, user, assistant ne-tool messages dzinoverengwa. developer inoverengwa se-system. Message isina content uye isina tool call haiwedzere chinhu.
Tool calls nemhedzisiro Tool calls dze-assistant turns dzekare nemhedzisiro yadzo zviri chikamu che-count, pa-endpoints dzese mbiri.
Mifananidzo Mufananidzo watumirwa mukati me-body (base64 kana data: URL) unowedzera token imwe pa-28 × 28 pixel patch imwe neimwe: ceil(width / 28) × ceil(height / 28). Mufananidzo wakapiwa se-http(s) URL haudhaunirodhwi ne-endpoints idzi uye unoverengwa 1,024.

Muenzaniso: mufananidzo we-1,024 × 768 pixel unoverengwa ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 tokens.

Count uye zvinobhadharwa ne-request

Kuverenga kwe-request yose kunoitwa nenzira imwechete ne-input count ye-request chaiyo ine model, messages ne-tools zvakafanana. Mhinduro inoratidza nhamba iyoyo se-usage.prompt_tokens pa-Chat Completions, se-usage.input_tokens pa-Responses, uye se-usage.input_tokens pamwe ne-usage.cache_read_input_tokens pa-Messages.

  • Count ndiyo input isati yawana cached-input discount. Request chaiyo inogona kuverenga chikamu che-input iyoyo kubva ku-cache ndokuibhadhara pa-cached rate. Kuchenza kwe-prompt
  • Mufananidzo wakapiwa se-http(s) URL unoverengwa 1,024 pano. Request chaiyo inodhaunirodha mufananidzo ndokuuverenga kubva pa-size yawo mu-pixels, saka nhamba mbiri dzinogona kusiyana. Tumira mufananidzo se-base64 kuti uwane nhamba imwechete.
  • Output haisi chikamu che-count. Mhinduro ye-request chaiyo inobhadharwa se-output tokens pamusoro, reasoning yakaverengwa.
  • text count haina chat formatting. Iishandise kuyera document kana chikamu che-prompt, uye messages form kuyera request.

Kushandura count kuita mutengo, ichiwanza ne-input price ye-model pa1M tokens. Ma-model nemitengo

Limits

Limit Value Pamusoro payo
Kureba kwe-text 4,000,000 bytes (UTF-8) 413 ine message text too long
Request body 32 MiB 413
Pa-request Text imwe kana hurukuro imwe Tumira request imwe pa-text imwe kuverenga ma-text akawanda.

Counting calls hadziverengwi mu-limit ye-120 requests paminiti imwe. Limits ne-balance

Errors

Status Mhando Message Rinhi
400 invalid_request_error tokenize is available for the hosted open models; unknown model: <model> /v1/tokenize ine model isiri hosted open-weight id.
400 invalid_request_error count_tokens is available for the hosted open models; unknown model: <model> /v1/messages/count_tokens ine model isiri hosted open-weight id, kana isina model.
400 invalid_request_error send `text` or `messages` /v1/tokenize isina text kana messages.
401 authentication_error Missing authentication / Invalid API key Hapana key yakatumirwa, kana key haina kunaka.
413 invalid_request_error text too long text yakareba kupfuura 4,000,000 bytes. Body inopfuura 32 MiB inopindurwawo ne-413.
415 invalid_request_error Expected request with `Content-Type: application/json` Request haina JSON content type.
422 invalid_request_error Failed to deserialize the JSON body into the target type: … Field inodiwa haisipo (model pa-/v1/tokenize, messages pa-/v1/messages/count_tokens) kana field ine type isiriyo.
503 api_error token counting is temporarily unavailable for this model Count haigoni kuitwa ye-model iyi panguva ino. Edza zvakare gare gare.

/v1/tokenize inodzosa errors mu-OpenAI shape. Pa-/v1/messages/count_tokens errors dze-endpoint pachayo (400 ye-model, 503) dzinouya mu-Anthropic shape, uye 401, 413, 415 ne-422 dzinouya mu-OpenAI shape. Verenga status code kutanga, wozoverenga error.type ne-error.message, dziripo mu-shapes dzese mbiri.

400 /v1/tokenize
{
  "error": {
    "type": "invalid_request_error",
    "message": "tokenize is available for the hosted open models; unknown model: shannon-3"
  }
}
400 /v1/messages/count_tokens
{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
  }
}