Yiya kumxholo
Ukubala ii-token

Ukubala ii-token

Bala ii-token zombhalo okanye zesicelo esipheleleyo phambi kokuba usithumele.

POST https://api.shannon-ai.com/v1/tokenize

POST https://api.shannon-ai.com/v1/messages/count_tokens

Zozibini ii-endpoint zibala nge-tokenizer yemodeli oyichazayo, kwaye akukho modeli iqhubayo. Zigubungela iimodeli ze-open-weight ezisingathiweyo. I-/v1/tokenize yamkela umbhalo olula okanye incoko ye-Chat Completions. I-/v1/messages/count_tokens yamkela isicelo kwifomathi ye-Anthropic Messages, nobu bubizo obwenziwa yi-Anthropic SDK kunye ne-Claude Code.

Ukubala kusimahla. Ubizo lufuna i-API key yakho, ayithathi nto kwibhalansi yakho kwaye ayivelanga kwi-log yakho ye-usage.

Bala umbhalo

Thumela i-model kunye ne-text. Umbhalo ubalwa njengoko unjalo, ngaphandle kwefomathi yencoko ejikelezileyo.

import requests

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
        "text": "Hello, world",
    },
)
print(response.json()["tokens"])
200 Impendulo
{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 3
}

Amanani kwiimpendulo zeli phepha ngemizekelo. Umbhalo ofanayo unika inani elahlukileyo kwimodeli eyahlukileyo.

Bala isicelo sencoko

Thumela i-model kunye ne-messages, kunye ne-tools xa isicelo sinazo, ngokufanayo nanjengoko ungazithumela kwi-/v1/chat/completions. Impendulo bubungakanani begalelo lonke.

import requests

request = {
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is the weather in Paris?"},
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
}

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json=request,
)
print(response.json()["tokens"])
200 Impendulo
{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 164
}

Iintsimi ze-/v1/tokenize

Intsimi Uhlobo Inkcazelo
model string Iyafuneka. I-id yemodeli ye-open-weight esingathiweyo. Oonobumba abakhulu nabancinci bathathwa ngokufanayo.
text string Umbhalo wokubala njengoko unjalo, ngaphandle kwefomathi yencoko. Ukuya kuthi ga kwi-4,000,000 yeebhayithi. Thumela i-text okanye i-messages; xa zozibini zikhona, i-text iyabalwa.
messages array Imiyalezo yencoko kwifomathi ye-Chat Completions. Ibalwa njengegalelo elipheleleyo lesicelo: umyalezo ngamnye kunye nefomathi etemplate yencoko yemodeli iyibeka ngeenxa zonke.
tools array Uchazo lwee-tool oluza kubandakanywa kwinani. Lusetyenziswa kunye ne-messages.

Impendulo yi-JSON object enezi ntsimi:

Intsimi Uhlobo Inkcazelo
model string I-id yemodeli enikwe inani, ngendlela yayo epapashiweyo yokubhala.
tokens integer Kunye ne-text: ii-token zombhalo. Kunye ne-messages: ii-token zegalelo lonke, kubandakanywa nemifanekiso.

Bala isicelo se-Messages

Thumela umzimba obuya kuwuthumela kwi-/v1/messages: i-model, i-messages, kunye ne-system ne-tools xa uzisebenzisa. Ii-Anthropic SDK ezisemthethweni zibiza le endpoint nge-messages.count_tokens.

import anthropic

client = anthropic.Anthropic(
    api_key="YOUR_API_KEY",
    base_url="https://api.shannon-ai.com",
)

count = client.messages.count_tokens(
    model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    system="You are a concise assistant.",
    messages=[
        {"role": "user", "content": "Summarise the attached report."}
    ],
)
print(count.input_tokens)
200 Impendulo
{
  "input_tokens": 21
}

Iintsimi ze-/v1/messages/count_tokens

Intsimi Uhlobo Inkcazelo
model string Iyafuneka. I-id yemodeli ye-open-weight esingathiweyo.
messages array Iyafuneka. Imiyalezo kwifomathi ye-Anthropic Messages. Iibhloko ze-text, image, tool_use kunye ne-tool_result ziyabalwa.
system string | array I-system prompt: umtya okanye uluhlu lweebhloko zombhalo.
tools array Uchazo lwee-tool kunye ne-name, i-description kunye ne-input_schema.

Ziyamkelwa ukuze kuhambelane, azinampembelelo kwinani: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Unokudlulisa umzimba wesicelo sokwenene ngaphandle kotshintsho.

Impendulo yi-JSON object enezi ntsimi:

Intsimi Uhlobo Inkcazelo
input_tokens integer Ii-token zegalelo lonke: i-system prompt, imiyalezo, ii-tool kunye nemifanekiso.

Iimodeli ezixhaswayo

Zozibini ii-endpoint zibala iimodeli ze-open-weight ezisingathiweyo. I-GET /v1/models idwelisa i-/v1/tokenize kunye ne-/v1/messages/count_tokens kwi-endpoints yemodeli nganye ezixhasayo. Naliphi na elinye ixabiso le-model, kubandakanywa nee-id zeShannon, liphendulwa nge-400.

  • DeepSeek-V4-Pro-0813-3BIT-REAP
  • GLM-5.2-3BIT-REAP
  • Kimi-K3-3BIT-REAP
  • Nemotron3Ultra-3BIT-REAP
  • MiniMax-M3-3BIT-REAP
  • DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP
  • Kimi-K2.6-W4A16-AUTOROUND-REAP
  • Laguna-S-2.1-W4A16-AUTOROUND-REAP
  • inkling-W4A16-AUTOROUND-REAP
  • MiMo-V2.5-Pro-W8A16
  • MiMo-V2.5-W8A16
  • Hy3-W8A16

Kwimodeli yeShannon, funda amanani ee-token kwi-object ye-usage yempendulo.

Indlela inani elenziwa ngayo

Imodeli nganye ibalwa nge-tokenizer yayo kunye netemplate yayo yencoko. Akusetyenziswa ngqikelelo esekelwe kwinani lohlobo okanye lamagama.

Okubalwayo Umthetho
Umbhalo Ii-token zomtya njengoko uthunyelwe. Umtya ongenanto ubalwa njenge-0.
Imiyalezo Imiyalezo nee-tool zicwangciswa ngetemplate yencoko yemodeli ngokwayo, ukuya kuthi ga kwindawo apho impendulo iqala khona, kwaye lonke olo prompt luyabalwa.
Iindima Imiyalezo ye-system, user, assistant kunye ne-tool iyabalwa. I-developer ibalwa njenge-system. Umyalezo ongenamxholo kwaye ongenabizo lwe-tool awongezi nto.
Ubizo lwee-tool neziphumo Ubizo lwee-tool lwemijikelo yangaphambili yesincedisi kunye neziphumo zazo ziyinxalenye yenani, kuzo zozibini ii-endpoint.
Imifanekiso Umfanekiso othunyelwe ngaphakathi kumzimba (i-base64 okanye i-URL ye-data:) wongeza i-token enye kwiqhekeza ngalinye leepikseli ezingama-28 × 28: ceil(width / 28) × ceil(height / 28). Umfanekiso onikwe njenge-URL ye-http(s) awulandwa zezi endpoint kwaye ubalwa njenge-1,024.

Umzekelo: umfanekiso oyi-1,024 × 768 yeepikseli ubalwa njenge-ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 yee-token.

Inani kunye nokuhlawulwa kwesicelo

Ukubalwa kwesicelo esipheleleyo kwenziwa ngendlela efanayo nokubalwa kwegalelo lesicelo sokwenene kunye nemodeli, imiyalezo kunye nee-tool ezifanayo. Impendulo ibika eli nani njenge-usage.prompt_tokens kwi-Chat Completions, njenge-usage.input_tokens kwi-Responses, nanjenge-usage.input_tokens kunye ne-usage.cache_read_input_tokens kwi-Messages.

  • Inani lilixabiso legalelo phambi kwesaphulelo segalelo le-cache. Isicelo sokwenene sinokufunda inxalenye yegalelo kwi-cache sihlawule loo nxalenye ngexabiso le-cache. I-Prompt caching
  • Umfanekiso onikwe njenge-URL ye-http(s) ubalwa njenge-1,024 apha. Isicelo sokwenene silanda umfanekiso sibale ngobungakanani bawo kwiipikseli, ngoko amanani amabini anokwahluka. Thumela umfanekiso njenge-base64 ukufumana inani elifanayo.
  • Okuphumayo akuyonxalenye yenani. Impendulo yesicelo sokwenene ihlawulwa njengee-token zokuphuma ngaphezulu, kubandakanywa ukuqiqa.
  • Inani le-text alinalo uhlobo lwencoko. Lisebenzise ukulinganisa uxwebhu okanye inxalenye ye-prompt, kwaye uhlobo lwe-messages ukulinganisa isicelo.

Ukuguqula inani libe yindleko, lilinganise ngexabiso legalelo lemodeli kwii-token ezingu-1M. Iimodeli namaxabiso

Imida

Umda Ixabiso Ngaphezu kwawo
Ubude be-text 4,000,000 yeebhayithi (UTF-8) 413 kunye nomyalezo othi text too long
Umzimba wesicelo 32 MiB 413
Ngesicelo Umbhalo omnye okanye incoko enye Thumela isicelo esinye ngombhalo ukubala imibhalo emininzi.

Iminxeba yokubala ayibalwa kumda wezicelo ezili-120 ngomzuzu. Imida nebhalansi

Iimpazamo

Isimo Uhlobo Umyalezo Xa
400 invalid_request_error tokenize is available for the hosted open models; unknown model: <model> I-/v1/tokenize kunye ne-model engeyiyo i-id ye-open-weight esingathiweyo.
400 invalid_request_error count_tokens is available for the hosted open models; unknown model: <model> I-/v1/messages/count_tokens kunye ne-model engeyiyo i-id ye-open-weight esingathiweyo, okanye ngaphandle kwe-model.
400 invalid_request_error send `text` or `messages` I-/v1/tokenize ngaphandle kwe-text okanye i-messages.
401 authentication_error Missing authentication / Invalid API key Akukho sitshixo sithunyelweyo, okanye isitshixo asisebenzi.
413 invalid_request_error text too long I-text ide kuneebhayithi ezingu-4,000,000. Umzimba ongaphezu kwe-32 MiB uphendulwa nge-413 nawo.
415 invalid_request_error Expected request with `Content-Type: application/json` Isicelo asinalo uhlobo lomxholo lwe-JSON.
422 invalid_request_error Failed to deserialize the JSON body into the target type: … Intsimi efunekayo ayikho (i-model kwi-/v1/tokenize, i-messages kwi-/v1/messages/count_tokens) okanye intsimi inohlobo olungalunganga.
503 api_error token counting is temporarily unavailable for this model Ukubala akunakwenziwa kule modeli okwangoku. Zama kwakhona kamva.

I-/v1/tokenize ibuyisa iimpazamo ngesimo se-OpenAI. Kwi-/v1/messages/count_tokens iimpazamo ze-endpoint ngokwayo (i-400 yemodeli, i-503) zifika ngesimo se-Anthropic, kwaye i-401, i-413, i-415 kunye ne-422 zifika ngesimo se-OpenAI. Funda ikhowudi yesimo kuqala, emva koko i-error.type kunye ne-error.message, ezikhoyo kuzo zozibini izimo.

400 /v1/tokenize
{
  "error": {
    "type": "invalid_request_error",
    "message": "tokenize is available for the hosted open models; unknown model: shannon-3"
  }
}
400 /v1/messages/count_tokens
{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
  }
}