Peke ki ngā ihirangi
Te tatau token

Te tatau token

Tatauria ngā token o tētahi kuputuhi, o tētahi tono katoa rānei i mua i te tuku.

POST https://api.shannon-ai.com/v1/tokenize

POST https://api.shannon-ai.com/v1/messages/count_tokens

Ka tatau ngā endpoint e rua ki te tokenizer o te model ka whakaingoatia e koe, ā, kāore he model e rere. Ka kapi i ngā model open-weight hosted. Ka whakaae te /v1/tokenize ki tētahi kuputuhi noa, ki tētahi kōrero Chat Completions rānei. Ka whakaae te /v1/messages/count_tokens ki tētahi tono i te hōputu Anthropic Messages, ko te karanga tēnei ka mahia e te SDK Anthropic me Claude Code.

He kore utu te tatau. E hiahia ana tētahi karanga ki tō API key, kāore e tango i tētahi mea i tō toenga, ā, kāore e puta i tō log whakamahinga.

Tatau i tētahi kuputuhi

Tukuna te model me te text. Ka tatauria te kuputuhi pēnei i a ia, me te kore whakahōputu chat huri noa.

import requests

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
        "text": "Hello, world",
    },
)
print(response.json()["tokens"])
200 Whakautu
{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 3
}

He tauira ngā tau i ngā whakautu o tēnei whārangi. Ka hoatu te kuputuhi ōrite i tētahi tatau rerekē i runga i tētahi model rerekē.

Tatau i tētahi tono chat

Tukuna te model me te messages, me te tools ina he tā te tono, pēnei tonu i tā koe tuku ki te /v1/chat/completions. Ko te whakautu te rahi o te input katoa.

import requests

request = {
    "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    "messages": [
        {"role": "system", "content": "You are a concise assistant."},
        {"role": "user", "content": "What is the weather in Paris?"},
    ],
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Current weather for a city",
                "parameters": {
                    "type": "object",
                    "properties": {"city": {"type": "string"}},
                    "required": ["city"],
                },
            },
        }
    ],
}

response = requests.post(
    "https://api.shannon-ai.com/v1/tokenize",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json=request,
)
print(response.json()["tokens"])
200 Whakautu
{
  "model": "DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
  "tokens": 164
}

Ngā āpure o te /v1/tokenize

Āpure Momo Whakaahuatanga
model string E hiahiatia ana. He id model open-weight hosted. Ka whakaarohia te pū nui me te pū iti hei mea ōrite.
text string He kuputuhi hei tatau pēnei i a ia, me te kore whakahōputu chat. Tae atu ki te 4,000,000 paita. Tukuna te text te messages rānei; ina kei reira ngā mea e rua, ka tatauria te text.
messages array Ngā karere chat i te hōputu Chat Completions. Ka tatauria hei input katoa o tētahi tono: ia karere me te whakahōputu ka whakanōhia e te tauira chat a te model huri noa i a ia.
tools array Ngā whakamārama tool hei uru ki te tatau. Ka whakamahia tahi me te messages.

He ahanoa JSON te whakautu me ēnei āpure:

Āpure Momo Whakaahuatanga
model string Te id model i mahia te tatau mōna, i tōna tuhinga kua whakaputaina.
tokens integer Me te text: ngā token o te kuputuhi. Me te messages: ngā token o te input katoa, tae atu ki ngā whakaahua.

Tatau i tētahi tono Messages

Tukuna te tinana ka tukuna e koe ki te /v1/messages: model, messages, me te system me te tools ina whakamahi koe i a rāua. Ka karanga ngā SDK Anthropic ōkawa i tēnei endpoint mā te messages.count_tokens.

import anthropic

client = anthropic.Anthropic(
    api_key="YOUR_API_KEY",
    base_url="https://api.shannon-ai.com",
)

count = client.messages.count_tokens(
    model="DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP",
    system="You are a concise assistant.",
    messages=[
        {"role": "user", "content": "Summarise the attached report."}
    ],
)
print(count.input_tokens)
200 Whakautu
{
  "input_tokens": 21
}

Ngā āpure o te /v1/messages/count_tokens

Āpure Momo Whakaahuatanga
model string E hiahiatia ana. He id model open-weight hosted.
messages array E hiahiatia ana. Ngā karere i te hōputu Anthropic Messages. Ka tatauria ngā poraka text, image, tool_use me tool_result.
system string | array Te system prompt: he aho, he array poraka kuputuhi rānei.
tools array Ngā whakamārama tool me te name, description me te input_schema.

E whakaaetia ana mō te hototahi, kāore he pānga ki te tatau: tool_choice, max_tokens, temperature, top_p, stop_sequences, stream, thinking. Ka taea e koe te tuku i te tinana o tētahi tono tūturu kāore i huri.

He ahanoa JSON te whakautu me ēnei āpure:

Āpure Momo Whakaahuatanga
input_tokens integer Ngā token o te input katoa: system prompt, ngā karere, ngā tool me ngā whakaahua.

Ngā model e tautokohia ana

Ka tatau ngā endpoint e rua mō ngā model open-weight hosted. Ka rārangi te GET /v1/models i te /v1/tokenize me te /v1/messages/count_tokens i ngā endpoints o ia model e tautoko ana i a rāua. Ka whakautua tētahi atu uara model, tae atu ki ngā id Shannon, ki te 400.

  • DeepSeek-V4-Pro-0813-3BIT-REAP
  • GLM-5.2-3BIT-REAP
  • Kimi-K3-3BIT-REAP
  • Nemotron3Ultra-3BIT-REAP
  • MiniMax-M3-3BIT-REAP
  • DeepSeek-V4-Flash-0731-W4A16-AUTOROUND-REAP
  • Kimi-K2.6-W4A16-AUTOROUND-REAP
  • Laguna-S-2.1-W4A16-AUTOROUND-REAP
  • inkling-W4A16-AUTOROUND-REAP
  • MiMo-V2.5-Pro-W8A16
  • MiMo-V2.5-W8A16
  • Hy3-W8A16

Mō tētahi model Shannon, pānuitia ngā tatau token mai i te ahanoa usage o tētahi whakautu.

Te āhua o te tatau

Ka tatauria ia model ki tōna ake tokenizer me tōna ake tauira chat. Kāore e whakamahia he whakatau mai i ngā tohu, i ngā kupu rānei.

Te mea e tatauria ana Ture
He kuputuhi Ngā token o te aho pēnei i tō tuku. Ka tatau tētahi aho kau hei 0.
Ngā karere Ka whakatakotoria ngā karere me ngā tool ki te tauira chat ake a te model, tae noa ki te wāhi e tīmata ai te whakautu, ā, ka tatauria taua prompt katoa.
Ngā tūranga Ka tatauria ngā karere system, user, assistant me tool. Ka tatauria te developer hei system. Kāore tētahi karere kāore he content kāore he karanga tool e tāpiri i tētahi mea.
Ngā karanga tool me ngā hua He wāhanga o te tatau ngā karanga tool o ngā hurihanga assistant o mua me ō rātou hua, i ngā endpoint e rua.
Ngā whakaahua Ka tāpiri tētahi whakaahua i tukuna i roto i te tinana (base64 he URL data: rānei) i te token kotahi mō ia kōpaki pika 28 × 28: ceil(width / 28) × ceil(height / 28). Kāore ēnei endpoint e tikiake i tētahi whakaahua i hoatu hei URL http(s), ā, ka tatau hei 1,024.

Tauira: ka tatau tētahi whakaahua 1,024 × 768 pika hei ceil(1024 / 28) × ceil(768 / 28) = 37 × 28 = 1,036 token.

Te tatau me te mea e utua ana e tētahi tono

Ka mahia te tatau o tētahi tono katoa i te ara ōrite ki te tatau input o tētahi tono tūturu me te model, ngā karere me ngā tool ōrite. Ka pūrongo tētahi whakautu i taua tau hei usage.prompt_tokens i Chat Completions, hei usage.input_tokens i Responses, hei usage.input_tokens me te usage.cache_read_input_tokens i Messages.

  • Ko te tatau te input i mua i te whakaitinga input-cached. Ka taea e tētahi tono tūturu te pānui i tētahi wāhanga o taua input mai i te cache, ā, ka whakarite i taua wāhanga ki te reiti cached. Tūhono caching
  • Ka tatau tētahi whakaahua i hoatu hei URL http(s) hei 1,024 i konei. Ka tikiake tētahi tono tūturu i te whakaahua, ā, ka tatau mai i tōna rahi pika, nō reira ka taea te rerekē o ngā tau e rua. Tukuna te whakaahua hei base64 kia whiwhi koe i te tau ōrite.
  • Kāore te output e uru ki te tatau. Ka whakaritea te whakautu o tētahi tono tūturu hei token output i runga ake, tae atu ki te reasoning.
  • Kāore he whakahōputu chat o tētahi tatau text. Whakamahia hei ine i tētahi tuhinga, i tētahi wāhanga prompt rānei, ā, te āhua messages hei ine i tētahi tono.

Hei huri i tētahi tatau hei utu, whakareatia ki te utu input a te model mō ia 1M token. Ngā model me ngā utu

Ngā tepe

Tepe Uara Kei runga ake
Te roa o te text 4,000,000 paita (UTF-8) 413 me te karere text too long
Tinana tono 32 MiB 413
Mō ia tono Kotahi kuputuhi, kotahi kōrero rānei Tukuna tētahi tono mō ia kuputuhi hei tatau i ētahi kuputuhi maha.

Kāore ngā karanga tatau e tatau ki te tepe o te 120 tono ia meneti. Ngā tepe me te toenga

Ngā hapa

Tūnga Momo Karere Āhea
400 invalid_request_error tokenize is available for the hosted open models; unknown model: <model> Te /v1/tokenize me tētahi model ehara i tētahi id open-weight hosted.
400 invalid_request_error count_tokens is available for the hosted open models; unknown model: <model> Te /v1/messages/count_tokens me tētahi model ehara i tētahi id open-weight hosted, kāore rānei he model.
400 invalid_request_error send `text` or `messages` Te /v1/tokenize kāore he text kāore hoki he messages.
401 authentication_error Missing authentication / Invalid API key Kāore he key i tukuna, ehara rānei te key i te mea tika.
413 invalid_request_error text too long Roa ake te text i te 4,000,000 paita. Ka whakautua hoki tētahi tinana neke atu i te 32 MiB ki te 413.
415 invalid_request_error Expected request with `Content-Type: application/json` Kāore he momo content JSON o te tono.
422 invalid_request_error Failed to deserialize the JSON body into the target type: … Kei te ngaro tētahi āpure e hiahiatia ana (model i te /v1/tokenize, messages i te /v1/messages/count_tokens) he hē rānei te momo o tētahi āpure.
503 api_error token counting is temporarily unavailable for this model Kāore e taea te tatau mō tēnei model i tēnei wā. Whakamātauria anō a muri ake nei.

Ka whakahoki te /v1/tokenize i ngā hapa i te āhua OpenAI. I runga i te /v1/messages/count_tokens ka tae mai ngā hapa o te endpoint ake (400 mō te model, 503) i te āhua Anthropic, ā, ka tae mai te 401, 413, 415 me te 422 i te āhua OpenAI. Pānuitia te waehere tūnga i te tuatahi, kātahi te error.type me te error.message, kei ngā āhua e rua.

400 /v1/tokenize
{
  "error": {
    "type": "invalid_request_error",
    "message": "tokenize is available for the hosted open models; unknown model: shannon-3"
  }
}
400 /v1/messages/count_tokens
{
  "type": "error",
  "error": {
    "type": "invalid_request_error",
    "message": "count_tokens is available for the hosted open models; unknown model: shannon-3"
  }
}