LLM Token Counter

This page is the canonical guide to counting LLM tokens in the browser with DevUtilities. Use the interactive Prompt Budget Cockpit above while you read, or jump to a topic below. Everything runs client-side via js-tiktoken — prompts never leave your machine.

The LLM Token Counter (Prompt Budget Cockpit) tokenizes text with the same BPE tables OpenAI ships in tiktoken. It compares three encodings side-by-side, visualizes token boundaries, estimates input cost for GPT-4o, GPT-4o mini, and Claude 3.5 Sonnet, and helps you shrink prompts before they hit a production API.

What you get

• Concurrent counts for o200k_base (GPT-4o), cl100k_base (GPT-4 / Claude approximation), and p50k_base (legacy Codex)
• Color-coded Token Boundary Map with hoverable Token IDs
• Minify JSON and Strip Spaces actions to reduce prompt weight
• Per-query and monthly cost extrapolations from static published input rates
• Copyable multi-encoding metrics for design docs and budget reviews

Use this tool when

• You need to know whether a system prompt + tools + user message fits a context window
• You are comparing GPT-4o vs Claude-ish token budgets before picking a model
• You want a local, privacy-safe count without calling OpenAI or Anthropic APIs
• You are trimming RAG chunks, few-shot examples, or JSON schemas that inflate cost

Do not expect

• Byte-identical Claude billing — Anthropic uses a different production tokenizer; cl100k_base is a useful English approximation
• Chat wrapper overhead (role tags, tool envelopes) that APIs add around raw text
• Live market prices — the cost panel uses static published rates that can drift

Token IDs are encoding-specific. The same English sentence can produce different lengths and different IDs under o200k_base vs cl100k_base. Always budget with the encoding your target model actually uses.

Switch the Token Boundary Map encoding independently of the comparison cards when you need to inspect merges for a specific model family.

Follow this walkthrough the first time you open the cockpit. After that, most sessions are paste → compare → optimize → copy.

1. Paste the full prompt payload you care about: system message, user turn, tool JSON, or a concatenated RAG context block.
2. Read the three comparison cards. Note which encoding is highest — that is your conservative budget if you still need to pick a model.
3. Open the Token Boundary Map and hover spans that look surprisingly long (URLs, emoji, indented JSON) to see concrete Token IDs.
4. If the input is JSON-heavy, click Minify JSON. If it is prose with double spaces or padded newlines, click Strip Spaces.
5. Enter your estimated monthly call volume to project GPT-4o, GPT-4o mini, and Claude 3.5 Sonnet input spend.
6. Copy the metrics summary into a design doc, ticket, or PR description so the team shares one number.

What “good” looks like

• System + tools + average user turn leave headroom for the model’s reply inside the chosen context window
• JSON schemas are minified in production prompts even if humans edit pretty-printed copies
• Cost extrapolations use the encoding that matches the billed model

Problem: a support agent prompt keeps truncating mid-conversation after you add policies, tone rules, and three few-shot examples.

How this tool solves it

1. Paste the current system prompt alone and record the o200k_base (or cl100k_base) count.
2. Append each few-shot example one at a time and note the delta — drop the least valuable example first.
3. Paste a representative user message and the tools JSON you send on every call; sum the pieces against your model’s context window.
4. Use Strip Spaces on prose and Minify JSON on schemas, then re-count to quantify savings.
5. Copy the final budget into your agent runbook so future prompt edits stay under the same ceiling.

Outcome: you know exactly which section blew the budget and how many tokens each optimization recovered — before another production truncation.

Problem: OpenAI function-calling schemas are authored with indentation for readability, then pasted into live requests. Token spend climbs even though the logical schema did not change.

How this tool solves it

1. Paste the pretty-printed tools array or parameters object into the input panel.
2. Note the token count on the encoding your API model uses (usually o200k_base for GPT-4o).
3. Click Minify JSON and compare the new count — the difference is pure whitespace tax.
4. Inspect the boundary map: repeated keys and long enum lists often dominate; shorten descriptions before cutting fields.
5. Ship the minified schema in API calls; keep the pretty copy only in source control or the Agent Builder editor.

Outcome: lower input cost on every tool-enabled call without changing tool behavior.

Problem: finance asks for a monthly LLM spend estimate before you launch a feature that will call the model thousands of times.

How this tool solves it

1. Build a realistic average payload (system + tools + typical user text) and paste it here.
2. Confirm the comparison card for the encoding that matches your production model.
3. Set Estimated monthly volume to your projected call count.
4. Read the GPT-4o, GPT-4o mini, and Claude 3.5 Sonnet monthly columns to compare vendors on the same payload.
5. Document that rates are static published input prices — refresh the estimate when vendors change pricing pages.

Outcome: a defensible, payload-specific forecast instead of a vague “tokens are cheap” guess.

Problem: a RAG pipeline retrieves five chunks per query. Some chunks are mostly whitespace or boilerplate headers, and the model still misses the answer because useful text was truncated.

How this tool solves it

1. Paste each candidate chunk separately and record token counts under your retrieval model’s encoding.
2. Prefer denser chunks: Strip Spaces, remove repeated navigation headers, and drop near-duplicate paragraphs.
3. Concatenate the top-k set you plan to send and verify the total still leaves room for the question and the answer.
4. Use the boundary map to spot tokenizer-expensive sequences (long URLs, tables, emoji) that look short to humans.
5. Set a hard token budget per query in your retriever using the measured totals from this tool.

Outcome: retrieval packs more signal per token and fails less often from silent context truncation.

Symptoms: API 400 context_length_exceeded, truncated completions, or agents that “forget” early instructions mid-session.

Why it happens

The combined system prompt, tools, history, and user turn exceed the model’s context window. Pretty-printed JSON and long few-shot banks are common silent offenders.

Diagnose

Paste the exact payload your client sends (or reconstruct it). Compare the token total to the context window listed for your model. Identify the largest sections by counting them in isolation.

Fixes

1. Minify JSON tool schemas and strip excess whitespace from prose.
2. Move rarely used policies into on-demand retrieval instead of the always-on system prompt.
3. Cap chat history (summarize or drop oldest turns) before each call.
4. Switch to a larger-context model only after measuring — do not guess.

Re-count after every change. A 10% whitespace cut on a 8k-token tools blob often recovers more headroom than deleting a short policy paragraph.

Symptoms: local counts look fine, but OpenAI usage dashboards disagree — or Claude bills differently than your spreadsheet.

Why it happens

You budgeted with the wrong encoding, or you compared raw text to an API that wraps messages with role and tool envelopes.

Diagnose

Confirm the model’s documented tokenizer. For GPT-4o family use o200k_base; for GPT-4 / 3.5 use cl100k_base. Treat Claude figures here as approximations only.

Fixes

1. Switch the comparison focus and boundary map to the matching encoding.
2. Count the serialized messages your SDK actually sends, not just the human-readable prompt draft.
3. Add a small overhead buffer (often a few dozen to a few hundred tokens) for chat formatting when planning hard limits.
4. For Anthropic billing, validate critical prompts against Anthropic’s own counter before locking SLAs.

Symptoms: token count drops dramatically after Minify JSON or Strip Spaces with no semantic change to the prompt.

Why it happens

BPE encodings charge for indentation newlines, alignment spaces, and repeated blank lines. Humans ignore them; tokenizers do not.

Diagnose

Paste the pretty payload, note the count, run Minify JSON / Strip Spaces, and compare. Large deltas mean whitespace was a primary cost driver.

Fixes

1. Store human-editable pretty JSON in git; minify at request time.
2. Avoid padding ASCII diagrams or huge indented logs inside system prompts.
3. Collapse multi-blank-line section separators in long policy docs.

The boundary map makes whitespace tokens visible as separate colored spans — useful when teaching teammates why minify matters.

The Token Boundary Map paints each BPE merge as a colored span. Colors cycle for visual separation only — they do not encode token categories.

How to read it

• Hover a span to reveal its numeric Token ID for the selected map encoding
• Long unbroken spans on URLs or base64 often mean expensive single-token or few-token chunks worth shortening
• Emoji and CJK text may use more tokens than Latin prose of the same visual length
• Change map encoding when comparing how GPT-4o vs GPT-4 would segment the same string

Cost cards multiply local token counts by static published input rates: GPT-4o $5 / 1M, GPT-4o mini $0.15 / 1M, Claude 3.5 Sonnet $3 / 1M. Monthly volume scales the per-query cost.

Caveats

• Output tokens are not included — add completion estimates separately for full TCO
• Cached-input or batch discounts are not modeled
• Vendor list prices change; treat figures as planning aids, not invoices

• Budget with the encoding that matches production, then keep a 10–20% safety margin for chat wrappers and retries
• Count the full request shape (system + tools + history + user), not only the system prompt
• Minify machine-consumed JSON; keep pretty copies for humans
• Re-run counts whenever you add few-shot examples, MCP configs, or long policy appendices
• Use monthly volume projections in design reviews so cost is visible before launch
• Prefer local counting for proprietary prompts — this tool never uploads your text

Boundary colors map to tiktoken byte-pair encodings. Special tokens like <|endoftext|> are counted when present as literal text.