What Is a Token in AI?

Defining the Token: The Fundamental Unit of AI

A token in AI is the basic, discrete unit of data that a neural network processes, analyzes, and generates. In natural language processing (NLP) and large language models (LLMs), neural networks cannot directly interpret human text, characters, or raw words as strings. Instead, incoming text is segmented into smaller numerical components—ranging from whole words and subwords to individual characters or punctuation marks—through a process called tokenization. What are tokens and how to count them? - OpenAI Help Center Understanding tokens - .NET | Microsoft Learn

Once broken into tokens, each unit is mapped to a unique integer ID within the model's fixed vocabulary and subsequently transformed into a dense mathematical vector known as an embedding. Tokens serve as both the fundamental semantic currency during model training and inference and the computational metric used to define context window limits, throughput speed, and API pricing structures across AI platforms. Understanding tokens - .NET | Microsoft Learn What Are AI Tokens? The Language and Currency Powering Modern AI

Raw Text:        "Understanding tokenization is essential."
                    │
Tokenization:    ["Understand", "ing", " token", "ization", " is", " essential", "."]
                    │
Token IDs:       [14821, 292, 11241, 1634, 374, 8491, 13]
                    │
Vector Space:    [[0.024, -0.912, ...], [0.418, 0.119, ...], ...]

The Mechanics of Tokenization

Tokenization sits at the interface between human-readable data and machine learning architectures. The transformation from raw text into numerical representations follows a sequence of deterministic steps before reaching the neural layers of a transformer model. Understanding tokens - .NET | Microsoft Learn

The Text-to-Vector Pipeline

1. Text Normalization and Pre-Tokenization: The raw input string is cleaned and split into preliminary boundaries, such as whitespace or language-specific delimiters. Case folding, Unicode normalization (e.g., NFC/NFKC), and whitespace preservation occur at this phase.
2. Subword Segmentation: The tokenizer applies a statistical model against a pre-compiled vocabulary to split strings into optimal subword pieces.
3. Vocabulary Lookup (Token IDs): Each token string corresponds to an entry in a lookup table. For instance, in a model with a 100,000-token vocabulary, each token maps directly to an index between $0$ and $99,999$.
4. Vector Embedding: The token IDs are passed to the model's embedding matrix. An ID of index $i$ retrieves the $i$-th row of the matrix, creating a high-dimensional vector of dimension $d_{\text{model}}$ (e.g., 4,096 dimensions in many standard models) that encodes the semantic and syntactic properties of the unit. Understanding tokens - .NET | Microsoft Learn

Token Granularity: Words, Characters, and Subwords

Tokenization methods have evolved significantly to balance vocabulary size with semantic expressiveness.

Modern generative architectures rely almost universally on subword-level tokenization to achieve high computational efficiency without sacrificing vocabulary coverage. Tokenization algorithms - Hugging Face

Common Subword Tokenization Algorithms

Modern tokenizers construct their vocabularies using automated statistical algorithms optimized on massive multilingual corpora.

Byte-Pair Encoding (BPE)

Originally a data compression technique, Byte-Pair Encoding iteratively merges the most frequently co-occurring adjacent characters or character sequences in a training corpus until a predefined vocabulary size is reached. Many prominent models, including the GPT family and Llama series, use Byte-level BPE, which operates directly on raw bytes rather than Unicode characters to ensure that any arbitrary byte sequence can be processed without generating out-of-vocabulary tokens. Tokenization algorithms - Hugging Face

WordPiece

Used prominently in models such as BERT, WordPiece initializes with basic characters and incrementally adds subword pairs. Unlike BPE—which merges pairs based strictly on frequency—WordPiece selects merges that maximize the likelihood of the training data according to a probabilistic language model. Tokenization algorithms - Hugging Face

Unigram and SentencePiece

The Unigram algorithm reverses the BPE process: it begins with a vast initial vocabulary of candidate tokens and iteratively removes subwords that contribute the least to overall corpus likelihood until reaching the desired vocabulary threshold.

SentencePiece is an open-source library that treats the entire input string as a continuous sequence of raw characters, treating whitespace as an explicit symbol (e.g., _ or ) rather than relying on language-dependent pre-tokenization rules. It can implement both BPE and Unigram algorithms, making it widely used for multilingual models such as T5 and Gemma. Tokenization algorithms - Hugging Face

Tokens in Practice: Text Ratios and Modalities

Rule-of-Thumb Conversion Ratios

In standard English text processed by modern tokenizers:

• $1 \text{ token} \approx 0.75 \text{ words}$ (or roughly 4 characters of text). What are tokens and how to count them? - OpenAI Help Center
• $1,000 \text{ tokens} \approx 750 \text{ words}$. What are tokens and how to count them? - OpenAI Help Center

However, token density varies substantially across domains and languages:

• Standard English Prose: Highly compressed. Frequent words (e.g., "the", "system", "development") are single tokens.
• Code and Markup: Punctuation-dense languages (e.g., JSON, Python, C++) consume more tokens due to indentation, brackets, variable names, and special symbols.
• Non-Latin Scripts: Languages with rich morphological inflection or non-Latin alphabets (such as Arabic, Hindi, Chinese, or Japanese) often require multiple tokens per word if the tokenizer's training corpus was predominantly English, leading to higher token usage for identical semantic content. What are tokens and how to count them? - OpenAI Help Center

Multimodal Tokens: Vision and Audio

The concept of a token extends beyond text into multimodal architectures:

• Vision Tokens: In Vision Transformers (ViTs), an image is divided into a grid of non-overlapping patches (e.g., $16 \times 16$ pixels). Each patch is flattened and linearly projected into a vector embedding, serving as an image token.
• Audio Tokens: Continuous acoustic waveforms are discretized using neural audio codecs into discrete codebook indices, which are processed sequentially like text tokens.

Operational and Computational Implications

Tokens govern virtually every technical constraint and financial consideration when building and deploying AI systems.

Context Windows and Memory Complexity

Large language models operate within a strict context window (e.g., 8,192, 32,768, or 1,000,000+ tokens), representing the maximum combined sequence length of input prompts and output responses the model can process at one time.

In standard transformer attention mechanisms, the computational complexity and memory footprint of the attention layer scale quadratically with sequence length:

where $N$ is the number of tokens in the context sequence. Longer token sequences require substantially higher GPU VRAM during processing and key-value (KV) cache allocation.

API Metering and Cost

Commercial model providers bill developer usage based on token volume, separating pricing into:

• Input (Prompt) Tokens: Ingested and processed in parallel by the model's prefill phase.
• Output (Completion) Tokens: Generated autoregressively one token at a time, requiring significantly higher computational time and memory bandwidth. What are tokens and how to count them? - OpenAI Help Center What Are AI Tokens? The Language and Currency Powering Modern AI

The Basic Unit AI Models Actually Read

A token in AI is the smallest chunk of data a model reads or writes. For language models, it is a fragment of text — sometimes a whole word, sometimes part of a word, sometimes just a punctuation mark or a space. Before a model can process the sentence "Tokenization is unavoidable," a component called a tokenizer splits it into pieces, converts each piece into an integer ID, and hands the model a list of numbers rather than raw characters. Everything the model does afterwards — attention, prediction, generation — operates on those units. What are AI tokens? Definition, counting & cost | Decagon Understanding tokens - .NET | Microsoft Learn

This makes tokens simultaneously three things: the model's alphabet, the unit in which its memory limits are measured, and the unit in which commercial APIs bill you. Understanding them explains a surprising amount of otherwise mysterious AI behaviour, from why a chatbot miscounts letters to why the same prompt costs more in Japanese than in English. What Are AI Tokens? The Language and Currency Powering Modern AI

Why not just use words or letters?

The obvious alternatives both fail in instructive ways.

If a model's vocabulary were a list of whole words, it would need an entry for every word it might ever see. Human vocabularies are effectively open-ended: names, typos, URLs, product codes, inflected forms, and newly coined words appear constantly. Any fixed word list eventually meets something it cannot represent and must fall back on an "unknown" placeholder, destroying information. Word vocabularies are also brittle across languages that don't separate words with spaces.

If the vocabulary were individual characters, nothing would ever be unknown, but sequences become extremely long. A 500-word paragraph becomes roughly 3,000 units instead of 700, and because transformer attention cost grows with sequence length, this wastes compute while forcing the model to learn spelling patterns from scratch before it can learn meaning.

Subword tokenization is the compromise that won. Frequent words get their own single token; rare or novel words are decomposed into recognizable pieces. unbelievable might become un + believ + able; a rare surname might break into three or four fragments. Nothing is truly out-of-vocabulary, and common text stays compact. How tokenizers work in AI models: A beginner-friendly guide - Nebius What Is a Token in AI? An Explainer - Couchbase

How tokenizers are built

The dominant method is byte pair encoding (BPE). It began life as a data-compression algorithm described by Philip Gage in 1994 and was adapted to neural machine translation by Sennrich, Haddow and Birch, who used it to give translation systems open-vocabulary behaviour with a compact symbol set. That paper is the reason most large language models today ship with a BPE-style tokenizer. Neural Machine Translation of Rare Words with Subword ... Byte Pair Encoding: Subword Tokenization - Interactive

The training procedure is simple to describe:

1. Start with a vocabulary of individual characters (or raw bytes).
2. Count all adjacent symbol pairs in a large text corpus.
3. Merge the most frequent pair into a new single symbol and record the merge rule.
4. Repeat until the vocabulary reaches a target size.

The result is an ordered list of merge rules that can be replayed deterministically on any new string. Related families include WordPiece (used by BERT-style models, choosing merges by likelihood rather than raw frequency), SentencePiece (which trains directly on raw text without pre-splitting on whitespace, useful for Chinese, Japanese and Thai), and byte-level BPE (which starts from the 256 possible bytes so that any Unicode input, including emoji and corrupted text, is representable).

Vocabulary size is a design trade-off. Larger vocabularies mean shorter sequences and cheaper inference per unit of text, but a bigger embedding table and more parameters spent on rare entries. Modern models commonly use vocabularies in the range of tens of thousands to a few hundred thousand entries, and the exact figure differs by model family.

Tokenizers also reserve special tokens that carry structural meaning rather than text: sequence start and end markers, padding, and — in chat models — delimiters that mark where the system instruction ends and the user turn begins. These are part of why a chat API's token count is slightly higher than a naive count of your visible message.

Counting tokens in practice

There is no exact conversion between tokens and words, because it depends on the tokenizer and the text. For ordinary English, widely quoted rules of thumb are that one token is roughly four characters, roughly three-quarters of a word, so about 100 tokens per 75 words. What are tokens and how to count them? - OpenAI Help Center

Those averages degrade quickly outside typical English prose:

• Other languages. Text in languages under-represented in the tokenizer's training corpus fragments into more pieces. The same meaning can consume two to several times as many tokens, which raises both cost and effective context consumption.
• Code. Indentation, brackets, and identifiers such as getUserById split into multiple tokens; code generally has a lower character-per-token ratio than prose.
• Numbers. Long numerals are often chopped into arbitrary digit groups, which is one reason arithmetic is unreliable without tool use.
• Whitespace and formatting. Leading spaces are typically attached to the following word, so " cat" and "cat" are usually different tokens.

For anything cost- or limit-sensitive, count tokens with the model's own tokenizer library rather than estimating from word counts. Character-based estimates are fine for rough capacity planning and misleading for tight budgets.

From tokens to predictions

Once text is tokenized, each integer ID indexes into an embedding table, retrieving a learned vector. The transformer then processes the whole sequence of vectors and produces, at the final position, a probability distribution over the entire vocabulary — a score for every possible next token. A sampling step (greedy, temperature, top-p, and so on) selects one. That token is appended to the sequence, and the process repeats.

This is the mechanical meaning of the phrase "large language models predict the next token." Generation is inherently sequential and token-by-token, which is why:

• Streaming responses appear word-fragment by word-fragment.
• Output length is capped by a maximum-tokens parameter, and hitting that cap truncates mid-sentence rather than producing a shorter, complete answer.
• Latency has two distinct components: time to process the prompt (prefill) and time per generated token (decode).

Tokens as the currency of context and cost

Every limit and price tag in a modern model API is denominated in tokens. The context window is the maximum number of tokens the model can attend to at once, counting the system prompt, conversation history, retrieved documents, tool definitions, tool results, and the response being generated. Contemporary flagship models advertise windows ranging from tens of thousands to a million or more tokens, though large windows do not guarantee uniform recall across the whole span. Long context | Gemini API - Google AI for Developers

Billing follows the same unit. Providers typically quote a price per million tokens and charge input (prompt) tokens and output (completion) tokens at different rates, with output usually more expensive because it requires a full forward pass per token. Several providers add further categories: discounted cached input tokens for repeated prefixes, and separately metered reasoning or thinking tokens that a model generates internally before its visible answer. Rates, tiers and surcharges change frequently and vary by provider, model and plan, so treat any specific figure as something to verify against current documentation. What are tokens and how to count them? - OpenAI Help Center Pricing - Claude Platform Docs

The practical consequence is that token accounting is engineering work. Long chat histories re-send the entire conversation on every turn, so cost grows roughly quadratically over a session unless you summarize, truncate, or cache. Verbose system prompts and bloated retrieved documents are paid for repeatedly.

Tokens beyond text

The concept generalizes. In vision transformers, an image is cut into fixed-size patches, each linearly projected into a vector and treated exactly like a text token; a 224×224 image split into 16×16 patches yields 196 tokens. Audio is commonly converted into spectrogram patches or into discrete codes produced by a learned audio codec, and video adds a temporal dimension on top of spatial patches. Multimodal models feed these heterogeneous token streams into a shared sequence so that attention can relate a phrase to a region of an image. Learning to tokenize in Vision Transformers How LLMs See Images, Audio, and More

This is why image and audio inputs consume context budget and appear on invoices in token units, and why a high-resolution image can cost more than a page of text.

Where tokenization leaks into behaviour

Because the model never sees letters directly, character-level tasks are structurally hard. The classic demonstration is asking how many times r appears in strawberry: if the word is tokenized as something like straw + berry, the model has no direct view of the individual characters and must rely on memorized associations. Research on this failure mode finds models handle letters that appear once within a token far better than letters repeated inside a token, which matches the tokenization explanation rather than a general reasoning deficit. Why Do Large Language Models (LLMs) Struggle to Count Letters? Why LLMs Can't Spell 'Strawberry' And Other Odd Use Cases - Runpod

Related artefacts include unreliable reversal of strings, inconsistent rhyming and syllable counting, digit-grouping errors in arithmetic, and so-called glitch tokens — rare vocabulary entries that were barely trained and can trigger strange outputs. Some are mitigated by prompting the model to work character-by-character or by delegating to a tool; none disappear entirely, because they follow from the representation itself.

A note on the word "token"

"Token" is heavily overloaded in computing, and the AI sense is unrelated to several neighbours you may encounter in the same search results:

• AI/LLM token — a unit of text or other data in a model's vocabulary (this article).
• Crypto token — a unit of value on a blockchain.
• Authentication token — a credential such as an API key or JWT used to authorize requests. Confusingly, you use an auth token to spend AI tokens.
• Compiler token — a lexical unit produced by a programming-language lexer, which is a close conceptual cousin: same idea of segmenting a stream into meaningful units, different vocabulary construction.

Keeping these separate matters when reading documentation, where a single page may mention token limits, token pricing, and bearer tokens within a few paragraphs.