Tokenizers in Language Models - A Practical Comparison

Introduction

A tokenizer is one of the most important, yet often overlooked, components of every large language model. Its choice has a direct impact on model performance, output quality, and text processing efficiency.
In this article:
- We'll look at the practical behavior of different tokenizers
- We'll compare their behavior on multilingual texts, emoji, and code
- We'll build our own simple tokenizer from scratch

This material was developed based on knowledge gained from a short, free course available on the deeplearning.ai platform: How Transformer LLMs Work
Source code: Google Colab - Tokenizer Comparison

Tokenizer - the bridge between humans and the model

A tokenizer serves as the entry point to every large language model. You could say it acts as a bridge between humans and the model, because the model doesn't operate directly on words or letters, but on tokens. In practice, it's often simplified that word = token, but in reality one word can consist of many tokens.
Every LLM has its own token vocabulary - each token has a unique ID. The tokenizer's job is to convert text into a sequence of tokens and pass the list of their IDs so the model can do its work.
In this post I'll walk through the behavior of different tokenizers in practice and observe the differences between them, without going into technical details. We'll use the Hugging Face API for this purpose.

Practical demonstration

To convert text into tokens, just a few lines of code are needed:

from transformers import AutoTokenizer
sentence = "Hello world!"
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
tokens = tokenizer(sentence)

AutoTokenizer is one of the Hugging Face API classes that provides thousands of open-source models. Based on the given model name (in this case bert-base-cased) it automatically:

1. Downloads the appropriate tokenizer from the Hugging Face repository
2. Loads its configuration and vocabulary
3. Saves it locally in cache for future use

This means we don't have to manually check which specific tokenizer to use - AutoTokenizer recognizes the model type and loads the correct implementation.

BatchEncoding object structure

The returned tokens object is an instance of the BatchEncoding class, which implements a dictionary interface and contains the following components:

print(tokens)
# Output:
# {'input_ids': [101, 8667, 1362, 106, 102], 
#  'token_type_ids': [0, 0, 0, 0, 0], 
#  'attention_mask': [1, 1, 1, 1, 1]}

Decoding tokens

To decode token IDs back to actual words, simply use the decode function:

for id in token_ids:
    print(tokenizer.decode(id))

[CLS]
Hello
world
!
[SEP]

In the decoded tokens, beyond the words we can see so-called special tokens, which have the following meanings:

• [CLS] (classification) - sequence-initializing token, used in classification tasks
• [SEP] (separator) - delimiter that segments or terminates the sequence
• [UNK] (unknown) - representation of tokens absent from the vocabulary
• [PAD] (padding) - equalizes sequence lengths in batches

The above example demonstrates the operations performed by every LLM when handling our queries. First, the input prompt is converted into tokens, then the model processes those tokens, and finally they are decoded back into text for the user to read.

Tokenizer comparison

To systematically analyze the differences in tokenizer implementations, a test text was prepared containing challenges characteristic of natural language processing:

• English texts with varied capitalization
• Emoticons and Unicode symbols (🎵 🥸 鸟)
• Source code fragments with logical operators
• Whitespace sequences (tabs, spaces)
• Numerical and mathematical expressions
• Polish text with diacritic characters

text = """
English and CAPITALIZATION
🎵 🥸  鸟
show_tokens False None elif == >= else: two tabs:"    " Three tabs: "       "
12.0*50=600
Przykładowe zdanie w języku polskim, żółć
"""

Comparison results

BERT base-cased

Characteristics: BERT model with case preservation, vocabulary: 28,996 tokens

Vocab length: 28996
[CLS] English and CA ##PI ##TA ##L ##I ##Z ##AT ##ION [UNK] [UNK] [UNK] show _ token ##s F ##als ##e None el ##if = = > = else : two ta ##bs : " " Three ta ##bs : " " 12 . 0 * 50 = 600 P ##rz ##yk ##ła ##do ##we z ##dan ##ie w j ##ę ##zy ##ku p ##ols ##kim , ż ##ó ##ł ##ć [SEP]

Observations:
- Use of the ## prefix to mark sub-tokens (WordPiece)
- Polysyllabic words were split into numerous tokens
- No emoji support → [UNK] tokens
- Handles Polish diacritic characters but splits them into separate tokens

BERT base-uncased

Characteristics: BERT variant with lowercasing normalization, vocabulary: 30,522 tokens

Vocab length: 30522
[CLS] english and capital ##ization [UNK] [UNK] [UNK] show _ token ##s false none eli ##f = = > = else : two tab ##s : " " three tab ##s : " " 12 . 0 * 50 = 600 pr ##zy ##k ##ła ##do ##we z ##dan ##ie w je ##zy ##ku pol ##ski ##m , z ##o ##ł ##c [SEP]

Observations:
- Complete loss of capitalization information
- Slightly larger vocabulary than the cased version
- Similar issues with special character representation, plus loss of some information (ż -> z)

Xenova/gpt-4

Characteristics: GPT-4 tokenizer implementation, vocabulary: 100,263 tokens

Vocab length: 100263

 English  and  CAPITAL IZATION 
 � � �  � � �    � � � 
 show _tokens  False  None  elif  ==  >=  else :  two  tabs :"      "  Three  tabs :  "         "
 12 . 0 * 50 = 600 
 Pr zy k ł adow e  zd anie  w  j ę zy ku  pol sk im ,  ż ół ć 

Observations:
- Significantly larger vocabulary enables more efficient tokenization with fewer tokens for polysyllabic words
- Better handling of whitespace and code structure
- Moderate support for Polish, still splits Polish words into many tokens
- Problematic emoji representation

gpt2

Characteristics: Classic GPT-2 tokenizer (BPE), vocabulary: 50,257 tokens

Vocab length: 50257

 English  and  CAP ITAL IZ ATION 
 � � �  � � �    � � � 
 show _ t ok ens  False  None  el if  ==  >=  else :  two  tabs :"        "  Three  tabs :  "              " 
 12 . 0 * 50 = 600 
 Pr zyk ł adow e  z dan ie  w  j � � zy ku  pol sk im ,  � � ó ł ć 

Observations:
- Significant degradation of Unicode character representation
- Imprecise handling of whitespace sequences
- No support for some Polish characters

google/flan-t5-small

Characteristics: Compact T5 (Text-to-Text Transfer Transformer) model with instruction fine-tuning, vocabulary: 32,100 tokens

Vocab length: 32100
English and CA PI TAL IZ ATION  <unk>  <unk>  <unk> show _ to ken s Fal s e None  e l if = = > = else : two tab s : " " Three tab s : " " 12. 0 * 50 = 600 Pr zy k <unk> a dow e  z d ani e  w  j <unk> zy ku  pol s kim ,  <unk> ó <unk>  </s>

Observations:
- </s> token as end-of-sequence marker (characteristic of T5)
- <unk> for characters outside the vocabulary
- Limited efficiency for multilingual texts

BigCode StarCoder2-15B

Characteristics: Specialized model for code generation, vocabulary: 49,152 tokens

Vocab length: 49152

 English  and  CAPITAL IZATION 
 � � �  � � �     � � 
 show _ tokens  False  None  elif  ==  >=  else :  two  tabs :"      "  Three  tabs :  "         " 
 1 2 . 0 * 5 0 = 6 0 0 
 Pr zy k ł adow e  z d anie  w  j ę zy ku  pol sk im ,  ż ó ł ć 

Observations:
- Precise handling of programming syntax (operators, keywords)
- Atomization of digits in numerical expressions
- Reasonable representation of Polish diacritic characters
- Still problematic emoji handling

xlm-roberta-large

Characteristics: Multilingual Transformer model, vocabulary: 250,002 tokens

Vocab length: 250002
<s> English and CAP ITA LIZA TION  🎵  <unk>  鸟 show _ tok ens Fal se No ne el if  == > = else : two tab s : " " Three tab s : " " 1 2.0 * 50 = 600 Przy kład owe z danie w język u polskim ,  żół ć </s> 

Observations:
- Best Polish language support among all tested models, most likely due to the largest vocabulary
- Recognition of the musical emoji 🎵 and Chinese character 鸟
- Minimal fragmentation of Polish words
- <s> and </s> tokens at the beginning and end of the sequence

Key observations

What does this mean in practice?

The choice of the right tokenizer should depend on the specific use case:

• For English texts:
  • Most tokenizers will provide good results
  • GPT-4 and XLM-RoBERTa offer the best efficiency
• For code generation:
  • StarCoder - dedicated, precise in syntax handling
  • GPT-4 - universal, also works well with code
• For multilingual texts (including Polish):
  • XLM-RoBERTa - unrivaled leader
  • English-language models (BERT, GPT-2) may significantly fragment text
• For emoji and Unicode:
  • Newer models (XLM-RoBERTa, GPT-4, Qwen)
  • Avoid older tokenizers (GPT-2, early BERT)

Building your own tokenizer

This example of a custom tokenizer comes from Andrej Karpathy's course: Building makemore

We can also create our own tokenizer. It won't be as advanced as the ones discussed earlier, but it's great for educational purposes. We'll build the simplest possible tokenizer where the tokens are individual characters.
We'll start with the text we want to tokenize. We'll use a publicly available dataset containing all of Shakespeare's texts: Tiny Shakespeare
After loading the entire dataset, simply convert it to a set to get the collection of unique characters. Then convert back to a list and sort:

chars = sorted(list(set(text)))
vocab_size = len(chars)
print(''.join(chars))
print(vocab_size)

 !$&',-.3:;?ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
65

Our vocabulary contains 65 tokens - all letters of the alphabet and some special characters.
Next we need functions to encode text into tokens and decode tokens back into text. For this we create two mapping dictionaries:

• stoi (string to integer) - characters → numbers
• itos (integer to string) - numbers → characters

# Character to number mapping and vice versa
stoi = { ch:i for i,ch in enumerate(chars) }
itos = { i:ch for i,ch in enumerate(chars) }

# Encoding and decoding functions
encode = lambda s: [stoi[c] for c in s]  # text → list of numbers
decode = lambda l: ''.join([itos[i] for i in l])  # list of numbers → text

print(encode("hii there"))
print(decode(encode("hii there")))

[46, 47, 47, 1, 58, 46, 43, 56, 43]
hii there

And that's it! Such a simple tokenizer certainly won't allow building an advanced LLM, but you can use it to build a simple transformer and observe how the attention mechanism works. More on that in future posts.

Summary

A tokenizer is an often underappreciated but crucial element of every LLM. As the comparisons above show, the differences between tokenizers can be significant - especially when working with languages other than English, special characters, or source code.
The choice of tokenizer has a direct impact on:
- Efficiency - fewer tokens = faster processing and lower API costs
- Quality - better representation = better context understanding by the model
- Universality - support for different languages and text formats

It's worth experimenting with different models and tokenizers to find the optimal solution for your use case.

Fun fact: The 🥸 emoji (face with mustache and glasses) is relatively new (Unicode 13.0, 2020), which is why none of the tested tokenizers recognized it correctly - most models were trained earlier and don't have this character in their vocabulary.

Useful links

• Hugging Face Tokenizers
• Tokenizer Arena - interactive comparison
• Course: How Transformer LLMs Work
• OpenAI Tokenizer