Marcin Piotrowski
software engineer
☰ Menu

Tokenizers in Language Models - A Practical Comparison

NLP tokenization transformers LLM BERT GPT

A practical guide to tokenizers in large language models. Comparison of BERT, GPT-4, GPT-2, T5, StarCoder and XLM-RoBERTa with multilingual examples, plus building your own tokenizer from scratch.

Read more →