Bytes Are Not Big Enough

In this blog post, the author discusses their experience building a small language model, focusing on the challenges of tokenization and the quest for an ideal token size. Initially, the model emitted coherent text, but it lacked practical utility. The author explores the significance of finding a balance between token size and model performance, ultimately determining that an average of 3.47 characters per token strikes a good balance for achieving both nuance in understanding text and efficiency in processing speed.