How an LLM becomes more coherent as we train it

This blog post discusses the training process of a GPT-2-style language model and how its output improves over time. The author shares their personal experiences, detailing checkpoints of the training run and showcasing generated outputs that range from nonsensical to relatively coherent text. The post highlights differences between RNNs and modern transformers-based models in terms of learning structure, provides insights into the model's development, and emphasizes the importance of refining training to produce accurate and meaningful text.