Writing an LLM from scratch, part 32j -- Interventions: trying to train a better model in the cloud

In this blog post, the author details their ongoing efforts to improve a locally trained GPT-2-style language model through various interventions. After experimenting with methods like removing dropout, gradient clipping, adjusting weight decay, and updating the learning rate, they find that certain combinations lead to incremental improvements in loss, though the ultimate performance still falls short of the original GPT-2 weights. The post thoroughly documents the experiments, outcomes, and reflections on the impact of different interventions, showcasing a hands-on approach to training language models in the cloud.