A quick(ish) Chinchilla check

This blog post details an experiment assessing the Chinchilla heuristic of training large language models. The author compares the performance of GPT-2 style models trained using different token counts and parameter scaling. Results indicate that while adhering to the Chinchilla-optimal training strategy generally yielded better models, the improvements were marginal and may fall within noise variance. The post provides insights into model training complexities and offers personal reflections on scalability challenges in AI modeling.