Use the built-in GELU, don't roll your own!

This blog post describes a personal discovery in using PyTorch's built-in GELU function for model training, revealing that it can significantly outperform a hand-rolled version, increasing processing speed by about 20%. The author shares insights on their training experiences with GPT-2 models, comparing performance metrics between different implementations of the GELU function and emphasizing the benefits of using built-in libraries when optimizing model performance.