👩💻 Join our community of thousands of amazing developers!
The other day at work I noticed a slowdown in runtime between a model with random weights compared to tuned ones. It turned out to be due to denormal numbers computation on the cpu being much slower than the normal arithmetic. Denormal numbers are very low magnitude floats, treated differently to keep precision. For deep learning, those are largely below significance and can be flushed to zero without accuracy loss. To do so for example in PyTorch, either set the appropriate flag: torch....