Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

The blog post introduces Gated DeltaNet-2, a novel approach to linear attention in neural networks that enhances the process of memory management by separating the functions of erasing and writing data. This architecture, tested with 1.3 billion parameters on extensive datasets, demonstrates superior performance in language modeling and retrieval tasks, particularly in long-context scenarios.