MechInterp: TinyStories-1Layer-21M Model Embed, Attention and MLP Analysis (Part 2 - MLP Analysis)

Table of Contents Table of Contents Introduction Resources Summary Introduction This post begins the exploration of mechanistic interpretability for large language models. We start off by attempting to understand the MLP layer from a model from the TinyStories paper. For this blog post, we will be using the roneneldan/TinyStories-Instuct-1Layer-21M model which can be found on HuggingFace. Resources Jupyter Notebook: The Notebook which contains all of the code and visualisations mentioned in t...

Submit