Exploring Text Classification Models with Sparse Autoencoders

Authors

Daniel Kerrigan (Capital One), Brian Barr (Capital One), Enrico Bertini (Northeastern University)

Presentation

Session: Is the Model Even Thinking?
Time: Thursday, Nov 12, 08:27 – 08:36 (US/Eastern) · session 08:00 – 09:30
Location: Hall America south

Keywords

Sparse autoencoder, dictionary learning, mechanistic interpretability, text classification, visual analytics.

Abstract

As language models (LMs) rise in prominence, there is interest in making them more transparent in order to better understand their internal behavior. Recent interpretability work has focused on using sparse autoencoders (SAEs) to break down neuron activations at a given layer in the LM into human-understandable features, where each feature represents a concept that the model has learned. In this paper, we share work on using SAEs to analyze the behavior of text classification LMs. We present techniques for exploring the relationships between the SAE's features and the model's predictions and errors. We integrate these techniques into SAEfarer, a tool for analyzing concepts learned by text classification LMs. We assess SAEfarer in an expert pilot evaluation with five Ph.D. students.

For Practitioners

Data scientists, machine learning engineers, and other ML practitioners who want to explore the behavior of transformer-based classification models may be interested in reading this paper. We provide an open-source tool that can serve as a starting point for practitioners that want to apply the techniques in the paper to their own work.