FIND: A Function Description Benchmark for Evaluating Interpretability Methods (2309.03886v3)

Abstract: Labeling neural network submodules with human-legible descriptions is useful for many downstream tasks: such descriptions can surface failures, guide interventions, and perhaps even explain important model behaviors. To date, most mechanistic descriptions of trained networks have involved small models, narrowly delimited phenomena, and large amounts of human labor. Labeling all human-interpretable sub-computations in models of increasing size and complexity will almost certainly require tools that can generate and validate descriptions automatically. Recently, techniques that use learned models in-the-loop for labeling have begun to gain traction, but methods for evaluating their efficacy are limited and ad-hoc. How should we validate and compare open-ended labeling tools? This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating the building blocks of automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate. The functions span textual and numeric domains, and involve a range of real-world complexities. We evaluate methods that use pretrained LMs to produce descriptions of function behavior in natural language and code. Additionally, we introduce a new interactive method in which an Automated Interpretability Agent (AIA) generates function descriptions. We find that an AIA, built from an LM with black-box access to functions, can infer function structure, acting as a scientist by forming hypotheses, proposing experiments, and updating descriptions in light of new data. However, AIA descriptions tend to capture global function behavior and miss local details. These results suggest that FIND will be useful for evaluating more sophisticated interpretability methods before they are applied to real-world models.

• The paper introduces FIND, a benchmark suite that leverages LLMs to generate function descriptions for evaluating interpretability methods.
• It evaluates numeric, string, and synthetic neural functions using code replication and language-based unit tests to assess method robustness.
• Results show GPT-4’s superior performance and underscore the need for interactive interpretability agents to enhance AI model explanations.

Overview of the FIND Benchmark for Evaluating Interpretability Methods

The paper, "FIND: A Function Description Benchmark for Evaluating Interpretability Methods," introduces an innovative benchmark suite, FIND, designed to assess the efficacy of automated interpretability methods for neural networks. The motivation behind FIND arises from the escalating complexity and size of AI models, necessitating scalable and efficient interpretability techniques. FIND capitalizes on the proliferation of LLMs to autonomously generate descriptions of function behaviors, a critical capability as AI systems become intricate and less interpretable to humans.

FIND encompasses a diverse array of procedurally generated functions, emulating components akin to those in neural networks across textual and numeric domains. These functions are deliberately constructed to challenge interpretability methods with inherent complexities like noise, approximation, and domain corruption.

Description of FIND Components

1. Functions with Numeric Inputs: FIND incorporates a suite of 1000 numeric functions, with 85% being parameterized atomic functions and the remainder being compositions. Atomic functions include mathematical operations and neural activations such as ReLU. Composition functions combine operations and require evaluators to decipher composite behaviors. Additionally, some functions introduce noise or domain corruption, testing the robustness of interpretability methods against these challenges.
2. String Manipulation Functions: Featuring 1000 string functions, this component of FIND explores the capabilities required to reverse-engineer symbolic text operations. Atomic functions encompass common string manipulations, while compositions introduce layered operations. These functions are evaluated both by how well the \verb|interpreter| replicates function behavior in code and by descriptive accuracy.
3. Synthetic Neural Modules: Leveraging the LLM Vicuna-13B, synthetic neural modules function as black-box systems to process word-level inputs. They implement both entity recognition and factual relations, requiring evaluators to infer semantic similarities and factual mappings. FIND includes 140 entity functions and 75 relation functions, challenging interpretability methods with real-world concepts.

Evaluation Protocol

FIND's evaluation protocol measures the accuracy of function descriptions via code and language. Numeric and string function evaluations focus on code-based replication accuracy, whereas language-based evaluations employ unit testing. The unit testing method uses a fine-tuned Vicuna evaluator that assesses the match between descriptions and function behavior, allowing for an adaptable approach to function interpretation.

Baseline Interpretability Methods

FIND evaluates several interpretability methods, including:

• Non-interactive approaches, akin to the milan paradigm, provide predetermined data for LLMs to describe, lacking interactivity.
• Automated Interpretability Agents (aia), encouraging LLMs to generate hypotheses, conduct experiments, and evolve their understanding through interaction.
• Hybrid methods combining pre-selected data (milan) with interactive exploration (aia).

These methods utilize prominent LLMs such as GPT-4 and Llama-2, each demonstrating varied success rates in unraveling function complexities. Notably, GPT-4 exhibited superior performance across evaluation metrics, underscoring its potential as a foundational component in future interpretability toolkits.

Implications and Future Directions

FIND provides a structured environment for developing and benchmarking advanced interpretability methods. The benchmark offers insights into an \verb|interpreter|'s ability to generalize and adapt across varying function complexities. It stresses the necessity of incorporating tools like example synthesis into interpretability agents to improve their reasoning breadth.

This paper emphasizes that while LLM-driven agents show promise in automating interpretability tasks, there is ample room for refinement to achieve robust, nuanced model explanations. Future iterations of FIND may extend to white-box settings, offering greater insights into model internals beyond black-box functional descriptions. The findings from this benchmark can guide the development of more sophisticated interpretability methods, paving the way for deeper and actionable insights into the behavior of AI models.