Exploring Inference Patterns of LLMs as Symbolic Interactions

Lei Cheng; Wen Shen; Quanshi Zhang; Yuchen Huang; Yiming Zhang; Yuxiao Yang; Mingjie Li

In this paper, we examine whether a set of sparse symbolic interactions can faithfully explain the complex inference patterns of large language models (LLMs), thereby using interactions to diagnose the detailed representations of LLMs. Since previous studies lack a comprehensive proof for interaction-based explanations, we first experimentally verify the sparsity of interactions in LLMs, i.e., we observe that 50--100 interactions are usually powerful enough to accurately mimic all subtle changes in the LLM's output scores over an exponential number of arbitrary masked inputs. More crucially, interactions provide new perspectives for analyzing an LLM. (1) We propose to disentangle the interactions encoded by the LLM in a coarse-to-fine manner and use interactions to analyze the increasing representational complexity of features during forward propagation. The clear disentanglement of interactions enables us to identify incorrect inference patterns hidden in an LLM. (2) We empirically demonstrate that LLMs with different architectures encode similar interactions for text generation, and simple interactions are more likely to be consistently extracted by different LLMs than complex interactions. (3) We also empirically demonstrate that an LLM encodes similar interactions over semantically similar input sentences. Again, simple interactions are more likely to be stably extracted than complex interactions. (4) We empirically find that low-order interactions can serve as indicators for guiding model compression.

Exploring Inference Patterns of LLMs as Symbolic Interactions

Abstract