On Relation-Specific Neurons in Large Language Models

Arxiv link

Overview

This code identifies relation-specific neurons that focus on relations rather than entities in decoder-only LLMs (specifically on Llama-2 models). With respect to their capacity for encoding relation information, we give evidence for the following three properties of relation-specific neurons. (i) Neuron cumulativity. The neurons for $r$ present a cumulative effect so that deactivating a larger portion of them results in the degradation of more facts in $r$. (ii) Neuron versatility. Neurons can be shared across multiple closely related as well as less related relations. Some relation neurons transfer across languages. (iii) Neuron interference. Deactivating neurons specific to one relation can improve LLM generation performance for facts of other relations. The code is based on Self-Conditioning Pre-Trained Language Models and On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons.

Data Path

# Ground Truth Texts
assets/Language/sense/
# Model-Generated Texts
outputs/

Quick Start

Installation

The requirements are listed in frozen_requirements.txt.
The code has been tested using Python 3.8.
Run the following for installation:

Create a virtual environment

cd <path_to_this_project>
conda create -n relation_neuron python=3.8
conda activate relation_neuron
pip install -U pip wheel

Install selfcond (recommended for reproducibility)

pip install -r frozen_requirements.txt
python -c "import nltk; nltk.download('punkt')"

---

1. Finding Relation-Specific Neurons

Model Support:

• llama-2-7b-hf
• llama-2-13b-hf

1.1 Generate prompts

Run the following script to generate prompts for the neuron identification. You can first modify the generate_prompts.py script on your own needs. The script will generate the identification prompt set (for identification) and test prompt set (for evaluation) for each relation $r$.

python scripts/generate_prompts.py

1.2 Create dataset & collect responses from a model

Run the following script to create a dataset and collect responses from a model. The script will first create the real dataset (a json file) for a single relation $r$, which contains positive examples (prompts from the relation $r$) and negative examples (prompts from other relations). Then the script will record the neuron output for each prompt.

Taking company_ceo relation as an example:

bash main_prod_env.sh "llama-2-7b compute_responses Relation company_ceo 6000 on_p50 expertise_limited_6000_both"

The responses will be saved inside mit_output/llama-2-7b-hf/sense/company_ceo/responses.

1.3 Compute expertise

The expertise score of a neuron is defined as the Average Precision (AP) achieved when its responses are considered prediction scores for the classification problem (positive examples and negative examples).

Taking company_ceo relation as an example:

bash main_prod_env.sh "llama-2-7b compute_expertise Relation company_ceo 6000 on_p50 expertise_limited_6000_both"

The expertise results are saved as a CSV file in mid_output/llama-2-7b-hf/sense/company_ceo/expertise.

1.4 Limit expertise (only Top-k)

Run the following script to to limit expertise to only Top-k neurons (sorted by the AP values) in FFNs.

Taking company_ceo relation as an example:

python scripts/make_limited_expert_exe.py --model-name llama-2-7b-hf --language company_ceo --num-units 6000 --neuron_type mlp

2. Controlled Generation

We measure the effect of selectively deactivating candidate neurons specific to relation $r$ on the LLM's ability to handle (1) prompts whose relation is $r$ and (2) prompts whose relation is a different relation $r^{\prime }\ne r$. The deactivation is done by overriding the neuron output values with 0 during inference.

python generate_conditioned.py --model-name llama-2-7b-hf --compute-all --neuron_group top --forcing zero --show-overlap --neuron_type mlp

3. Analysis & Visualization

3.1 Find common neurons

Run the following script to find common neurons across all relations and visualizing the overlap.

python compare_neurons.py --compute_all --neuron_group top

3.2 Visualize the intra-relation performance

python analysis/accuracy_analyisis.py
python analysis/draw_accuracy_table.py

3.3 Visualize the inter-relation performance

python scripts/plot_accuracy.py --model_name llama-2-7b-hf --num_units 3000 --neuron_type mlp

3.4 Multilingual evaluation

Taking de (German) as an example:

python generate_conditioned_multilingual.py --model-name llama-2-13b-hf --compute-all --neuron_group top --forcing zero --show-overlap --neuron_type mlp --language de

Citation

If you find our code or models useful for your research, please consider citing:

@misc{liu2025relationspecificneuronslargelanguage,
      title={On Relation-Specific Neurons in Large Language Models}, 
      author={Yihong Liu and Runsheng Chen and Lea Hirlimann and Ahmad Dawar Hakimi and Mingyang Wang and Amir Hossein Kargaran and Sascha Rothe and François Yvon and Hinrich Schütze},
      year={2025},
      eprint={2502.17355},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.17355}, 
}