In silico discovery of representational relationships across visual cortex

Here we provide the code from the paper:
"In silico discovery of representational relationships across visual cortex".
Alessandro T. Gifford, Maya A. Jastrzębowska, Johannes J.D. Singer, Radoslaw M. Cichy

📖 Theoretical motivation

Seeing is underpinned by a complex interconnected network of multiple brain regions of interest (ROIs) jointly representing visual information. However, the representational content of each ROI is typically studied in isolation, and using limited sets of experimenter-picked stimuli. Here, we addressed this by developing Relational Neural Control (RNC). RNC generates and explores in silico functional magnetic resonance imaging (fMRI) responses for large amounts of images, finding controlling images that align or disentangle responses across ROIs, thus indicating their shared or unique representational content.

We developed two RNC variants:

• Univariate RNC controls univariate fMRI responses (i.e., responses averaged over all voxels within an ROI), thus exploring representational relationships for visual information encoded in the strongest activation trends common across all ROI voxels.
• Multivariate RNC controls multivariate fMRI responses (i.e., population response of all voxels within a ROI), thus exploring representational relationships for visual information encoded in the multi-voxel response patterns.

To thoroughly explore the visual space in search for controlling stimuli, we applied RNC to in silico neural responses for thousands of naturalistic images, that is, neural responses generated through encoding models from the Neural Encoding Dataset (NED). Specifically, this includes NED-generated in silico fMRI responses for the 73,000 Natural Scenes Dataset (Allen et al., 2022) images, the 50,000 ImageNet 2012 Challenge (Russakovsky et al., 2015) images, or the 26,107 THINGS Database (Hebart et al., 2019) images.

For any question regarding this code, the project data, or RNC in general, you can get in touch with Ale ([email protected]).

🚀 RNC tutorials

We created interactive tutorials where you can learn how to use univariate and and multivariate RNC. These tutorials are available on either Google Colab (univariate RNC, multivariate RNC) or Jupyter Notebook (univariate RNC, multivariate RNC).

♻️ Reproducibility

⚙️ Installation

To reproduce the paper's results, you can download and run the Python code from this repository. We recommend running the code using an Anaconda environment with the following packages installed:

argparse, h5py, matplotlib, ned, nibabel, nsdcode, numpy, pandas, pillow, scipy, sklearn, statsmodels, torchvision, tqdm

🧰 Data

To run the code you will need to download the following:

• The Neural Encoding dataset (https://www.alegifford.com/projects/ned/).

• The 73,000 stimulus images from the Natural Scenes Dataset (https://naturalscenesdataset.org/).

• The 50,000 ILSVRC-2012 validation images (https://www.image-net.org/challenges/LSVRC/2012/index.php).

• The 26,107 images from THINGS (https://osf.io/jum2f/).

• The in vivo fMRI responses for the controlling images collected in this project (https://openneuro.org/datasets/ds005503).

📦 Code description

• 00_generate_insilico_fmri_responses: Generate in silico fMRI responses for naturalistic images, using trained encoding models from the Neural Encoding Dataset.
• 01_in_silico_fmri_encoding_accuracy: Compute the encoding accuracy and perform a noise analysis on the in silico fMRI responses.
• 02_univariate_rnc: Apply univariate RNC on the in silico fMRI responses.
• 03_generative_univariate_rnc: Apply generative univariate RNC on in silico fMRI responses.
• 04_multivariate_rnc: Apply multivariate RNC on the in silico fMRI responses.
• 05_multivariate_rnc_retinotopy: Perform the retinotopy analysis on the in silico fMRI resposnes for the V1 vs. V4 multivariate RNC controlling images.
• 06_in_vivo_validation: Analyze the in vivo fMRI responses for the V1 vs. V4 univariate and multivariate RNC controlling images.

❗ Issues

If you experience problems with the code, please submit an issue!

📜 Citation

If you use any of our data or code, please cite:

• Gifford AT, Jastrzębowska M, Singer JJD, Cichy RM. 2024. In silico discovery of representational relationships across visual cortex. arXiv preprint, arXiv:2411.10872. DOI: https://doi.org/10.48550/arXiv.2411.10872
• Gifford AT, Cichy RM. 2024. The Neural Encoding Dataset. In preparation. https://github.com/gifale95/NED