SPRAS benchmarking

Benchmarking datasets for the SPRAS project. This repository contains gold standard datasets to evaluate on as well as paper reproductions & improvements incorporating new methodologies. The results of every benchmarking run are deployed on GitHub pages. (See the current web output).

Setup

This repository depends on SPRAS. If you want to reproduce the results of benchmarking locally, you will need to setup SPRAS. SPRAS depends on Docker and Conda. If it is hard to install either of these tools, a devcontainer is available for easy setup.

conda env create -f spras/environment.yml
conda activate spras
pip install ./spras

To run the postprocess output scripts, we have a pyproject.toml which can be used with your desired python package manager. This separates the spras conda environment from the small scripts we have. (on CI, we use uv.)

To run the benchmarking pipeline, use:

snakemake --cores 1 --configfile configs/dmmm.yaml --show-failed-logs -s spras/Snakefile

Note

Each one of the dataset categories (at the time of writing, DMMM and PRA) are split into different configuration files. Run each one as you would want.

Organization

There are four primary folders in this repository:

.
├── configs
├── datasets
├── spras
└── web

spras is the cloned submodule of SPRAS, web is an astro app which generates the spras-benchmarking output, configs is the YAML file used to talk to SPRAS, and datasets contains the raw data.

The workflow runs as so:

1. For every dataset, run its inner Snakefile with Snakemake. This is orchestrated through the top-level run_snakemake.sh shell script.
2. Run each config YAML file in configs/ with SPRAS.
3. Build the website in web with the generated output from all of the SPRAS runs, and deploy it on GitHub Pages. To see how to build the website, go to its README.

For more information on how to add a dataset, see CONTRIBUTING.md.