Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Added script for generating and saving a local leaderboard #2015

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Pringled
Copy link
Contributor

@Pringled Pringled commented Feb 7, 2025

Description

Added a script to create and save a leaderboard locally for a given benchmark as discussed in #2011.

The script retrieves results, parses them to a Pandas DF, prints them, and (optionally) saves them to a CSV. It uses the same functionality used for generating the live leaderboard. This makes it much easier to generate results locally and check aggregate scores before submitting them to MTEB.

Usage

The script can be used as follows:
python scripts/make_leaderboard --benchmark "MTEB(eng)" --save-path "leaderboard_results"

Optionally, a custom results repo can be used (default is https://github.com/embeddings-benchmark/results), and specific models can be requested, for example:
python scripts/make_leaderboard --benchmark "MTEB(eng)" --results_repo "https://github.com/Pringled/mteb-results" --models "potion-base-8M" --save-path "results"

Code Quality

  • Code Formatted: Format the code using make lint to maintain consistent style.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant