Backend de "La Leaderboard"

To evaluate the models in the requests dataset, run python3 -m main_eval_basic_queue.py
To evaluate a custom combination of models and tasks, run python3 -m main_eval_internal_queue.py with an optional argument containing the path to the JSON file with the tasks to run which defaults to "internal_queue/tasks_todo.json"
To evaluate the models in the cluster's queue managed by slurm, run python3 -m main_eval_slurm_queue.py

Notes:

Name	Name	Last commit message	Last commit date
Latest commit mariagrandury fix setuptools package discovery Dec 7, 2024 bc0f728 · Dec 7, 2024 History 27 Commits
basic_queue	basic_queue	feat: upload results only for full evals	Nov 6, 2024
internal_queue	internal_queue	update tasks todo	Dec 5, 2024
slurm_queue	slurm_queue	fix lm eval cli command	Oct 30, 2024
.gitattributes	.gitattributes	feat: implement backend	Oct 22, 2024
.gitignore	.gitignore	fix setuptools package discovery	Dec 7, 2024
.pre-commit-config.yaml	.pre-commit-config.yaml	feat: implement backend	Oct 22, 2024
README.md	README.md	update lm eval version	Dec 6, 2024
TODO.md	TODO.md	update tasks todo	Nov 6, 2024
main_eval_basic_queue.py	main_eval_basic_queue.py	fix: update repo name and remove unexpected arg	Oct 23, 2024
main_eval_internal_queue.py	main_eval_internal_queue.py	pass task todo path file as argument	Dec 4, 2024
main_eval_slurm_queue.py	main_eval_slurm_queue.py	fix slurm config path	Oct 30, 2024
pyproject.toml	pyproject.toml	fix setuptools package discovery	Dec 7, 2024
requirements.txt	requirements.txt	move dependencies to pyproject	Dec 7, 2024

Provide feedback