Add Gene-MTEB tasks #1959
Add Gene-MTEB tasks #1959
Conversation
| dataset={ | ||
| "path": "metagene-ai/HumanMicrobiomeProjectDemonstration", | ||
| "name": "disease", | ||
| "revision": "main", |
There was a problem hiding this comment.
You need to specify exact revision of your dataset
There was a problem hiding this comment.
I think this is from your fork. Please update the README.
| test_size = 1 - (desired_train_samples / M) | ||
| split_datasets = full_train_dataset.train_test_split( | ||
| test_size=test_size, | ||
| shuffle=True, | ||
| seed=42) | ||
| new_train_dataset = split_datasets['train'] | ||
| new_test_dataset = split_datasets['test'] | ||
| self.dataset = datasets.DatasetDict({ | ||
| 'train': new_train_dataset, | ||
| 'test': new_test_dataset | ||
| }) |
There was a problem hiding this comment.
Can you upload dataset with these splits created directly?
There was a problem hiding this comment.
Or can you create function in this file that would be used in all tasks
There was a problem hiding this comment.
I would def. upload it directly (unless there is a very strong reason for doing otherwise)
| def __init__(self, **kwargs): | ||
| super().__init__(**kwargs) | ||
| self.method = "logReg" |
There was a problem hiding this comment.
Your tasks can be run with only a logreg?
| if self.data_loaded: | ||
| return | ||
|
|
||
| from transformers.trainer_utils import set_seed |
There was a problem hiding this comment.
Can you move imports to start of file?
| 'train': new_train_dataset, | ||
| 'test': new_test_dataset | ||
| }) | ||
| print(f"\nSplitting the data with test_size={test_size}") |
There was a problem hiding this comment.
Please, remove prints or change to logging
| main_score="accuracy", | ||
| date=("2009-10-09", "2012-11-22"), | ||
| domains=["Medical"], | ||
| task_subtypes=None, |
There was a problem hiding this comment.
| task_subtypes=None, | |
| task_subtypes=[], |
| date=("2009-10-09", "2012-11-22"), | ||
| domains=["Medical"], | ||
| task_subtypes=None, | ||
| license="not specified", |
There was a problem hiding this comment.
No license? This can really limit how useable the task is
|
|
||
| class HumanMicrobiomeProjectDemonstrationClassificationDisease(AbsTaskClassification): | ||
| metadata = TaskMetadata( | ||
| name="HumanMicrobiomeProjectDemonstrationClassificationDisease", |
There was a problem hiding this comment.
This is a very long title. I will appear very odd in the leaderboard. Would it be worth reducing it?
| test_size = 1 - (desired_train_samples / M) | ||
| split_datasets = full_train_dataset.train_test_split( | ||
| test_size=test_size, | ||
| shuffle=True, | ||
| seed=42) | ||
| new_train_dataset = split_datasets['train'] | ||
| new_test_dataset = split_datasets['test'] | ||
| self.dataset = datasets.DatasetDict({ | ||
| 'train': new_train_dataset, | ||
| 'test': new_test_dataset | ||
| }) |
There was a problem hiding this comment.
I would def. upload it directly (unless there is a very strong reason for doing otherwise)
| }, | ||
| type="Classification", | ||
| category="s2s", | ||
| modalities=["text"], |
There was a problem hiding this comment.
A text modality seems someone misleadering. It input is a string of base pairs right?
There was a problem hiding this comment.
keeping "text" here might be fine but it should be clear for the description. I would also make it clear in the task subtypes.
Closing #1781
Code Quality
make lintto maintain consistent style.Documentation
Testing
make testormake test-with-coverageto ensure no existing functionality is broken.Adding datasets checklist
Reason for dataset addition: To add specific benchmarks tailored to metagenomic analysis, where tasks involve classification, multi-label classification and clustering based on public available metagenomic data samples.
mteb -m {model_name} -t {task_name}command.sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2intfloat/multilingual-e5-smallself.stratified_subsampling() under dataset_transform()make test.make lint.