Skip to content

Issues: open-compass/opencompass

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

211 Open 345 Closed
211 Open 345 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

您好，请问L-Eval的主观题最终得分是使用rougeLsum这个分数吗？还有就是L-Eval数据集缺少了codeU和sci_fi有相关的评测配置文件么？
#1721 opened Nov 28, 2024 by 13416157913
2 tasks done
@acylam
[Feature] 请问使用API评测如何支持自定义数据集？
#1718 opened Nov 28, 2024 by Jimmy-L99
1 task
@acylam
[Bug] 测评结果为空，求助
#1716 opened Nov 26, 2024 by Jimmy-L99
2 tasks done
@liushz
3
[Bug] strategyqa answer extraction error
#1715 opened Nov 25, 2024 by Linzwcs
2 tasks done
@bittersweet1999
[Bug] stop_at_stop_token 删除了生成的方法体导致没有评估结果
#1710 opened Nov 23, 2024 by 886gb
2 tasks done
@tonysy
[Bug] DO NOT Use relative import
#1703 opened Nov 20, 2024 by jinmingyi1998
2 tasks done
@MaiziXiao
2
[Feature] HumanEvalX use Chat Mode as humaneval_openai_sample_evals_gen_159614
#1687 opened Nov 13, 2024 by tonysy
1 task done
@MaiziXiao
[Bug] The evaluation of configurations requiring a TopKRetriever, such as for flores_datasets, failed with a KeyError: 'metadata'.
#1686 opened Nov 13, 2024 by DespairL
2 tasks done
[Bug] TypeError: Expected a datasets.Dataset or a datasets.DatasetDict object, but got {'train': <modelscope.msdatasets.ms_dataset.MsDataset object at xxx>, 'test': <modelscope.msdatasets.ms_dataset.MsDataset object at xxx>}
#1678 opened Nov 12, 2024 by MrWiffer
2 tasks done
[Bug] v0.3.5版本评测Qwen/Qwen2.5-72B得分显著下降
#1675 opened Nov 11, 2024 by guoshengCS
2 tasks done
@MaiziXiao @bittersweet1999
10
[Feature] dataset for humaneval-multipl
#1673 opened Nov 9, 2024 by jyshee
1 task
@tonysy
1
[Feature] MMLU CEVAL MATH test results for some models
#1664 opened Nov 4, 2024 by jinmingyi1998
1 task
@bittersweet1999
1
[Bug] mmlupro 正则提取错误
#1661 opened Nov 4, 2024 by bittersweet1999
2 tasks done
@tonysy
[Bug] Error in downloading GSM8k dataset
#1654 opened Oct 31, 2024 by acse-ym722
2 tasks done
@acylam
1
[Bug] 多种测试方法测试结果差异大
#1649 opened Oct 29, 2024 by DietDietDiet
2 tasks done
@liushz
[Bug] TruthfulQA同时使用多个评估metric时报错
#1646 opened Oct 28, 2024 by XEric7
2 tasks done
@bittersweet1999
[Bug] 通过调用vllm启动服务的api接口，多次运行测试，得分结果不一致
#1634 opened Oct 23, 2024 by luhairong11
2 tasks done
@MaiziXiao
2
SafetyBench数据集评测bug
#1622 opened Oct 18, 2024 by shutttttdown
2 tasks done
@tonysy
[Bug] 使用Circular方式评测，few-shot数据拼接错误
#1615 opened Oct 17, 2024 by Alexyuanfun
2 tasks done
@acylam
[Feature] 请求添加BigCodeBench评测集 backlog
#1613 opened Oct 16, 2024 by BinBrent
1 task
@MaiziXiao
2
[Bug] 评估wikitext ppl数据集时候由于没有reference无法计算结果
#1608 opened Oct 15, 2024 by LanDisen
2 tasks done
@liushz
[Bug] hf推理与vllm推理评测结果不一致
#1594 opened Oct 9, 2024 by luhairong11
2 tasks done
@MaiziXiao
1
[Bug] AttributeError: 'ConfigDict' object has no attribute 'reader_cfg'
#1591 opened Oct 9, 2024 by xtuhcy
2 tasks done
@MaiziXiao
1
[Bug] Silent GPU failures, works with --debug
#1585 opened Oct 4, 2024 by anuragprat1k
2 tasks done
@tonysy
[Bug] pid_params.py dumps not successfully.
#1580 opened Sep 29, 2024 by tonysy
2 tasks done
@tonysy
ProTip! Find all open issues with in progress development work with linked:pr.