Only two submissions to competition ml2021spring-hw2

[Bihui-Jin]

Hi MLE-Bench Developers,

I'm currently copying data from your listed competitions and noticed something odd with the ml2021spring-hw2 competition. There appear to be only two submissions (kernels), but the leaderboard shows scores for over a thousand participants with no visible submissions. Is this correct? Did you use these two submissions as your dev set when constructing your MLE-Bench paper?

Thanks for your help!

[thesofakillers]

I'm not sure I'm following your question.

There appear to be only two submissions (kernels), but the leaderboard shows scores for over a thousand participants with no visible submissions. Is this correct?

On kaggle, leaderboard entries don't always have submissions publicly associated with them. Sometimes users may choose to attach a kernel but I think it depends on the competition and on the user.

In MLE-bench we only use the leaderboard scores, not the associated submissions, so if the leaderboard is missing associated submissions this is not an issue, unless I am misunderstanding your question. So, this seems "correct" to me.

Did you use these two submissions as your dev set when constructing your MLE-Bench paper?

We did not afaik, but I'm not sure they would be super relevant for a dev set since the MLE-Bench dataset is not made of submissions, but of competition descriptions and leaderboards alongside the associated metric implementations.

---

Let me know if I've answered your question