Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[metrics] (MeanScores) Add a mean scores metrics inspired from IL2M #683

Merged

Conversation

Mathieu4141
Copy link
Contributor

@Mathieu4141 Mathieu4141 commented Jun 23, 2021

To prepare #676

This is a metric that was used to show bias toward new classes in the IL2M paper.
The scores of the true class are averaged by past and new classes.

  • For the train plugin I only used the last epoch, the graphs are hard to read if we add more
  • For the eval plugin the full stream needs to be considered, so that we can compare old and new classes

This is what I get for the simple example:
Screen Shot 2021-06-23 at 10 27 40 PM

The MeanScores metric can be reused for the IL2MPlugin

Edit: Ready for review :)

@Mathieu4141 Mathieu4141 marked this pull request as draft June 23, 2021 20:32
@Mathieu4141 Mathieu4141 mentioned this pull request Jun 23, 2021
@coveralls
Copy link

coveralls commented Jun 23, 2021

Pull Request Test Coverage Report for Build 969419435

  • 1 of 5 (20.0%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage decreased (-0.02%) to 79.095%

Changes Missing Coverage Covered Lines Changed/Added Lines %
avalanche/evaluation/metrics/mean.py 1 5 20.0%
Totals Coverage Status
Change from base Build 964673211: -0.02%
Covered Lines: 9504
Relevant Lines: 12016

💛 - Coveralls

@AndreaCossu
Copy link
Collaborator

Thanks @Mathieu4141, seems good! Have you considered building your plugin metric on the GenericPluginMetric interface? If it was possible, it would make the metrics more homogenous. Let me know 😄

@Mathieu4141
Copy link
Contributor Author

Thanks @Mathieu4141, seems good! Have you considered building your plugin metric on the GenericPluginMetric interface? If it was possible, it would make the metrics more homogenous. Let me know 😄

@AndreaCossu I tried at first to do it with GenericPluginMetric. However this plugin doesn't quite fit the template so I switch to a custom plugin, and it simplified things a lot
Some of the issues I had:

  • The eval plugin would get reset during eval, so we would loose the new/old classes distinction that can only be taken from the previous training. In a custom plugin I simply get the classes in before_training_exp, and reset at before_training
  • The train plugin would work on all epochs, but the graphs are really difficult to read and interpret then, doing it on the last epoch seems better

Those issues are really specific to this plugin, so I didn't want to change the generic class only for that

@Mathieu4141 Mathieu4141 marked this pull request as ready for review June 24, 2021 19:00
@AndreaCossu
Copy link
Collaborator

Thank you again @Mathieu4141 , seems all good to me, I will merge this 😄

@AndreaCossu AndreaCossu removed the request for review from EdenBelouadah July 5, 2021 12:52
@AndreaCossu AndreaCossu merged commit a109973 into ContinualAI:master Jul 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants