Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change hide_test:bool to hiding displayed metrics, not preventing evaluation form happening. #18

Open
aiqc opened this issue Mar 25, 2021 · 0 comments
Labels
good first issue Good for newcomers refactor Make code more modular/efficient or change schema.

Comments

@aiqc
Copy link
Owner

aiqc commented Mar 25, 2021

Difficulty: ★☆☆☆☆

Background

When using a validation split/fold to prevent evaluation bias top tune parameters/ topologies, you're not supposed to look at the holdout set.

Problem

Currently, the hide_test=True argument can be set when creating a Batch. However, this prevents the metrics from being generated on the test split entirely.

Solution

  • It would be better to have an argument on plot and metric functions for hiding the test split from the displayed data.
@aiqc aiqc changed the title Change how hide_test:bool to hiding metrics, not preventing eval. Change hide_test:bool to hiding displayed metrics, not preventing evaluation form happening. Mar 25, 2021
@aiqc aiqc added good first issue Good for newcomers refactor Make code more modular/efficient or change schema. labels Apr 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Good for newcomers refactor Make code more modular/efficient or change schema.
Projects
None yet
Development

No branches or pull requests

1 participant