Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug Report] Resnet50 accuracy is way off on Wormhole #16895

Open
pavlepopovic opened this issue Jan 20, 2025 · 1 comment
Open

[Bug Report] Resnet50 accuracy is way off on Wormhole #16895

pavlepopovic opened this issue Jan 20, 2025 · 1 comment
Assignees
Labels
bug Something isn't working P1

Comments

@pavlepopovic
Copy link
Contributor

pavlepopovic commented Jan 20, 2025

Describe the bug
Resnet50 accuracy on Wormhole, when running ImageNet dataset is 0.369.
This is also visible when running a standalone demo (without dataset), from the repo:

Image

Bisecting revealed that this is a regression, coming from this commit: 282a7b2

It is not caught with regular tests, since the PCC check logic is also broken for wormhole, and is accidentally picking up PCC targets from Grayskull, which are lower.

To Reproduce
Steps to reproduce the behavior:

Single demo run:
WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml pytest --disable-warnings models/demos/wormhole/resnet50/demo/demo.py::test_demo_sample

ImageNet dataset run:
WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml pytest --disable-warnings models/demos/wormhole/resnet50/demo/demo.py::test_demo_imagenet

Expected behavior
Accuracy should be 0.76 on ImageNet dataset.

@pavlepopovic pavlepopovic added bug Something isn't working P1 labels Jan 20, 2025
pavlepopovic pushed a commit that referenced this issue Jan 20, 2025
… resnet. Also, fix PCC check in the demo, as it was not picking up correct values for wormhole
@pavlepopovic
Copy link
Contributor Author

The first conv in Resnet50 has a config which doesn't work (self.conv1_config.act_block_h_override = 49 * 32)
Setting it to 2 * 32 restores the accuracy back to 0.76.
Other values: 7 * 32, 14 * 32 also aren't working.

Using the working ConvConfig and fixing the PCC checker here: #16896
Leaving the issue open until the conv is fixed

pavlepopovic added a commit that referenced this issue Jan 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working P1
Projects
None yet
Development

No branches or pull requests

2 participants