Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Golden images in /build/paparazzi/failures are not stored in some error cases #1716

Open
SimonMarquis opened this issue Nov 22, 2024 · 0 comments · May be fixed by #1827
Open

Golden images in /build/paparazzi/failures are not stored in some error cases #1716

SimonMarquis opened this issue Nov 22, 2024 · 0 comments · May be fixed by #1827
Labels
bug Something isn't working
Milestone

Comments

@SimonMarquis
Copy link
Contributor

SimonMarquis commented Nov 22, 2024

I'm not sure if this should be classified as a bug of a feature request:

Description
Golden images are stored in /build/paparazzi/failures when there is a regression (next to the delta-*.png files) but, they are not when:

  • The reference image (in /test/snapshots/images) is not present:

        java.lang.AssertionError: File module/ScreenshotTesting/src/test/snapshots/images/...ScreenshotTest_test.png does not exist
            at app.cash.paparazzi.SnapshotVerifier$newFrameHandler$1.handle(SnapshotVerifier.kt:60)
            at app.cash.paparazzi.Paparazzi$apply$1.evaluate$lambda$0(Paparazzi.kt:73)
            at app.cash.paparazzi.PaparazziSdk.takeSnapshots(PaparazziSdk.kt:333)
            at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:178)
            at app.cash.paparazzi.PaparazziSdk.snapshot$default(PaparazziSdk.kt:176)
            at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:173)
            at app.cash.paparazzi.Paparazzi.snapshot(Paparazzi.kt:103)
    
  • The reference image has a different size:

        java.lang.IllegalStateException: Images are different sizes
            at app.cash.paparazzi.internal.OffByTwo.compare(OffByTwo.kt:11)
            at app.cash.paparazzi.internal.ImageUtils.compareImages(ImageUtils.kt:144)
            at app.cash.paparazzi.internal.ImageUtils.assertImageSimilar(ImageUtils.kt:55)
            at app.cash.paparazzi.SnapshotVerifier$newFrameHandler$1.handle(SnapshotVerifier.kt:73)
            at app.cash.paparazzi.Paparazzi$apply$1.evaluate$lambda$0(Paparazzi.kt:73)
            at app.cash.paparazzi.PaparazziSdk.takeSnapshots(PaparazziSdk.kt:333)
            at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:178)
            at app.cash.paparazzi.PaparazziSdk.snapshot$default(PaparazziSdk.kt:176)
            at app.cash.paparazzi.PaparazziSdk.snapshot(PaparazziSdk.kt:173)
            at app.cash.paparazzi.Paparazzi.snapshot(Paparazzi.kt:103)
    

Steps to Reproduce
I can create a simple project for that if it is really necessary, but I think the description is fairly simple, and easy to reproduce, without special setup. Please let me know.

Expected behavior
Having golden screenshots generated even when the test fail because it does not match the reference (or if the reference is missing) would be very valuable in CI scenarios, where the newly created golden images could be immediately extracted without having to run an entire pipeline with a new execution of recordPaparazzi Gradle task.
Also, running the recordPaparazzi might lead to unrelated golden images being updated (with invisible variations) that would flood the review diff.

Additional information:

  • Paparazzi Version: 1.3.5

Note:
In previous versions (pre-OffByTwo), when there was a regression with different image sizes, the delta and golden images were stored to disk on /out/failures.

@SimonMarquis SimonMarquis added the bug Something isn't working label Nov 22, 2024
SimonMarquis added a commit to SimonMarquis/paparazzi that referenced this issue Feb 2, 2025
This will make `OffByTwo.kt` behaves identical to the original `PixelPerfect` differ.
But most importantly, it will restore the initial behavior of `ImageUtils.assertImageSimilar()` itself, checking for differences in size and correctly producing the expected delta (and actual) images in case of a size mismatch.

Relates to cashapp#1716
SimonMarquis added a commit to SimonMarquis/paparazzi that referenced this issue Feb 2, 2025
This will make `OffByTwo.kt` behaves identical to the original `PixelPerfect` differ.
But most importantly, it will restore the initial behavior of `ImageUtils.assertImageSimilar()` itself, checking for differences in size and correctly producing the expected delta (and actual) images in case of a size mismatch.

Relates to cashapp#1716
@geoff-powell geoff-powell added this to the 2.0.0-alpha01 milestone Feb 7, 2025
SimonMarquis added a commit to SimonMarquis/paparazzi that referenced this issue Feb 8, 2025
…t present

This is the 2nd PR to finally close cashapp#1716.

Also relates to:
- cashapp#1692
- cashapp#1695

since this still requires the presence of the `src/test/snapshots/images` input to work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants