Discovering the weakness of the RealPLKSR model architecture (yet to be confirmed).

[avan06]

Hi Philip,

First of all, thank you for training so many excellent models. Recently, I’ve been using the 4xNomosWebPhoto_RealPLKSR model in my program. However, I’ve found that when testing with the sample images provided on your page, I couldn’t achieve the expected results no matter what I tried. Later, I identified the cause and wanted to share this case with you.

In my program, when an input image’s height is less than 300, the program doubles its size. When I tested with the smaller sample images, doubling their size first and then using the 4xNomosWebPhoto_RealPLKSR model to upscale them 4x, the output was far from satisfactory. However, when I removed the resizing step, the output was as expected.

---

Below are the case images:

Original image1

Model output from the original image1

Original image1 resized to double the size

Model output from the resized image1

---

Original image2

Model output from the original image2

Original image2 resized to double the size

Model output from the resized image2

---

Original image3

Model output from the original image3

Original image3 resized to double the size

Model output from the resized image3

---

Original image4

Model output from the original image4

Original image4 resized to double the size

Model output from the resized image4

---

Original image5

Model output from the original image5

Original image5 resized to double the size

Model output from the resized image5

---

As shown in the test results above, the RealPLKSR model produces suboptimal results in this specific scenario. I’m not sure what the cause might be, but thank you for looking into it!

[Phhofm]

tldr: model is not trained for that use case. Its not an architecture thing but degradation/training thing.

Hey, thanks for the post
Yeah this has a very simple reason, that model was not trained for upsized content.
Well its not entirely true, the training (or rather, LR creation) setting was this

Meaning it might be able to handle 1.25x upsized content.
I simply didnt think someone would 2x size content before the upscale since if an image (like a photo) gets put on a website, to increase page speed, it often gets sized down and compressed (not the other way around).

So to answer your question, this model was simply not trained to handle someone preprocessing input with a 2x resize. Thats also what you are seeing in your tests.

Even though this model was trained to be used without this purpose in mind, I can show something that would improve your results, by simply changing your order (if you need those image dimensions):
First we resize 2x with cubic, then upscale with this model, the expected bad result

Now we instead upscale first, then resize 2x with cubic, it should look better

We could alternatively also upscale, then upscale that output again, then resize 0.5x with cubic

And just to show in general, an otf trained model might be better equipped to handle a specific case where someone 2x sized content before upscaling, because it has a chance to up_up with up to 1.5x and then 1.2x in the two degradation steps which in theory could come up to 1.8x input during training
2x cubic sized, then upscaled, with 4xBHI_realplksr_dysample_otf_nn (so still a realplksr model because I wanted to show the training difference, not some architecture difference)

Hope this helps for understanding

[avan06]

Thank you very much to the model author for the detailed explanation and for suggesting improvement methods. I now fully understand!

Regarding the 2x enlargement of small images, I only noticed it this time because I duplicated someone else's Image Upscale Space and modified it to support the RealPLKSR model. I’ve already removed the resizing code, so there’s no issue now.

Once again, thank you so much!