Privacy Policy / Cookie Consent being included in the response, exploding its size

[antonioalegria]

If you look at a website like https://widn.ai/en you can see that it brings a huge section with each of the data partners listed in the cookie consent.

This doesn't happen with, for example, firecrawl

[nomagick]

This is because the consent popup on this particular website uses more complex CSS techniques to hide the popup. It's visually hidden but still exists inside the page as a legit content.

For this particular website, try this:

curl https://r.jina.ai/https://widn.ai/en -H 'x-remove-selector: [class*="cky-"]'