Cached parsing

[whalemare]

Hi, it's is possible to split the whole crawling process to 3 steps?

1. Fetch all required webpages and save it to local files
2. Parse this files, instead of doing fresh requests
3. Save parsed data related to this page

This approach is very handy in case where pages is very unstructured, so you can freely experiment with parsing function (on step 2) without making a real request to website

[janbuchar]

Hello, and thanks for your interest in Crawlee! This usecase is not explicitly supported by the framework, but it should be pretty easy to achieve anyways. For caching of websites, I'd recommend using a standalone caching proxy such as https://www.npmjs.com/package/@loopback/http-caching-proxy.

Then I believe you can just repeat steps 2 and 3, overwrite the results each time, and adjust your parsing until you're happy with the result.

Does this work for you?