Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run scrapers against live source websites in regular interval (nightly/weekly) #1057

Open
tillprochaska opened this issue Nov 14, 2024 · 0 comments · May be fixed by #1059
Open

Run scrapers against live source websites in regular interval (nightly/weekly) #1057

tillprochaska opened this issue Nov 14, 2024 · 0 comments · May be fixed by #1059

Comments

@tillprochaska
Copy link
Collaborator

We mock HTTP requests in our scraper tests, but it would be good to occasionally run them against live source websites, too, to make sure we’re notified about breaking changes as soon as possible.

I think the easiest way would probably be to optionally disable/enable request mocking in tests. Then we could set up a scheduled GitHub Actions workflow that runs the tests against the live sources.

@tillprochaska tillprochaska changed the title Run scrapers against live source websites in regulari interval (nightly/weekly) Run scrapers against live source websites in regular interval (nightly/weekly) Nov 16, 2024
@tillprochaska tillprochaska linked a pull request Nov 17, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant