Add HRP countries to population theme #71
Conversation
tests/test_main.py
Outdated
| "NGA", | ||
| "TCD", | ||
| "YEM", | ||
| ] |
There was a problem hiding this comment.
I included this to try to control which countries are run in the scraper, but it's not having the effect I hoped for. The tests fail looking for data for CMR.
@mcarans is there a way to limit the countries I'm running like in the covid viz scraper?
There was a problem hiding this comment.
This will require making a change to HDX Python Scraper to allow limiting the countries
Pull Request Test Coverage Report for Build 7717136522
💛 - Coveralls |
| project_config_dict = subset_scraper_countries( | ||
| project_config_dict, | ||
| countries_to_remove, | ||
| ) |
There was a problem hiding this comment.
@mcarans I created a function "subset_scraper_countries" to remove scrapers from the configs as a workaround to changing hdx-python-scraper. It feels like adding this functionality to hdx-python-scraper isn't worth the number of times we would use it, but if you think it would be useful I'm happy to look into it!
There was a problem hiding this comment.
This looks fine. How scalable do you think the process is? Is it a lot of work for all the countries?
As for adding a function to hdx-python-scraper, I think what you wanted is likely to be needed again although rather than deleting keys, I'd add a parameter somewhere to specify desired countries. I'll look into this sometime.
|
Sounds good! I'll create a ticket and add a to-do in the code. For the themes we currently have it's definitely scaleable (except for the HNO data). They're either global data that we can just change the filter on, or similar enough between countries that adding more resources will be straightforward. |
I haven't figured out how to address the tests yet - I was planning to limit the tests to the initial six countries, unless I should include all 25 of them. I just thought that number of files would be overwhelming and the tests would take a very long time to run.