New database

[t-downing]

Key changes:

• .github/workflows/run_update_exposure.yml: add step to trigger second workflow
• .github/workflows/run_update_raster_stats.yml: created to calculate raster stats, and configured to be portable to another repository
• pipelines/update_database.py: deleted as we will no longer use this database table
• pipelines/update_raster_stats.py: script created to calculate raster stats and store in new database table
  • aligned with the public schemas (from ds-raster-stats), called app.floodscan_exposure
  • includes admin0 and 1 stats (add admin1 and admin0 stats #6)
• src/utils/database.py: define schema of new database table

Also:

• adding Mali

So, pipeline now would work by:

• calculating exposure raster with run_update_exposure.yml
• calculating exposure raster stats with run_update_raster_stats.yml
  • this workflow is configured to be triggered by the first workflow, and can eventually be moved to another repository

Note that I left in the .parquet calculation in floodscan.calculate_recent_flood_exposure_rasterstats() so we still have these as a backup that gets updated daily. When we are more sure about the new database, and are writing to it daily, we can get rid of this function.

[hannahker]

Maybe want to change the name of this in secrets to not reference DEV -- not essential though.

[t-downing]

Yeah in general I think having a better naming structure for our env vars would be good. Following up with Mike on including them as organization secrets as well, which would force us to be consistent.

[hannahker]

Awesome! Only minor comments/clarifications. This looks good to me.

Not to be done in this PR, but I'm assuming next steps from here would be:

• Reconfigure processing of exposure rasters to contain full geographic scope (ie. not split by iso3)
• Then replace the raster stats computations here with what we have in ds-raster-stats repo.

[t-downing]

Not to be done in this PR, but I'm assuming next steps from here would be:

Reconfigure processing of exposure rasters to contain full geographic scope (ie. not split by iso3)
Then replace the raster stats computations here with what we have in ds-raster-stats repo.

Yup! Noted as #12, #13, OCHA-DAP/ds-raster-stats#31