Database outputs

[hannahker]

Porting over code from https://github.com/OCHA-DAP/ds-floodexposure-monitoring-app, that should live as a pipeline instead. This is a quick and dirty copy and paste exercise -- there are still a number of things that we'll need to improve to make this our production pipeline. .

[t-downing]

Nice! In addition to the thing about locally storing the CODAB, I think we just need to add a line to run pipelines/update_database.py to the workflow .yml so this runs each day as well.

Then, as discussed, we can do a separate PR to bypass the intermediate .parquet storage and just write straight to the DB. But we can merge this one first to get the DB write happening automatically each day, cause I noticed it's getting out of date in the app.

[hannahker]

@t-downing I've addressed the database issues here and repopulated the flood_exposure database.

Then, as discussed, we can do a separate PR to bypass the intermediate .parquet storage and just write straight to the DB. But we can merge this one first to get the DB write happening automatically each day, cause I noticed it's getting out of date in the app.

I think we'll want to work on the migration from parquet and the daily updates together. Rerunning this update_database.py script as-is will rewrite data from all dates to the database, which takes some time and is quite redundant. It'll take a bit of refactoring to process the parquet files to only pull in the latest data, so at that point we may as well just refactor to get rid of the parquet files entirely. I'd vote for addressing those both in a separate PR once this one is merged.

[t-downing]

We can merge this now as an intermediate step to switching to the new database table