Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Monitoring 2025 #24

Open
wants to merge 102 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
102 commits
Select commit Hold shift + click to select a range
a717bee
add analysis of refined adm 1 AOIs for both ECMWF and INSIVUMEH
zackarno Feb 4, 2025
a4b7e97
make new threshold tables for technical note
zackarno Feb 4, 2025
61b6108
little notebook cleanup
zackarno Feb 4, 2025
54e03df
add url to rendered doc inside code chunk for easy reference
zackarno Feb 4, 2025
d4f47a7
refactor analysis for refined geographic thesholds
zackarno Feb 6, 2025
2a2be64
add joint RP table and current season outlook to qmd
zackarno Feb 7, 2025
53372a2
bugfix on var name - code chunk cleanup
zackarno Feb 7, 2025
279b4a6
re-structuring monitoring code
zackarno Feb 7, 2025
9158390
add datasource/insivumeh.R - required func i forgot to commit
zackarno Feb 7, 2025
b070f9a
Merge pull request #23 from OCHA-DAP/new-aoi-joint-rp
zackarno Feb 7, 2025
ff1f7f4
refactor monitoring for 2025 - test emails working.
zackarno Feb 9, 2025
2815580
add monitoring 2025 gha boilerplate yaml
zackarno Feb 9, 2025
0af0700
change storage location for insivumeh rasters on blob
zackarno Feb 10, 2025
a1c1b2a
clean up/commenting change green color on email map
zackarno Feb 10, 2025
1af09f4
fix insivumeh blob path in loading, add aoi gt table to email
zackarno Feb 10, 2025
1ce057a
fix dissolving of guatemala admin 1 prior to zonal stats
zackarno Feb 10, 2025
ec164cb
create req file to attempt to cache reqs on GHA for testing
zackarno Feb 10, 2025
7b67aff
fix caching file name
zackarno Feb 10, 2025
b33ca7f
setting up email list fitering- add temp depends_additoinal for box o…
zackarno Feb 10, 2025
4c2ec4e
fix yaml whitespace
zackarno Feb 10, 2025
10f7278
add test options for EMAIL_WHO
zackarno Feb 10, 2025
f9b4cf0
add box to main dependency installation file
zackarno Feb 10, 2025
c915a72
add test_gha_settings.R script to make sure GHA is setting env vars c…
zackarno Feb 10, 2025
e7871a5
add testing script to yaml run
zackarno Feb 10, 2025
aed1aa9
fix filepath typo
zackarno Feb 10, 2025
3100646
fix filepath typo
zackarno Feb 10, 2025
d5a74fd
edit yaml
zackarno Feb 10, 2025
9594529
edit yaml to retry install of gghdx()
zackarno Feb 10, 2025
b1f1d51
add r scripts back to yaml
zackarno Feb 11, 2025
0c0d8a3
add deps additional again for gghdx
zackarno Feb 11, 2025
e06f140
add linux dep for gghdx dep
zackarno Feb 11, 2025
392f3e1
rename to re-cache
zackarno Feb 11, 2025
0a9e0b3
retrieve gghdx from cran for cache invalidation
zackarno Feb 11, 2025
a56325b
gghdx installed successfully - trying mock email
zackarno Feb 11, 2025
32fbd63
get rid of development code
zackarno Feb 11, 2025
7f981c1
add RPostgres dep
zackarno Feb 11, 2025
6f2e7cc
update cache action version
zackarno Feb 11, 2025
786aec7
consolidate depends
zackarno Feb 11, 2025
6417fc0
add cumulus and rpostgres in as deps correctly?
zackarno Feb 11, 2025
ae57102
test adding postgres docker service
zackarno Feb 11, 2025
dec469f
add dbplyr dep
zackarno Feb 11, 2025
6d0f895
add loggers
zackarno Feb 11, 2025
245b986
Merge branch 'aoi-refine-2025' into monitoring-2025
zackarno Feb 11, 2025
6a82193
add geoarrow to see if helps w/ loading admin files in gha
zackarno Feb 11, 2025
cb85198
reinstall latest cumulus from blob
zackarno Feb 11, 2025
147631e
re-cache pkg install all from one depends script
zackarno Feb 11, 2025
772aee6
add gghdx() in to update_activation_status
zackarno Feb 12, 2025
0168e49
clearing cache and reinstalling - new first time bug
zackarno Feb 12, 2025
14e7d4e
remove clear cache
zackarno Feb 12, 2025
bc2486d
add libcur dep
zackarno Feb 12, 2025
35ea230
try adding curl dep
zackarno Feb 12, 2025
5e30519
add script back in for running afte re-installing cache
zackarno Feb 12, 2025
3ef5354
switch location of gghdx() - put inside rmd?
zackarno Feb 12, 2025
af81791
rm gghdx also add loggers
zackarno Feb 14, 2025
3518495
update loggers
zackarno Feb 14, 2025
7c6598a
add RUN_DATE_USE to env section of yaml
zackarno Feb 14, 2025
0699af8
fix yaml
zackarno Feb 14, 2025
f1b343e
add gghdx back into rmd
zackarno Feb 14, 2025
2d2c4bf
turn off gghdx()
zackarno Feb 14, 2025
4553a25
try to add gghdx$ to all gghex calls
zackarno Feb 14, 2025
bc4c38f
get rid of duplciate ggplot2 call
zackarno Feb 14, 2025
338404c
try debug w/ gdb
zackarno Feb 14, 2025
98ef4a7
rm debug -- change port in settings
zackarno Feb 14, 2025
6df3a0a
remove map from email
zackarno Feb 14, 2025
0cf6b2c
chg subj
zackarno Feb 14, 2025
5821417
try adding gghdx() into map.R
zackarno Feb 14, 2025
cf822f7
get rid of gghdx in main update script
zackarno Feb 14, 2025
c3abca5
rm dev code
zackarno Feb 14, 2025
169fcc8
pin R version to 4.4.1
zackarno Feb 14, 2025
ccbec90
back to R 4.x
zackarno Feb 14, 2025
2cd6309
going to recach pkgs
zackarno Feb 14, 2025
3abf3e3
add action back
zackarno Feb 14, 2025
34ef3cc
rm r run to cache packages
zackarno Feb 14, 2025
a2a4882
add gghdx() back in
zackarno Feb 14, 2025
cdb6478
add comments explaining RUN_DATE_USE
zackarno Feb 14, 2025
9d5c82a
add gghdx into eu module?
zackarno Feb 14, 2025
9e4546a
rm gghdx()
zackarno Feb 14, 2025
d9e84da
rm default args
zackarno Feb 14, 2025
3359a87
change unset to july
zackarno Feb 14, 2025
cb17d60
remove get_run_date() func
zackarno Feb 14, 2025
4c482ab
breakdown ifelse
zackarno Feb 14, 2025
cfd20cd
change obj name from run_date to run_date_set to rm any issue w/ arg …
zackarno Feb 14, 2025
4ac1ae9
add m_choro print
zackarno Feb 14, 2025
c4700f9
rm magrittr
zackarno Feb 14, 2025
3addfdf
change env var to WHEN_TO_MONITOR
zackarno Feb 14, 2025
4914b7b
try using gghdx::theme_hdx() instead of gghdx()
zackarno Feb 14, 2025
a004553
wipe gghdx() calls
zackarno Feb 14, 2025
d9a3048
try setting default to postrera
zackarno Feb 14, 2025
2e635c6
put back to and subject in email
zackarno Feb 14, 2025
eb7d593
get rid of double subject in smtp_send
zackarno Feb 14, 2025
a5b07a7
try sysfonts
zackarno Feb 14, 2025
0841ca9
try making map in rmd instead of .r
zackarno Feb 14, 2025
0676b27
m_choro back to R script
zackarno Feb 14, 2025
da42a97
smtp_send(verbose=TRUE)
zackarno Feb 14, 2025
0f49678
try ssl = F just for fun
zackarno Feb 14, 2025
21ed5bc
clean up all comments yaml
zackarno Feb 14, 2025
c0ec7d7
ssl=TRUE... doesnt make much diff
zackarno Feb 14, 2025
86d1f72
try add a new install.packages for curl/httr
zackarno Feb 15, 2025
5c17849
minor indentation adjustment
zackarno Feb 15, 2025
7cf9ce1
try adding remotes install first
zackarno Feb 15, 2025
5559269
add update script back into yaml - to be tried after caching
zackarno Feb 15, 2025
6c931cd
update creds
t-downing Feb 26, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions .github/depends_2025.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
#' requirements for 2025 monitoring.
#'
#'
required_pkgs <- c(
"RNetCDF", "terra", "sf", "dplyr", "exactextractr", "lubridate", "ggplot2",
"ggrepel", "tidyr", "janitor", "stringr", "glue", "rlang", "purrr", "forcats",
"readr", "gt", "arrow", "geoarrow", "AzureStor", "ncmeta", "logger", "blastula",
"tibble", "showtext", "remotes", "box", "dbplyr", "RPostgres", "curl"
)


install.packages(required_pkgs)

# Ensure remotes package is installed before using it
if (!requireNamespace("remotes", quietly = TRUE)) {
install.packages("remotes")
}

remotes::install_github("OCHA-DAP/cumulus")

# Additional packages
install.packages("gghdx")

# # List of required packages
# required_pkgs <- c(
# "RNetCDF",
# "terra",
# "sf",
# "dplyr",
# "exactextractr",
# "lubridate",
# "ggplot2",
# "ggrepel",
# "tidyr",
# "janitor",
# "stringr",
# "glue",
# "rlang",
# "purrr",
# "forcats",
# "readr",
# "gt",
# "arrow",
# "geoarrow",
# "AzureStor",
# "ncmeta",
# "logger",
# "blastula",
# "tibble",
# "showtext",
# "remotes",
# "box",
# "dbplyr",
# "RPostgres",
# "curl"
# )
# install.packages(required_pkgs)
#
# install.packages(c(
# "gghdx"
# )
# )
#
# remotes::install_github(
# repo = "OCHA-DAP/cumulus"
# )
# install.packages("gghdx")
# # once we have `{remotes}` should be able to install_github


8 changes: 8 additions & 0 deletions .github/depends_additional_2025.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
install.packages(c(
"dbplyr"
)
)

remotes::install_github(
repo = "OCHA-DAP/cumulus"
)
98 changes: 98 additions & 0 deletions .github/workflows/cadc_2025_monitoring.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
# This workflow uses actions to automatically run the dry corridor drought trigger monitoring.
# script.

name: cadc-2025-trigger-monitoring

on:
workflow_dispatch:
inputs:
EMAIL_WHO:
required: true
type: choice
default: 'core_developer'
options:
- core_developer
- developers
- internal_chd
- internal_ocha
- full_list
WHEN_TO_MONITOR:
required: true
type: choice
default: 'last_primera'
options:
- last_primera
- last_postrera
- current

jobs:
monitor:
runs-on: ubuntu-latest
env:
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
CHD_DS_EMAIL_USERNAME: ${{ secrets.DSCI_AWS_EMAIL_USERNAME }}
CHD_DS_EMAIL_PASSWORD: ${{ secrets.DSCI_AWS_EMAIL_PASSWORD }}
CHD_DS_HOST: ${{ secrets.DSCI_AWS_EMAIL_HOST }}
CHD_DS_PORT: 465
DS_AZ_BLOB_DEV_SAS: ${{ secrets.DSCI_AZ_BLOB_DEV_SAS_WRITE }}
DS_AZ_DB_PROD_UID_WRITE: ${{ secrets.DSCI_AZ_DB_PROD_UID_WRITE }}
DS_AZ_DB_PROD_HOST: ${{ secrets.DSCI_AZ_DB_PROD_HOST }}
DS_AZ_DB_PROD_PW_WRITE: ${{ secrets.DSCI_AZ_DB_PROD_PW_WRITE }}
DS_AZ_DB_PROD_NAME: ${{ secrets.DS_AZ_DB_PROD_NAME }}
EMAIL_WHO: ${{ inputs.EMAIL_WHO || 'core_developer' }}
WHEN_TO_MONITOR: ${{ inputs.WHEN_TO_MONITOR || 'current' }}

services:
postgres:
image: postgres:latest
env:
POSTGRES_USER: ${{ secrets.DS_AZ_DB_PROD_UID_WRITE }}
POSTGRES_PASSWORD: ${{ secrets.DS_AZ_DB_PROD_PW_WRITE }}
POSTGRES_DB: ${{ secrets.DS_AZ_DB_PROD_NAME }}
POSTGRES_HOST: localhost
ports:
- 5432:5432
# needed because the postgres container does not provide a healthcheck
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5

steps:
- uses: actions/checkout@v3
- name: Set up R
uses: r-lib/actions/setup-r@v2
with:
r-version: '4.x'
- uses: r-lib/actions/setup-pandoc@v2

- name: Install System Dependencies
run: |
sudo apt-get update
sudo apt-get install -y \
libxml2-dev \
libgdal-dev \
libproj-dev \
libgeos-dev \
libudunits2-dev \
libsodium-dev \
libfreetype6-dev \
libcurl4-openssl-dev

- name: Cache R dependencies
id: cache-r-deps
uses: actions/cache@v4
with:
path: ${{ env.R_LIBS_USER }}
key: R-dependency-cache-${{ hashFiles('.github/depends_2025.R') }}

- name: Install R dependencies
if: steps.cache-r-deps.outputs.cache-hit != 'true'
run: |
Rscript -e 'install.packages("remotes")'
Rscript .github/depends_2025.R

- name: Run R-script-test-env-vars
shell: bash
run: Rscript ./src/monitoring_2025/test_gha_settings.R

- name: Run R-script-cadc-2025-trigger-monitoring
shell: bash
run: Rscript ./src/monitoring_2025/update_activation_status.R
Loading
Loading