SEAS5 z-score and ASAP warnings #13

t-downing · 2025-02-14T18:59:36Z

Two main things:

Use SEAS5 z-score instead of absolute values for forecast trigger
- Processing of z-score and threshold calc all in analysis/ecmwf_switch_zcore
- Main things to check are that I'm doing the z-score calculation correctly, and plotting the RP correctly
- I also do some quick RP modeling- not critical that this is perfect, it's really just tweaking the thresholds.
Use JRC ASAP warnings for observational trigger
- Basic processing of raw file from ASAP in src/datasources/asap.py
- Everything else in exploration/asap_warnings
  - Main things to check is that I'm reading in the warnings correctly, and calculating the RP correctly
- Also I combine the forecast and observational triggers in the last section of exploration/asap_warnings
  - Main thing to check is that I'm calculating the combined RP correctly

Corresponding slide deck here.

hannahker

Still reviewing the code in more detail, but a couple reproducibility points and curious generally about SPI-3... Was doing a bit of reading and isn't SPI-3 calculated a bit differently than a z-score? Wouldn't we want to fit the precip points to a gamma distribution, normalize, etc? Not questioning the validity of a z-score for these purposes, but just wondering if it's misleading to conflate the two. It also looks like Copernicus has a SPI-3 forecast product based on SEAS5. Would it be worth investigating this instead of calculating ourselves?

hannahker · 2025-02-21T22:03:04Z

exploration/asap_warnings.md

+                ].min(axis=1)
+            else:
+                raise ValueError("invalid crop_range")
+            if biomass_only:


Not defined

And note doesn't seem to work if I set this to True:

KeyError Traceback (most recent call last) File ~/Desktop/pa-aa-bfa-drought/venv/lib/python3.13/site-packages/pandas/core/indexes/base.py:3805, in Index.get_loc(self, key) 3804 try: -> 3805 return self._engine.get_loc(casted_key) 3806 except KeyError as err: File index.pyx:167, in pandas._libs.index.IndexEngine.get_loc() File index.pyx:175, in pandas._libs.index.IndexEngine.get_loc() File pandas/_libs/index_class_helper.pxi:70, in pandas._libs.index.Int64Engine._check_type() KeyError: 'indicator' The above exception was the direct cause of the following exception: KeyError Traceback (most recent call last) Cell In[16], line 27 25 if True: 26 dff = dff["indicator"] ---> 27 dff = dff[dff["indicator"] >= alert_level] 28 adm_counts = ( 29 dff.groupby("year") 30 .agg( (...) 34 .reset_index() 35 ) 36 display(adm_counts) File ~/Desktop/pa-aa-bfa-drought/venv/lib/python3.13/site-packages/pandas/core/series.py:1121, in Series.__getitem__(self, key) 1118 return self._values[key] 1120 elif key_is_scalar: -> 1121 return self._get_value(key) 1123 # Convert generator to list before going through hashable part 1124 # (We will iterate through the generator there to check for slices) 1125 if is_iterator(key): File ~/Desktop/pa-aa-bfa-drought/venv/lib/python3.13/site-packages/pandas/core/series.py:1237, in Series._get_value(self, label, takeable) 1234 return self._values[label] 1236 # Similar to Index.get_value, but we do not fall back to positional -> 1237 loc = self.index.get_loc(label) 1239 if is_integer(loc): 1240 return self._values[loc] File ~/Desktop/pa-aa-bfa-drought/venv/lib/python3.13/site-packages/pandas/core/indexes/base.py:3812, in Index.get_loc(self, key) 3807 if isinstance(casted_key, slice) or ( 3808 isinstance(casted_key, abc.Iterable) 3809 and any(isinstance(x, slice) for x in casted_key) 3810 ): 3811 raise InvalidIndexError(key) -> 3812 raise KeyError(key) from err 3813 except TypeError: 3814 # If we have a listlike key, _check_indexing_error will raise 3815 # InvalidIndexError. Otherwise we fall through and re-raise 3816 # the TypeError. 3817 self._check_indexing_error(key) KeyError: 'indicator'

hannahker · 2025-02-21T22:24:24Z

src/datasources/asap.py

+):
+    specific_path = None
+    if data_type == "raw":
+        if data_type == "raw":


Repetition here

hannahker · 2025-02-21T22:28:09Z

src/utils/blob_utils.py

@@ -12,7 +12,7 @@
 from azure.storage.blob import ContainerClient, ContentSettings
 from dotenv import load_dotenv

-load_dotenv()
+load_dotenv(override=True)


Probably want to avoid overwriting other variables you have already defined. Which is this for?

This was just for checking things with the new blob creds (I generally don't use load_dotenv and have them all stored on my .zshenv so they're pre-loaded into the terminal, so this was to quickly overwrite the old ones without having to restart the terminal running my Jupyter).

Anyways, this is generally not needed so I'll remove it.

t-downing · 2025-02-24T01:49:34Z

Still reviewing the code in more detail, but a couple reproducibility points and curious generally about SPI-3... Was doing a bit of reading and isn't SPI-3 calculated a bit differently than a z-score? Wouldn't we want to fit the precip points to a gamma distribution, normalize, etc? Not questioning the validity of a z-score for these purposes, but just wondering if it's misleading to conflate the two. It also looks like Copernicus has a SPI-3 forecast product based on SEAS5. Would it be worth investigating this instead of calculating ourselves?

Yeah you're right, I think it's misleading to call this simplified version the SPI. To be honest, we could just use the anomaly instead of the z-score or SPI; I think this is more easily understood anyways. The only requirement was that we don't use the absolute values of the precipitation forecast, since these shouldn't be compared across areas with different precipitation. So the anomaly would fit the bill too. Anyways we can discuss further on a call.

hannahker · 2025-02-24T20:10:04Z

analysis/ecmwf_switch_zscore.md

+```
+
+```python
+df_seas5 = df_seas5_zscore_q.copy()


Do we need this?

hannahker · 2025-02-24T21:11:33Z

analysis/ecmwf_switch_zscore.md

+### Plot historical activations
+
+```python
+thresh_3, thresh_7


These aren't defined yet

hannahker · 2025-02-24T21:14:06Z

analysis/ecmwf_switch_zscore.md

+Fixing thresholds based on modeled RP (values calculated a few cells down)
+
+```python
+thresh_3 = -0.9


Maybe a bit dangerous to hard code here? If these are pulled from the modelled RP can we un-hard code them?

hannahker · 2025-02-24T21:17:26Z

analysis/ecmwf_switch_zscore.md

+```
+
+```python
+df_pivot.plot(x="year", y=["issued_3", "issued_7"])


Hmmm ok yes looking at this plot again I think 2000 seems like a very reasonable cut off year

hannahker · 2025-02-24T21:26:21Z

src/monitor_trigger.py

 import numpy as np
+import ochanticipy.utils.raster  # noqa: F401


Would still be good to remove the ochanticipy dependencies if possible

hannahker

Ok I think looks good to merge here to me! I've confirmed all the code runs and left some notes on reproducibility. Methodologically, everything seems logical to me, although I'll note that some of the plotting and ASAP data processing is challenging for me to follow (mostly because I'm not super familiar with that dataset).

As we discussed, I think it'd be interesting to look into the precipitation anomaly in addition to the z-score. I have the gut feeling that this could be more valid on a non-normal distribution than the z-score, although it's hard to pinpoint exactly why (and I'm still warming up my stats brain after some years of unuse 😆 ).

hannahker · 2025-02-24T21:32:48Z

exploration/asap_warnings.md

+```
+
+```python
+# this is literally just to make those little boxes on the plot


hannahker · 2025-02-24T21:35:17Z

exploration/asap_warnings.md

+def get_alert_gr_int(alert_gr_str):
+    try:
+        return int(alert_gr_str.removeprefix("Warning group "))
+    except ValueError:


This might potentially be too general and mask other errors that we aren't expecting... can we be more specific here?

hannahker · 2025-02-24T21:37:05Z

exploration/asap_warnings.md

+```python
+# set first possible trigger dekad to 3rd dekad of July
+min_dekad = 21
+# set last possible trigger dekad to 3rd dekad of July


Just a confusing comment here. This dekad is different?

t-downing added 2 commits February 14, 2025 09:39

raw warnings

df28d80

process raw warnings

46ec660

t-downing changed the base branch from main to switch-ecmwf February 14, 2025 18:59

t-downing added 3 commits February 17, 2025 15:10

asap processing

d320a31

combined rp

d70c101

comments

52ce5a3

t-downing changed the title ~~Asap vegetation~~ SEAS5 z-score and ASAP warnings Feb 19, 2025

update headers

67a9505

t-downing marked this pull request as ready for review February 19, 2025 23:52

t-downing requested a review from hannahker February 19, 2025 23:52

typo

28990ff

hannahker reviewed Feb 21, 2025

View reviewed changes

hannahker reviewed Feb 24, 2025

View reviewed changes

analysis/ecmwf_switch_zscore.md

```

```python

df_seas5 = df_seas5_zscore_q.copy()

Copy link

hannahker Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this?

hannahker reviewed Feb 24, 2025

View reviewed changes

analysis/ecmwf_switch_zscore.md

### Plot historical activations

```python

thresh_3, thresh_7

Copy link

hannahker Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These aren't defined yet

hannahker reviewed Feb 24, 2025

View reviewed changes

hannahker approved these changes Feb 24, 2025

View reviewed changes

t-downing added 2 commits February 25, 2025 11:49

further seas5 analysis

35e89ff

ipc quick analysis

fb95ab1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SEAS5 z-score and ASAP warnings #13

SEAS5 z-score and ASAP warnings #13

t-downing commented Feb 14, 2025 •

edited

Loading

hannahker left a comment

hannahker Feb 21, 2025

hannahker Feb 24, 2025

hannahker Feb 21, 2025

hannahker Feb 21, 2025

t-downing Feb 24, 2025

t-downing commented Feb 24, 2025

hannahker Feb 24, 2025

hannahker Feb 24, 2025

hannahker Feb 24, 2025

hannahker Feb 24, 2025

hannahker Feb 24, 2025

hannahker left a comment

hannahker Feb 24, 2025

hannahker Feb 24, 2025

hannahker Feb 24, 2025

		import numpy as np
		import ochanticipy.utils.raster # noqa: F401

SEAS5 z-score and ASAP warnings #13

Are you sure you want to change the base?

SEAS5 z-score and ASAP warnings #13

Conversation

t-downing commented Feb 14, 2025 • edited Loading

hannahker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t-downing commented Feb 24, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hannahker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t-downing commented Feb 14, 2025 •

edited

Loading