Percentile outlier QC #183
Conversation
* Implemented core functionality for season based threshold filter * Implemented script to compute thresholds using historical L1 (inferred) data * Added default thresholds * Updated setup.py to allow thresholds data file
ladsmund
force-pushed
the
features/percentile_qc
branch
from
6b45719
to
475805f
Compare
There was a problem hiding this comment.
All looks okay. The only thing is that there is an incompatibility with Python 3.8 in the process test (see screenshot below of Github Action read-out). We run on Python 3.8 on both azure-aws and glacio01, so either we need to solve this incompatibility or update everything to at least Python 3.9.
I think it is related to the syntax of this line including the type bool as a subscript of the output np.ndarray.
I tried changing it to this:
def get_season_index_mask(data_set: pd.DataFrame, season: str) -> np.ndarray
But then I get another error (unrelated to the previous syntax incompatibility). I guess this new error is linked to another compatibility issue to Python 3.8:
Traceback (most recent call last):
File "/home/pho/anaconda3/envs/pypromice/bin/get_l3", line 8, in <module>
sys.exit(get_l3())
File "/home/pho/anaconda3/envs/pypromice/lib/python3.8/site-packages/pypromice/process/get_l3.py", line 39, in get_l3
aws.process()
File "/home/pho/anaconda3/envs/pypromice/lib/python3.8/site-packages/pypromice/process/aws.py", line 78, in process
self.getL2()
File "/home/pho/anaconda3/envs/pypromice/lib/python3.8/site-packages/pypromice/process/aws.py", line 99, in getL2
self.L2 = toL2(self.L1A, vars_df=self.vars)
File "/home/pho/anaconda3/envs/pypromice/lib/python3.8/site-packages/pypromice/process/L1toL2.py", line 73, in toL2
ds = outlier_detector.filter_data(ds) # Flag and remove percentile outliers
File "/home/pho/anaconda3/envs/pypromice/lib/python3.8/site-packages/pypromice/qc/percentiles/outlier_detector.py", line 70, in filter_data
if not stid_thresholds.any():
File "/home/pho/anaconda3/envs/pypromice/lib/python3.8/site-packages/pandas/core/generic.py", line 1526, in __nonzero__
raise ValueError(
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
@ladsmund, have you tried running this PR on Python 3.8 to check it runs? I'm just cloning your branch, installing it into my Python environment and then running the get_l3 function on a random station.
|
All runs fine now @ladsmund! Feel free to merge. |
No description provided.