Checks are ignored if any of the columns is null (nan)?

[varszpet]

The code below does not show any Check rule violation, even though 40 is more then 30.

import pandas as pd
import numpy as np
import pandera as pa

df = pd.DataFrame(
    {'a': [100, 30, 30, 100],
     'b': [20, 40, 1, 30],
     'c': [2, np.nan, 2, 1]
     })
schema = pa.DataFrameSchema(
    columns={
        'a': pa.Column(pa.Int),
        'b': pa.Column(pa.Int),
    },
    checks=[
        pa.Check(lambda x: x['a'] > x['b']),
    ]
)
schema.validate(df)  

[jeffzi]

That's because the argument ignore_na of Check is True by default. For DataFrame checks, if set to True, any row with at least a NaN value will be ignored.

pa.Check(lambda x: x['a'] > x['b'], ignore_na=False) raises an error as you expected.

[varszpet]

Yes, true. But unfortunately it raises an error in case both of the inspected cells are nulls. I think it shold ignore it because a null value is not bigger then another null value.

[jeffzi]

Pandera will raise an error for the row with NaNs only if ignore_na=False, which is correct since NaN > Nan is not True and doesn't pass the check.

import pandas as pd
import numpy as np
import pandera as pa

df = pd.DataFrame({"a": [np.nan, 30.0], "b": [np.nan, 40.0]})
df
#>       a     b
0   NaN   NaN
1  30.0  40.0

schema = pa.DataFrameSchema(
    columns={
        "a": pa.Column(pa.Float, nullable=True),
        "b": pa.Column(pa.Float, nullable=True),
    },
    checks=[pa.Check(lambda x: x["a"] > x["b"], ignore_na=True)],
)
schema.validate(df)
#> Traceback (most recent call last):
#> ...
#> SchemaError: <Schema DataFrameSchema(
#>     columns={
#>         'a': <Schema Column(name=a, type=float)>
#>         'b': <Schema Column(name=b, type=float)>
#>     },
#>     checks=[
#>         <Check <lambda>>
#>     ],
#>     coerce=False,
#>     pandas_dtype=None,
#>     index=None,
#>     strict=False
#>     name=None,
#>     ordered=False
#> )> failed element-wise validator 0:
#> <Check <lambda>>
#> failure cases:
#>   column  index  failure_case
#> 0      a      1          30.0
#> 1      b      1          40.0

^{Created on 2021-04-16 by the reprexpy package}

Did I misunderstand your question? If that's the case, could you provide an example?

[varszpet]

Sorry I was wrong. Of course null is not bigger then another null, so it is okay, if it is reported as an error. What I want to have is to ignore all those cases from inspection when both (or any) of the inspected cells are nulls. Unfortunately np.nan does not equal np.nan, so it is not good to have x['a'] >= x['b']. I think it is a valid use case, to have something checked only if the values exist. Do you have any idea for a workaround?

[cosmicBboy]

What I want to have is to ignore all those cases from inspection when both (or any) of the inspected cells are nulls

If you want to ignore the rows where any of "a" or "b" are nulls (but not any other column), you can do something like:

# first drop rows where "a" or "b". downside of this is that the returned series index no longer matches the original
# series, meaning some of the error-reporting won't be as useful.
pa.Check(lambda x: x.dropna(how="any", subset=["a", "b"]).pipe(lambda y: y["a"] > y["b"]), ignore_na=False)

# mask rows containing nans in a, b with True
pa.Check(lambda x: (x["a"] > x["b"]).pipe(lambda s: s.mask(x["a", "b"].isna().any(axis=1))), ignore_na=False)

I wonder if ignore_na should support:any and all, with all being the default (and True being equivalent to any, to preserve current behavior)