Skip to content

Commit

Permalink
Revert info_per_bit tolerance to 1.5
Browse files Browse the repository at this point in the history
The xbitinfo implementation uses a tolerance factor of 1.5. I lowered the tolerance to 1.1, because I was getting poor results with my test data, but it was pointed out that the problem was that my test data were quantized. Quantization is an open issue with xbitinfo too, and we should address it first there, then patch the codec.
  • Loading branch information
thodson-usgs authored Mar 1, 2024
1 parent e9efcc4 commit 9fe6e66
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion numcodecs/bitinfo.py
Original file line number Diff line number Diff line change
Expand Up @@ -274,7 +274,7 @@ def get_keepbits(info_per_bit, inflevel=0.99):

def _cdf_from_info_per_bit(info_per_bit):
"""Convert info_per_bit to cumulative distribution function"""
tol = info_per_bit[-4:].max() * 1.1 # reduced from 1.5
tol = info_per_bit[-4:].max() * 1.5
info_per_bit[info_per_bit < tol] = 0
cdf = info_per_bit.cumsum()
return cdf / cdf[-1]

0 comments on commit 9fe6e66

Please sign in to comment.