`read_spice_l2_fits` fails on files with lost telemetry

[ebuchlin]

Describe the bug

read_spice_l2_fits() fails on SPICE files with lost telemetry as it considers non-window HDUs that are present in the FITS file in this case as windows.

To Reproduce

• Download this file
• Run the following code:

from sunraster.instr.spice import read_spice_l2_fits
data = read_spice_l2_fits('fits/level2/2022/04/02/solo_L2_spice-n-ras_20220402T121536_V01_100664003-000.fits')

Trace:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-73449fe396d0> in <module>
----> 1 data = read_spice_l2_fits('fits/level2/2022/04/02/solo_L2_spice-n-ras_20220402T121536_V01_100664003-000.fits')

~/.local/lib/python3.9/site-packages/sunraster/instr/spice.py in read_spice_l2_fits(filenames, windows, memmap, read_dumbbells)
     55         filenames = [filenames]
     56     # Read first file.
---> 57     first_cubes = _read_single_spice_l2_fits(
     58         filenames[0], windows=windows, memmap=memmap, read_dumbbells=read_dumbbells
     59     )

~/.local/lib/python3.9/site-packages/sunraster/instr/spice.py in _read_single_spice_l2_fits(filename, windows, memmap, read_dumbbells, output, spice_id)
    203                 wcs = WCS(new_header)
    204                 # Define exposure times from metadata.
--> 205                 exp_times = u.Quantity(np.zeros(hdu.data.shape[-1]) + meta.get("XPOSURE"), unit=u.s)
    206                 # Define data cube.
    207                 data = hdu.data

TypeError: unsupported operand type(s) for +: 'float' and 'NoneType'
> /home/eric/.local/lib/python3.9/site-packages/sunraster/instr/spice.py(205)_read_single_spice_l2_fits()
    203                 wcs = WCS(new_header)
    204                 # Define exposure times from metadata.
--> 205                 exp_times = u.Quantity(np.zeros(hdu.data.shape[-1]) + meta.get("XPOSURE"), unit=u.s)
    206                 # Define data cube.
    207                 data = hdu.data

What happened?

SPICE files with lost TM have additional HDUs: LOST_TELEMETRY, PLNAPRXPIXLIST* and APRXPLNPIXLIST* (the stars are for an additional tag identifying the corresponding window), as specified in the SPICE Data Product Description Document.

read_spice_l2_fits() fails on these files because _read_single_spice_l2_fits() assumes that every HDU that is not VARIABLE_KEYWORDS is a window, then it tries to get the XPOSURE header from the LOST_TELEMETRY HDU, and there is no such header in a non-window HDU.

List of HDUs in the example file (with astropy.io.fits.hdu.hdulist.HDUList.info()):

Filename: fits/level2/2022/04/02/solo_L2_spice-n-ras_20220402T121536_V01_100664003-000.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  Mg IX 706 - Peak    1 PrimaryHDU     353   (160, 830, 50, 1)   float32   
  1  N IV 765 - Peak    1 ImageHDU       353   (160, 830, 50, 1)   float32   
  2  Ne VIII 770 - Peak    1 ImageHDU       353   (160, 830, 50, 1)   float32   
  3  Ly-gamma-CIII group (2/3)    1 ImageHDU       359   (160, 830, 64, 1)   float32   
  4  Ly-gamma-CIII group (3/3)    1 ImageHDU       359   (160, 830, 64, 1)   float32   
  5  Ly Beta 1025 - SH    1 ImageHDU       359   (160, 830, 62, 1)   float32   
  6  Ly Beta 1025 - LH    1 ImageHDU       359   (160, 830, 62, 1)   float32   
  7  O VI 1032 - Peak    1 ImageHDU       353   (160, 830, 34, 1)   float32   
  8  VARIABLE_KEYWORDS    1 BinTableHDU    279   1R x 19C   [160D, 160I, 160I, 160I, 160I, 160I, 160E, 160E, 160E, 160E, 5I, 5I, 5I, 5I, 5J, 5J, 5J, 5J, 3680A]   
  9  LOST_TELEMETRY    1 BinTableHDU     41   1R x 1C   [1D]   
 10  APRXPLNPIXLIST[Mg IX 706 - Peak]    1 BinTableHDU     98   2R x 5C   [1I, 1I, 1I, 1I, 1I]   
 11  LOSTPLNPIXLIST[Mg IX 706 - Peak]    1 BinTableHDU     98   30R x 5C   [1I, 1I, 1I, 1I, 1I]   

Expected behavior

read_spice_l2_fits() should read such files with no error.

Possible solutions (I am trying to confirm which one would be the best): when _read_single_spice_l2_fits() builds the list of windows, it should either

• exclude also the LOST_TELEMETRY, PLNAPRXPIXLIST* and APRXPLNPIXLIST* HDUs
• exclude also everything after the VARIABLE_KEYWORDS HDU
• include only the first NWIN HDUs

Screenshots

No response

System Details

==============================
sunpy Installation Information

General
#######
OS: Ubuntu (21.10, Linux 5.13.0-40-generic)
Arch: 64bit, (x86_64)
sunpy: 3.1.5
Installation path: /home/eric/.local/lib/python3.9/site-packages

Required Dependencies
#####################
numpy: 1.22.3
packaging: 21.3
parfive: 1.5.1
astropy: 5.0.4

Optional Dependencies
#####################
asdf: 2.8.1
beautifulsoup4: 4.9.3
cdflib: 0.3.20
dask: 2021.1.0+dfsg
drms: 0.6.2
glymur: 0.9.9
h5netcdf: 0.11.0
h5py: 3.6.0
matplotlib: 3.5.0
mpl-animators: 1.0.0
pandas: 1.4.2
python-dateutil: 2.8.1
reproject: 0.8
scikit-image: 0.18.1
scipy: 1.8.0
sqlalchemy: 1.4.34
tqdm: 4.57.0
zeep: Missing, need zeep>=3.4.0; extra == "all"

Installation method

pip

[ebuchlin]

Better way of listing all windows according to the SPICE pipeline manager: list the primary HDU plus all extensions with with XTENSION= ‘IMAGE '

[EdBehn]

Perhaps a solution would be only to include HDUs with type PrimaryHDU or ImageHDU. The others have type BinTableHDU.

[nabobalis]

The test file loads with #205

If you find other files that are broken, please do either reopen this issue or create a new one please!

This is in the sunraster release 0.4.2, if you update to that pip install -U sunraster (you will have to wait about 30 minutes before it shows up on PyPi for install).

[ebuchlin]

Note for later: this will probably break with a future version of the SPICE FITS files, in which additional distortion information will be encoded in two WCSDVARR HDUs (with EXTVER = j for axis j), which are also ImageHDUs according to the draft WCS IV paper.

[nabobalis]

I reopened this issue to make sure, we keep track of that!

[ebuchlin]

Now we have such files with WCSDVARR; I have opened issue #208 for this.