0.7.290: missing SBCDEN_P value in combined FP_FP 24BQ11-Oct28/3DA7AA0F8Ba_pp.fits

[larnoldgithub]

Hi @njcuk9999
The pp file combining the FP_FP of the last night of the October run has no kw SBCDEN_P (FP density value) although all individual raw files have a correct value (value is 1.19 or 1.20)
Only one night in 24BQ11 is affected, but It happened with several nights from the september run.

Individual files:

FILE                                                                   	DPRTYPE 	SBCDEN_P	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091805a_pp.fits	DARK_FP 	1.19    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091806a_pp.fits	DARK_FP 	1.19    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091807a_pp.fits	FP_FP   	1.19    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091808a_pp.fits	FP_FP   	1.19    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091809a_pp.fits	FP_FP   	1.19    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091810a_pp.fits	FP_FP   	1.19    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091811a_pp.fits	FP_FP   	1.19    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091875a_pp.fits	DARK_FP 	1.20    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091876a_pp.fits	DARK_FP 	1.20    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091877a_pp.fits	FP_FP   	1.20    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091878a_pp.fits	FP_FP   	1.20    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091879a_pp.fits	FP_FP   	1.20    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091880a_pp.fits	FP_FP   	1.20    	
/data/spirou4/apero-data/quicklook290/tmp/24BQ11-Oct28/3091881a_pp.fits	FP_FP   	1.20   

pre_processed combined file:

FILE DPRTYPE SBCDEN_P
/data/spirou4/apero-data/quicklook290/red/24BQ11-Oct28/3DA7AA0F8Ba_pp.fits FP_FP

I'm using the value of the FP density to monitor the instrument, I never had this issue with the 0.6.132

[larnoldgithub]

Note also the kw is also missing in all subsequent reduced files 3DA7AA0F8Ba*

[njcuk9999]

Its not missing, by definition any key that changes moves to an extension COMBINE_TABLE due to not being able to have both values in the header.

For a few keys I define a "combing method" i.e. exptime etc where we take a mean or a sum or something.

If you want they key to be combined I just need to know what you want for it (i.e. take the mean of all values) otherwise you can find it in the COMBINE_TABLE extension in the HASH_pp.fits file.

Also the COMBINE_TABLE isn't passed to products using that file so subsequent processed files wont have this extension or the header key.

[larnoldgithub]

ok I understand. Maybe for the density value it's worth I chnage my code and look at the individual values if the recombined is missing.

[njcuk9999]

I'll look in the code to see whether if I make it give the "mean" value it will also stay in the combined table - if we could have both that seems to be the best solution. I guess it depends on what you are checking for with the density - if a sudden change needs to be flagged and this can happen between the combine frames you definitely want to be checking the COMBINE_TABLE.

[larnoldgithub]

No need, I have written a specific code to monitor the instrument. the density value is something important to check so what I do is: if the value is not present in the primary header I go to the COMBINE_TIME of the PP file, get the values, check max-min. If max-min < 0.02 this is normal dispersion and I go ahead with the mean, which I use for the following in the script. If it's >0.02, the value is set to N/A and it's flagged with a missing dot in the plot below, also a message when gathering the data for a run.

I generate csv files for every run to get infos like the density, the snr in the C channel etc. here is a plot showing the FP flux in the C channel for the last three run of 24B. The flux in the calibration FP_FP are the purple dots, and many were missing because to that 'non-unique' daily value of the density . In gray is the flux with the density set to 3.0, the max value.