Added analysis code used to calibrate Cookiebox and grating spectrometer

[daniloefl]

This PR includes the addition of an ingredients directory in EXtra, in which self-contained analysis steps are to be performed. Two elements have been added:

1. CookieboxCalib calibrates eTOF data from a calibration run (or a union thereof).
2. GratingCalib calibrates grating spectrometer data from a background run and a calibration run (or a union thereof).

GratingCalib

This device takes a background run, and monochromated runs, subtracts the background average data, sorts the monochromated runs by the mono energy and optionally rotates them.

After this step, it fits a line mapping the sample index of the grating data to energy.

The apply function applies the background subtraction and rotation for a new run and returns an xarray.Dataset containing the transformed data with an energy axis.

CookieboxCalib

eTOFs provide a 1D trace for all pulses, which can be transformed into traces per pulse by the AdqRawChannel Extra component. However, the trace sample axis is meaningless and needs to be converted into energy using a non-linear function. Usually one data analysis run is taken in which the undulator energy is scanned to obtain this map, which is then applied in the actual analysis run.

The objective is to take a calibration run (where an energy scan is made) and from that run calculate a) the map to be used to interpolate eTOF energy axis from "sample number" into eV; and b) calculate the transmission correction, which corrects the appropriate intensity of the data, given that the eTOF does not measure electrons with equal probability in all energy ranges.

The concrete steps taken when the object is initialized (obj = CookieboxCalib(calib_run)) from a calibration run are:

• From a calibration run, derive a sample number (proportional to time-of-flight) to energy non-linear map.
• Estimate the transformation Jacobian and use it to correct the spectrum intensity distribution, so that the probability of photo-electrons observed in a range of energies agrees with the probability in the corresponding time-of-flight range.
• Calculate a transmission correction due to the eTOF quantum efficiency as "photo-electron integral/Auger+Valence integral".
• Calculate a normalization correction due to the pulse energy as taken from the XGM: "Auger+Valence integral/XGM pulse energy".

The concrete steps taken when the obj.apply(other_run) is called with a run to be calibrated are:

• Use derived non-linear map to interpolate energy axis per eTOF
• Subtract per energy offset per eTOF.
• Scale data by the inverse of the absolute value of the Jacobian per eTOF, following the change-of-variable theorem in statistics.
• Divide by the transmission per eTOF.
• Divide by the normalization correction.

The variables calculated can be visualized using obj.plot_diagnostics() and similar other functions for validation and other cross-checks.

The current code works with fine-tuning of the constructor arguments first_pulse_offset, auger_start_roi, start_roi and stop_roi. There is code already to make it detect them automatically if None is set, but it cannot be guaranteed to always work.

Further steps proposed previously by @philsmt which have not been applied yet:

• Split the constructor to allow some of the steps to be repeated. This has been done by creating a series of update_... member functions and appropriate setters and getters. The setters and getters update a variable and call the update functions to redo calculations after resetting the corresponding variables.
• Accept AdqRawChannel as input. This is possible by setting them as the value of the map given in tof_settings. One could then set tof_settings={'1_A': AdqRawChannel(...), '1_C': AdqRawChannel(...)} to us this functionality. Providing a run is still necessary, as other objects (XGM, undulator reading and scans) are needed.
• Possibly use pasha if it shows a benefit when rewriting the code. This was not done, but only because it requires previous knowledge of the size of the structures. I would prefer to keep it as it is, instead of adding complexity on the discovery of it for now. I may change my mind later, but I think this is not the main performance hog at the moment.

[codecov]

Codecov Report

Attention: Patch coverage is 18.57585% with 526 lines in your changes missing coverage. Please review.

Project coverage is 64.29%. Comparing base (48bcd3e) to head (1c5c28a).

Files with missing lines	Patch %	Lines
src/extra/ingredients/cookiebox.py	18.44%	526 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #284      +/-   ##
==========================================
- Coverage   74.07%   64.29%   -9.78%     
==========================================
  Files          23       25       +2     
  Lines        3020     3666     +646     
==========================================
+ Hits         2237     2357     +120     
- Misses        783     1309     +526     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[daniloefl]

PS: A test could be written by comparing how the calibration applied to a run differs from the results of the code applied previously to the same run. The issue is that we don't have access to data in those tests.
Other tests are possible, but without data, it is not clear if they help much.

[daniloefl]

The CookieboxCalib and GratingCalib ingredients can be tested using the notebooks in:

• /gpfs/exfel/data/scratch/danilo/EXtra/Test CookieboxCalib.ipynb
• /gpfs/exfel/data/scratch/danilo/EXtra/Test GratingCalib.ipynb