integration test failing

[sbailey]

Although the full suite of unit tests now pass across various desi packages, the end-to-end integration test still fails. Adapted from etc/cron_dailyupdate.sh:

source /global/cfs/cdirs/desi/software/desi_environment.sh 22.1a
module swap desispec/0.51.3

export DAILYTEST_ROOT=$SCRATCH/dailytest
export PIXPROD=dailytest
export DESI_SPECTRO_DATA=$DAILYTEST_ROOT/spectro/sim/$PIXPROD
export DESI_SPECTRO_SIM=$DAILYTEST_ROOT/spectro/sim
export PRODNAME=dailytest
export SPECPROD=dailytest
export DESI_SPECTRO_REDUX=$DAILYTEST_ROOT/spectro/redux

#- Cleanup from previous tests
simdir=$DESI_SPECTRO_SIM/$PIXPROD
outdir=$DESI_SPECTRO_REDUX/$PRODNAME
rm -rf $simdir
rm -rf $outdir

#- Run the integration test
mkdir -p $simdir
mkdir -p $outdir

#- temporarily turn off while main survey target selection settles
python -m desispec.test.old_integration_test &> $outdir/dailytest.log
...
INFO:io.py:272:write_simspec: Wrote /global/cscratch1/sd/sjbailey/dailytest/spectro/sim/dailytest/20140203/00000000/simspec-00000000.fits
INFO:fibermap.py:261:write_fibermap: iotime 0.140 sec to write fibermap-00000000.fits at 2022-02-01T16:47:10.123482
INFO:util.py:98:runcmd: Tue Feb  1 16:47:10 2022
ERROR:util.py:107:runcmd: missing output /global/cscratch1/sd/sjbailey/dailytest/spectro/redux/dailytest/preproc/20140203/00000000/fibermap-00000000.fits
ERROR:util.py:107:runcmd: missing output /global/cscratch1/sd/sjbailey/dailytest/spectro/redux/dailytest/preproc/20140203/00000000/simspec-00000000.fits
CRITICAL:util.py:110:runcmd: FAILED outputs newexp-random --program flat --nspec 5 --night 20140203 --expid 0

It wrote the fibermap and simspec outputs to $simdir, but the wrapper was expecting them under $outdir. That doesn't sound too hard to fix, but there may be other lurking issues in directories and format assumptions because the integration tests date back to when we through ICS would provide the fibermap, but instead they provide the ingredients in various files and we put together the fibermap ourselves.

There could be value in reviving this simulation-based integration test to ensure that simulations still work for producing outputs useful for the pipeline, but it would also be valuable to augment that with an end-to-end processing test on real data that could be used as an afternoon checkout before proceeding with the nightly (daily) pipeline using whatever updates were made during the day. The main trick there is to do it fast, while still covering enough to be useful.

More context: the above is the "old_integration_test", which is a little closer to the current pipeline than the "integration_test", which tests a big-job MPI-centric workflow that we're not actively using for real data. More cleanup is possible...

@moustakas can you look into this?

In the meantime the nightly tests at NERSC will run unit tests but not the full integration test.