tile 25529 archive 20221008 vs 20221011

[sbailey]

It appears that tile 25529 was observed on 20221008, passed threshold and was QA accepted and archived on 20221013 based upon the LASTNIGHT=20221008 processing, i.e.

daily/tiles/cumulative/25529/20221008 -> ../../archive/25529/20221013

However, in the meantime it was reobserved on 20221011, though there was a delay in the processing and daily/tiles/cumulative/25529/20221011 didn't appear until October 13, presumably after the 20221008 version had been archived. However, some update of surveyops/trunk/ops/tiles-specstatus.ecsv set LASTNIGHT=20221011 even though that is not the LASTNIGHT that is actually archived.

The 20221008 data passed threshold, but our archived/mtl version isn't actually the full set of data we have and is inconsistent with what is in tiles-specstatus.ecsv. Should we re-archive this tile based upon 20221011?

Thanks for @weaverba137 for noticing this while doing jura QA

[geordie666]

I say yes to re-archiving this tile! I'll let @schlafly comment, too, though.

[schlafly]

Thanks Stephen, Ben! Good catch.

When this was observed on 20221011, it had EFFTIME_ETC = 93, not making the cuts. So it was right for the system to think it should be reobserved in the absence of an updated EFFTIME_SPEC from the pipeline. QANIGHT was 20221008, so Adam QAed based on only the 20221008 data, which makes sense, and that was the only data that was archived.

I haven't looked to figure out what was holding up pipeline processing, but I think this is all okay insofar as bright tiles frequently end up having more EFFTIME than the ETC expects, and if we're having trouble getting things processed this can lead us to QA and archive data that passes our cuts but is not all of the data we have taken. I don't think we need to change anything about our procedures here.

We should clarify what we expect LASTNIGHT to be. Currently it is just the last night a tile was observed on. I guess we don't separately track ARCHIVENIGHT which would be the night that got archived, though we do track ARCHIVEDATE which is when we pressed the button to archive it.

I'm happy to re-archive and re-MTL this tile; would you be able to do that, Adam? I'm also happy to do nothing, honestly, and wait for this to get picked up as a larger Jura re-MTLing event.

[geordie666]

I'm happy to re-MTL 25529 if someone on the spectro pipeline team wants to re-archive it.

Given the desire to launch Jura soon, though, I too am happy for this to be picked up as part of a larger Jura-based re-MTLing event.

That these sorts of inconsistencies keep cropping up increasingly convinces me that we should re-MTL after/based on major data releases.