populate dr1/survey/fiberassign

[weaverba137]

No description provided.

[sbailey]

In public/dr1/survey/fiberassign/ I created links to ../../../edr/survey/fiberassign/SV1/ and SV3/ . The equivalent SV2 link already existed.

I also moved + linked the additional needed dates from survey/fiberassign/special/ -> public/dr1/survey/fiberassign/special/, created their checksums: 20210708, 20210918 20210919 20210923 20211030 20211228 20220120 20220324 .

@weaver @weaverba137 that leaves the main survey tiles. What we need in pseudocode is

for every SURVEY="main" TILEID in $DESI_ROOT/spectro/redux/iron/tiles-iron.fits:
    TILEGROUP = f'{TILEID/1000:03d}'  # 3-digit zero-padded int(TILEID/1000)
    TILESTR = f'{TILEID:06d}'  # 6-digit zero-padded TILEID
    create $CFS/desi/public/dr1/survey/fiberassign/main/$TILEGROUP if needed
    move + link all $CFS/desi/survey/fiberassign/main/$TILEGROUP/*$TILESTR* files to $CFS/desi/public/dr1/survey/fiberassign/main/$TILEGROUP

Note that some of the files are of the form ${TILESTR}-*.* while others are *-${TILESTR}.*. Some $TILEGROUP directories might result in all files being moved + linked, but we still need to do this at the file level (not the directory level) since there is the possibility that additional tiles will be added to the same $TILEGROUP in the future as part of daily survey operations.

@weaver @weaverba137 if that is ambiguous/confusing please ping me on Slack and let's sort details out by Slack/Zoom if needed.

[weaverba137]

@sbailey, at first reading, I think I understand what is needed.

Also note, I am @weaverba137, not @weaver.

[weaverba137]

@sbailey, some, but not all of the special nights in survey/fiberassign/dr1 have a 082/ subdirectory. Are those archival? One way or another the files in those directories need to be checksummed as well (reminder to self).

[weaverba137]

Also, special does not match the pattern above. It appears to be YYYYMMDD/tilegroup instead of just tilegroup. Is that intentional?

[weaverba137]

Other items after moving the files:

• Change owner/group to desi/desi.
• Change permissions to archival values.
• Create checksums.
• Data model. Is this really what we want? There are potentially three different directory structures here:
  • SV1/SV2/SV3: YYYYMMDD/*files*
  • special: YYYYMMDD/*some files*, YYYYMMDD/082/*other files*
  • main: TILEGROUP/*files*

[weaverba137]

survey/fiberassign/main now has permissions set correctly and checksums. The remaining question has to do with differences in the data models for different surveys.

[sbailey]

For the data model, I suggest pragmatism:

• document the main/TILEGROUP files
• note in the datamodel that SV1/SV2/SV3/special exist in a different form (and for special, a potentially evolving form)
• do that as part of a desidatamodel ticket so that this ticket can remain focused on getting the files into place

@sbailey, some, but not all of the special nights in survey/fiberassign/dr1 have a 082/ subdirectory. Are those archival? One way or another the files in those directories need to be checksummed as well (reminder to self).
Also, special does not match the pattern above. It appears to be YYYYMMDD/tilegroup instead of just tilegroup. Is that intentional?

It appears that more recent special/DATE started using the TILEGROUP concept such that survey/fiberassign/special/DATE/TILEGROUP is analogous to survey/fiberassign/main/TILEGROUP, while still using the DATE to group batches of special tiles. It just happens that special tiles are all in the 082nnn range and thus have the 082/ subdirectory as the TILEGROUP. These are archival in the sense that we want to keep them, albeit while being flexible about the evolving structure and needs of one-off special program planning.

@weaverba137 please add the checksums. I think that is the last remaining data task for this ticket; the rest can be passed to desidatamodel, right?

[weaverba137]

OK, that sounds like a good way to proceed.

I think that is the last remaining data task for this ticket; the rest can be passed to desidatamodel, right?

Yes, I agree with that.

[weaverba137]

I've added the missing checksums to the special directories and opened desihub/desidatamodel#206.