DM-46562: Add bps templates for LSSTCam/LSSTComCam/LATISS calibration creation. #292

erykoff · 2025-01-22T23:02:45Z

No description provided.

czwa · 2025-01-22T23:21:53Z

bps/templates/bps_flat.yaml

+  output: ${USER_CALIB_PREFIX}${INSTRUMENT}/calib/${TICKET}/${TAG}/flatGen-${FLAT_BAND}.${RERUN}
+  butlerConfig: ${REPO}
+  inCollection: ${USER_CALIB_PREFIX}${INSTRUMENT}/calib/${TICKET}/${TAG},${RAW_COLLECTION},${CALIB_COLLECTIONS}
+  dataQuery: ${!SELECTION_FLAT_BAND}


I think this exclamation point is a typo.

This is bash "indirection" which should allow one template to be used for all the flats with different selection variables. That's the idea, at least. I haven't gotten to testing this part yet!

czwa · 2025-01-22T23:23:31Z

bps/templates/README.md

+```
+export FLAT_BAND=g
+export SELECTION_FLAT_BAND=SELECTION_FLAT_g
+export SELECTION_FLAT_g="instrument='LSSTComCam' and selection-string"


This uses selection-string instead of selection_string used elsewhere. Not really a "bug" as this is documentation.

Actual comment: If we have a band variable, we should constrain the query to just that band. This will prevent future accidents where we generate a 15 input g flat and a 1 input r flat in the same collection because an endpoint was misread.

True but not trivial because we need to know the physical filter in the constraint, so I can't easily template that generally for LATISS/LSSTComCam/LSSTCam. But we can put strong recommendation that SELECTION_FLAT_g and friends include a physical filter constraint!

I don't think we need to know the physical_filter. I assume pipetask and butler query-dimension-records use the same resolver, so something like this should work:

butler query-dimension-records /repo/main exposure --where "instrument='LSSTComCam' AND physical_filter.band='g'" --limit 1 instrument id day_obs group physical_filter obs_id exposure_time dark_time observation_type observation_reason seq_num seq_start seq_end target_name science_program tracking_ra tracking_dec sky_angle azimuth zenith_angle has_simulated can_see_sky timespan (TAI) ---------- ------------- -------- -------------------- --------------- -------------------- ------------- --------------- ---------------- ------------------ ------- --------- ------- ----------- --------------- ----------- ------------ --------- ------- ------------ ------------- ----------- ------------------------------------------ LSSTComCam 2021040100003 20210401 CALSET_20210401_1601 g_07 CC_O_20210401_000003 1.0 1.5343017578125 flat flat 3 3 3 UNKNOWN unknown None None None None None False None [2021-04-01T16:24:36, 2021-04-01T16:24:37)

"instrument='LSSTComCam' AND band='g'" should work and is shorter.

When getting a flat from a calib repo that doesn't work, so I guess I had assumed we needed it in the constraint as well.

czwa · 2025-01-22T23:32:42Z

bps/templates/bps_linearizer.yaml

+  output: ${USER_CALIB_PREFIX}${INSTRUMENT}/calib/${TICKET}/${TAG}/linearizerGen.${RERUN}
+  butlerConfig: ${REPO}
+  inCollection: ${USER_CALIB_PREFIX}${INSTRUMENT}/calib/${TICKET}/${TAG},${RAW_COLLECTION},${CALIB_COLLECTIONS}
+  dataQuery: ${SELECTION_PTC_LINEARIZER-$SELECTION_PTC}


This has a typo as well.

More bash magic that is supposed to use $SELECTION_PTC_LINEARIZER if it is set and otherwise use $SELECTION_PTC. Also not yet tested if the bps vars support this...

czwa · 2025-01-22T23:32:47Z

bps/templates/README.md

+Environment Variables
+---------------------
+
+These are the environment variables that must be used. Examples are substituted in.


This should also note that selection_string is a dummy variable here, and that it needs to be filled with something explicit. I was going to suggest putting in the information being used on DM-46562 (the ticket used below), but I see that's this ticket adding the BPS templates, not a generation ticket.

czwa

I think most of my comments are me not fully understanding how the variables work here.

czwa · 2025-01-28T21:38:22Z

bps/templates/README.md

+* `export SELECTION_PTC_LINEARIZER=$SELECTION_PTC`: The selection of raws to generate the linearizer; usually will be the same as the PTC selection.
+* `export SELECTION_PTC_BFK=$SELECTION_PTC`: The selection of raws to generate the brighter-fatter kernel; usually will be the same as the PTC selection.
+* `export SELECTION_PTC_CTI=$SELECTION_PTC`: The selection of raws to generate the charge-transfer-inefficiency dataset; usually will be the same as the PTC selection.
+* `export SELECTION_FLAT_g="instrument='LSSTComCam and selection_string": The selection of raws to generate the g-band flat. See below for additional info.


Missing trailing "`".

czwa · 2025-01-28T21:40:49Z

bps/templates/README.md

+
+These are the environment variables that must be used. Examples are substituted in.
+
+* `export USER_CALIB_PREFIX=""` or `export USER_CALIB_PREFIX=u/erykoff/`: Set to a blank string for official calibrations, or the user collection prefix (**with trailing slash**).


Maybe "an empty string"?

czwa · 2025-01-28T21:42:46Z

bps/templates/README.md

+* `export CALIB_COLLECTIONS=LSSTComCam/calib/DM-46825`: Comma-separated list of curated or previously generated calibration collections to use as a starting point.
+* `export TAG=newCalibs`: A human-readable tag to help indicate why a set of calibs were built (should also be findable from the ticket name).
+* `export RERUN=20250122a`: The rerun name to indicate when the calibrations were generated.
+* `export BOOTSTRAP_RUN_NUMBER=1`: The bootstrap run number ensures that the bootstrap run collections are unique in case of an initial mistake. (Because the run name for bootstraps are deterministic without a timestamp they cannot be rerun into the same collection).


This is a bit fragile, as the biasBootstrap could have run successfully, but the darkBootstrap needed N iterations. Maybe a comment here to be aware of that?

czwa · 2025-01-28T21:55:50Z

bps/templates/README.md

+
+Overview
+--------
+


Can this section also have a pointer to the "how to run BPS" documentation? We have the "check variables" example below, but no "here's how to run this after you've set up the variables."

czwa · 2025-01-28T21:56:41Z

bps/templates/README.md

+uses python `os.path.expandvars()` which is not as sophisticated. The most
+relevant limitation is that `envsubst` will replace an unset env var with a
+blank string, while `expandvars()` will leave that raw
+(e.g. `${USER_CALIB_PREFIX}` can be left verbatim). Nevertheless, this is a


alkjsdaisoghg.

czwa · 2025-01-28T21:57:38Z

bps/templates/README.md

+
+Due to limitations in BPS environment variable expansion, we have one template
+file for each band. There are template files for each of ugrizy. Note that it is
+**strongly** recommended that the relevant physical filter be used in the


Since we have per-filter flat files, can't be bake the band into the dataQuery?

czwa · 2025-01-28T22:02:07Z

bps/templates/bps_biasBootstrap.yaml

+campaign: "${TICKET}"
+
+biasBootstrapCollection: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.$RERUN"
+biasBootstrapRun: "{biasBootstrapCollection}/run${BOOTSTRAP_RUN_NUMBER}"


Is this missing a $ for biasBootstrapCollection?

This is a bps variable, and must be quoted because yaml gets confused otherwise.

czwa · 2025-01-28T22:02:43Z

bps/templates/bps_darkBootstrap.yaml

+
+darkBootstrapCollection: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/darkBootstrapGen.$RERUN"
+biasBootstrapRun: "${USER_CALIB_PREFIX}$INSTRUMENT/calib/$TICKET/$TAG/biasBootstrapGen.$RERUN/run${BOOTSTRAP_RUN_NUMBER}"
+darkBootstrapRun: "{darkBootstrapCollection}/run${BOOTSTRAP_RUN_NUMBER}"


Same question as on biasBootstrap re $.

I see inCollection is the same, so maybe I'm just confused about variable scope. Those defined within the yaml only need braces, but those that are inherited from the environment need ${}?

czwa · 2025-01-28T22:10:14Z

bps/templates/bps_flat_g.yaml

+campaign: "${TICKET}"
+
+payload:
+  payloadName: "${INSTRUMENT}_${TICKET}"


Does the payloadName need to be unique? Will it complain if you launch flats in all filters simultaneously? I supposed this could apply to any of the steps, but flats feel more parallelizable than the others.

czwa reviewed Jan 22, 2025

View reviewed changes

erykoff force-pushed the tickets/DM-46562 branch 7 times, most recently from 87341d4 to e8253b5 Compare January 28, 2025 16:27

czwa approved these changes Jan 28, 2025

View reviewed changes

erykoff added 17 commits January 28, 2025 16:19

Add bps template for biasBootstrap.

0373043

Add bps template for darkBootstrap.

d9baa70

Add bps template for flatBootstrap.

634f95a

Add bps template for defects.

ffeb010

Add bps template for linearizer.

c41d3fe

Add bps template for ptc.

a14470c

Add bps template for bfk.

766fb88

Add bps template for cti creation.

48cab25

Add bps template for bias.

fdb13ac

Add bps template for dark.

537de23

Add bps template for u-band flat.

c17713f

Add bps template for g-band flat.

fe3b5a1

Add bps template for r-band flat.

b2cd1ea

Add bps template for i-band flat.

1a6c506

Add bps template for z-band flat.

434c4e3

Add bps template for y-band flat.

44e2ac1

Add README for bps templates.

1db6606

erykoff force-pushed the tickets/DM-46562 branch from e8253b5 to 1db6606 Compare January 29, 2025 00:19

erykoff merged commit 036654c into main Jan 29, 2025
2 checks passed

erykoff deleted the tickets/DM-46562 branch January 29, 2025 20:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DM-46562: Add bps templates for LSSTCam/LSSTComCam/LATISS calibration creation. #292

DM-46562: Add bps templates for LSSTCam/LSSTComCam/LATISS calibration creation. #292

erykoff commented Jan 22, 2025

czwa Jan 22, 2025

erykoff Jan 22, 2025

czwa Jan 22, 2025

czwa Jan 22, 2025

erykoff Jan 22, 2025

czwa Jan 23, 2025

timj Jan 23, 2025

erykoff Jan 23, 2025

czwa Jan 22, 2025

erykoff Jan 22, 2025

czwa Jan 22, 2025

czwa left a comment

czwa Jan 28, 2025

czwa Jan 28, 2025

czwa Jan 28, 2025

czwa Jan 28, 2025

czwa Jan 28, 2025

czwa Jan 28, 2025

czwa Jan 28, 2025

erykoff Jan 29, 2025

czwa Jan 28, 2025

czwa Jan 28, 2025

czwa Jan 28, 2025


		These are the environment variables that must be used. Examples are substituted in.

		* `export USER_CALIB_PREFIX=""` or `export USER_CALIB_PREFIX=u/erykoff/`: Set to a blank string for official calibrations, or the user collection prefix (with trailing slash).


		Overview
		--------

DM-46562: Add bps templates for LSSTCam/LSSTComCam/LATISS calibration creation. #292

DM-46562: Add bps templates for LSSTCam/LSSTComCam/LATISS calibration creation. #292

Conversation

erykoff commented Jan 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

czwa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment