List of productions that the campaign management team has acknowledged as routine and accepted to oversee.
In progress
~3 iterations | DRP on Simulated ComCam data at USDF |
---|---|
Pilot/Co-Pilot | |
channel/contact | #ops-rehearsal-3-sim/ |
Reps: | (Middleware) James Chiang (USDF infrastructure and Raws) Yusra AlSayyad (Pipeline) Colin Slater (V&V) |
Schedule: | Iteration #1:Starting 22 February 2024 Iteration#2:Starting 8 March 2024 Iteration #3:Starting 22 March 2024 |
Data: | Simulated ComCam |
Pipeline: | Iteration #1: Latest weekly after w_2024_08 Iteration #2: w_2024_10 Iteration #3: w_2024_12 |
Collection Names: | First iteration:
Second iteration:
Third iteration:
|
Notes | DRPs in advance of the ops rehearsal are needed for refining the pipeline configs and generating a collection of difference imaging templates. We expect 3 iterations |
One time | Multisite Scalability Test 0 |
---|---|
Pilot/Co-Pilot | |
channel/contact | #dm-rucio-testing / James Chiang |
Reps: | Kian-Tat Lim (Middleware) Fabio Hernandez (Fr Infrastructure) George Beckett (UK infrastructure) James Chiang (US Infrastructure) Yusra AlSayyad (Pipeline) Colin Slater (V&V) |
Schedule: | Starting ~Oct 2023 |
Data: | PDR2 WIDE VVDS Field. |
Pipeline: | w_2023_48 (initially) |
Collection Names: | butler: /repo/main (at USDF) hsc_pdr2_multisite (at FRDF, UKDF) HSC/runs/PDR2-VVDS-FR/w_2023_48/DM-40654 HSC/runs/PDR2-VVDS-UK/w_2023_48/DM-40654 HSC/runs/PDR2-VVDS/w_2023_48/DM-40654 (at USDF) |
Notes | Multisite testing will be iterative. First iteration will be to send data (disjoint regions of sky) manually via Rucio to the sites, run step1/2a/2b, and send calexps and preSourceTable_visits back. Testing without cm_tools this iteration. DM-40654 |
On going | Monthly DRPs on RC2/DC2 datasets at USDF |
---|---|
Pilot/Co-Pilot | |
channel/contact | |
Reps: | No formal committee: Jeff, Angelo, and pipelines folks chime in on the hsc-reprocessing channel. Reviewed at the DRP Metrics Meetings |
Schedule | Schedule |
Campaign Names: | "the weekly XX RC2" |
Pipelines | |
Collection Names: | butler: /repo/main HSC/runs/RC2/w_2023_XX/DM-XXXXX |
Notes |
Request from EC: Could we add a couple of fields to each table: 1. what computing resources the will be used to process the campaign, 2. what tooling (e.g., panda, bps, cm_tools, prompt processing) will be used to process the campaign?
...
On going | Nightly LATISS Prompt Processing at USDF | ||
---|---|---|---|
Pilot/Co-Pilot | Hsin-Fang Chiang / <open> | ||
channel/contact | Currently during development, pre-campaign stage: #dm-#auxtel-prompt-processing / Ian Sullivan Eric Bellm | ||
Reps: | Kian-Tat Lim (Middleware) Yee LiDan Speck (Infrastructure) Ian Sullivan / Krzysztof Findeisen (Pipeline) Meredith Rawls ? (V&V) | ||
Schedule | Expected Start Feb 2023. Every other week. | During AuxTel imaging runs beginning May 2023 | |
Pipeline | https://github.com/lsst-dm/prompt_processing/blob/main/pipelines/LATISS/ApPipe.yaml for the "AUXTEL_PHOTO_IMAGING" survey. Fallback to single frame processing only if no templates available. | ||
Data | Butler: /repo/embargo | Data | LATISS science imaging during the biweekly reruns planned in 2023 |
Notes | Runs may become more frequent than fortnightly as observing staff become available on Cerro Pachón . Prompt Processing with AuxTel Imaging Survey Data 2024, 2023 (Includes collection names) |
On goingFew iterations | 2023 LATISS Data Release Production at USDF | ||
---|---|---|---|
Pilot/Co-Pilot | |||
channel/contact | ? #rubinobs-sitcom-surveys / Erik Dennihy | ||
Reps: | ? (Pipelines) Colin Slater (V&V) <No rep> (Middleware/Infrastructure) | ||
ScheduleExpected to start January 2023 | Every two weeks | ||
Data | AllCumulative LATISS imaging taken since | Oct May 1st (since the | filter and sequencer changes)new observing pattern and no_RG sequencer change). |
Collections | posted in: AuxTel Imaging Survey Data Descriptions | ||
Notes | Erik's Proposal | One Time | Internal Multi-site HSC PDR2 Data Release Production with v24.0.x
On going | Monthly AP Processing on HSC Cosmos, DECam HiTS/SahaBulge, and DC2 at USDF |
---|---|
Pilot/Co-Pilot |
channel/contact |
#dm-alert-prod / Ian Sullivan | |
Reps: | Eric Bellm (Pipelines) Meredith Rawls (V&V) ? (Middleware) ? (Infrastructure) Reviewed at the monthly AP QA meeting (links found here) |
Pipeline | Per-collection. See: /sdf/group/rubin/u/elhoward/repo-main-logs/DM-XXXXX |
Schedule | Monthly. Dataset chosen depends on goals. |
Notes and Collection Names: |
On going | Afternoon Calibration Production at Summit and intermittent combine/certify at USDF |
---|---|
Pilot/Co-Pilot | Huan Lin / <open> |
channel/contact |
|
Reps: | Tiago Ribeiro(ScriptQueue/Watcher) Cristián Silva (Infrastructure) Christopher Waters (Pipeline) TaxiCab |
? (V&V)
|
Schedule |
|
Data |
Notes | Two aspects:
|
One time | Scalability Test for USDF PanDA Server |
---|---|
Pilot/Co-Pilot | |
channel/contact | |
Reps: | |
Schedule: | Started January 30 2024 |
Data: | |
Campaign Names: | butler: repo/main u/sierrav/cm/pandatesting/DM-42607_test1 |
Collection Names: | butler: /repo/main |
Notes | Goal: Find the max simultaneous cores that can be run simultaneously with the new USDF PanDA server. Currently testing w/ 6000 cores, some limited PanDA iDDS issues but otherwise seems fine? |
Planned
On going | LATISS Rapid Analysis prompt processing at the Summit |
---|---|
Pilot/Co-Pilot | <open>/ <open> |
channel/contact | |
Reps: | |
Schedule | Earliest that pieces be in place to make it possible is April |
Notes |
...
I am having trouble envisioning a scenario where campaign management looks after this |
On going |
---|
LATISS 10am Analysis at USDF (TBD) |
---|
Pilot/Co-Pilot |
channel/contact |
Reps: | |
Schedule |
No formal committee: Jeff, Angelo, and pipelines folks chime in on the hsc-reprocessing channel.
Reviewed at the DRP Metrics Meetings
Pipeline: | |
Collection Names: | |
Notes |
Completed
One Time | Internal Multi-site HSC PDR2 Data Release Production with v24.0.x (USDF to start |
---|
) |
---|
Pilot/Co-Pilot |
channel/contact |
#ops-cm-team / Yusra AlSayyad |
Reps: |
Pipeline: Erik Dennihy Middleware?
#dm-alert-prod / Ian Sullivan
Eric Bellm , Kenneth Herner , Erin Howard
Reviewed at the monthly AP QA meeting (links found here)
...
Yusra AlSayyad (Pipeline) Jim Bosch (Middleware), Richard Dubois (Infrastructure) Colin Slater (V&V) | |
Schedule | To be completed July 2023 – (Steps 1-7 ready for evaluation as of Aug 29, 2023) Faro tract, rollup collection completed Sep 19, 2023. |
Data | 2023 Internal HSC PDR2 Reprocessing at the USDF PDR2: Last rerun was in 2020: S20 HSC PDR2 Reprocessing |
Pipeline | https://github.com/lsst/drp_pipe/blob/v24.0.x/pipelines/HSC/DRP.yaml |
Campaign Names: | "Pilot Campaign" "DRP Test Production Campaign" |
Collection Names: | butler: /repo/main , HSC/raw/all (has 26K exposure, superset of PDR2 17K), HSC/defaults (with calibs,skymap,masks, refcats) Input for step1: HSC/raw/PDR2/{WIDE,DEEP,UDEEP} HSC/runs/PDR2/WIDE/v24_1_0_rc2/DM-39132/ancillary Output for step1: HSC/runs/PDR2/{WIDE,DEEP,UDEEP}q/v24_1_0_rc2/DM-39132/step1 Inputs for step3: HSC/runs/PDR2/v24_1_0/DM-39132/step3_input,HSC/runs/PDR2/v24_1_0/DM-39132/ancillary Output for step3-6, inputs for steps 4-7: HSC/runs/PDR2/v24_1_0/DM-39132/step{3,4,5,6} Output for step7 (cumulative): HSC/runs/PDR2/v24_1_0/DM-39132/step7 (contains step3-6 outputs and ancillary skymap collection) Faro (tract-based metrics): HSC/runs/PDR2/v24_1_0/DM-39132/farotract Rollup collection: HSC/runs/PDR2/v24_1_0/DM-39132 (contains step 3-7 outputs, step3 inputs, farotract outputs, ancillary inputs (with skymap)). |
Notes | Produce pipeline quality metrics on real data, and test multi-site scalability with PanDA. |
RTN |
One time | Data Release Preview 0.2 with DC2 on IDF |
---|---|
Pilot/Co-Pilot | Huan Lin / |
channel/contact | #ops-df-proecessing / Hsin-Fang Chiang |
Reps: | Yusra AlSayyad (Pipeline) Tim Jenness (Middleware) Colin Slater (V&V) (didn't have an infrastructure rep, because execution and Hsin-Fang were on infrastructure at the time |
Pipeline | https://github.com/lsst/obs_lsst/blob/v23.0.x/pipelines/imsim/DRP.yaml |
Schedule | |
Notes |