List of productions that the campaign management team has acknowledged as routine and accepted to oversee.
In progress
One time | Multisite Scalability Test 0 |
---|---|
Pilot/Co-Pilot | |
channel/contact | #dm-rucio-testing / James Chiang |
Reps: | Kian-Tat Lim (Middleware) Fabio Hernandez (Fr Infrastructure) George Beckett (UK infrastructure) James Chiang (US Infrastructure) Yusra AlSayyad (Pipeline) Colin Slater (V&V) |
Schedule: | Starting ~Oct 2023 |
Data: | PDR2 WIDE VVDS Field. |
Collection Names: | butler: /repo/main HSC/runs/PDR2-VVDS/w_2023_XX/DM-XXXXX |
Notes | Multisite testing will be iterative. First iteration will be to send data (disjoint regions of sky) manually via Rucio to the sites, run step1/2a, and send calexps and preSourceTable_visits back. Testing without cm_tools this iteration. |
On going | Monthly DRPs on RC2/DC2 datasets at USDF |
---|---|
Pilot/Co-Pilot | |
channel/contact | #dm-hsc-reprocessing / Yusra AlSayyad |
Reps: | No formal committee: Jeff, Angelo, and pipelines folks chime in on the hsc-reprocessing channel. Reviewed at the DRP Metrics Meetings |
Schedule | Schedule |
Campaign Names: | "the weekly XX RC2" |
Collection Names: | butler: /repo/main HSC/runs/RC2/w_2023_XX/DM-XXXXX |
Notes |
On going | LATISS 10am Analysis at USDF (DRP-mode) |
---|---|
Pilot/Co-Pilot | Huan Lin / <open> |
channel/contact | #rubinobs-sitcom-surveys / Erik Dennihy |
Reps: | Pipeline: Erik Dennihy V&V: Colin Slater |
Schedule | Every 10am after auxtel data was taken the night before |
Collection Name: | |
Notes | AuxTel Imaging Survey Data Descriptions |
On going | Nightly LATISS Prompt Processing at USDF |
---|---|
Pilot/Co-Pilot | Hsin-Fang Chiang / <open> |
channel/contact | #auxtel-prompt-processing / Eric Bellm |
Reps: | Kian-Tat Lim (Middleware) Dan Speck (Infrastructure) Ian Sullivan / Krzysztof Findeisen (Pipeline) Meredith Rawls (V&V) |
Schedule | During AuxTel imaging runs beginning May 2023 |
Data | Butler: /repo/embargo |
Notes | Runs may become more frequent than fortnightly as observing staff become available on Cerro Pachón. |
Few iterations | 2023 LATISS Data Release Production at USDF |
---|---|
Pilot/Co-Pilot | Huan Lin / <open> |
channel/contact | ?/ Erik Dennihy |
Reps: | |
Schedule | When we have sufficient data taken consistently. |
Data | All LATISS imaging taken since May 1st (since the new observing pattern and no_RG sequencer change). Older data if necessary/processed separately? |
Notes | Erik's Proposal |
On going | Monthly AP Processing on HSC Cosmos/DECam HiTS at USDF |
---|---|
Pilot/Co-Pilot | |
channel/contact | #dm-alert-prod / Ian Sullivan |
Reps: | Eric Bellm , Kenneth Herner , Erin Howard Reviewed at the monthly AP QA meeting (links found here) |
Schedule | |
Notes |
Planned
On going | Afternoon Calibration Production at Summit and intermittent combine/certify at USDF |
---|---|
Pilot/Co-Pilot | Homer Neal / <open> |
channel/contact |
|
Reps: | Tiago Ribeiro(ScriptQueue/Watcher) Cristián Silva (Infrastructure) Christopher Waters (Pipeline) TaxiCab? (V&V)
|
Schedule |
|
Data | |
Notes | Two aspects:
|
On going | LATISS Rapid Analysis prompt processing at the Summit |
---|---|
Pilot/Co-Pilot | <open>/ <open> |
channel/contact | |
Reps: | |
Schedule | Earliest that pieces be in place to make it possible is April |
Notes |
One time | Scalability Test for USDF PanDA Server |
---|---|
Pilot/Co-Pilot | |
channel/contact | |
Reps: | |
Schedule: | Fall 2023 |
Data: | |
Campaign Names: | |
Collection Names: | butler: /repo/main |
Notes | Goal: Find the max simultaneous cores that can be run simultaneously with the new USDF PanDA server. |
Completed
One Time | Internal Multi-site HSC PDR2 Data Release Production with v24.0.x (USDF to start) |
---|---|
Pilot/Co-Pilot | |
channel/contact | #ops-cm-team / Yusra AlSayyad |
Reps: | Yusra AlSayyad (Pipeline) Jim Bosch (Middleware), Richard Dubois (Infrastructure) Colin Slater (V&V) |
Schedule | To be completed July 2023 – (Steps 1-7 ready for evaluation as of Aug 29, 2023) Faro tract, rollup collection completed Sep 19, 2023. |
Data | 2023 Internal HSC PDR2 Reprocessing at the USDF PDR2: Last rerun was in 2020: S20 HSC PDR2 Reprocessing |
Campaign Names: | "Pilot Campaign" "DRP Test Production Campaign" |
Collection Names: | butler: /repo/main , HSC/raw/all (has 26K exposure, superset of PDR2 17K), HSC/defaults (with calibs,skymap,masks, refcats) Input for step1: HSC/raw/PDR2/{WIDE,DEEP,UDEEP} HSC/runs/PDR2/WIDE/v24_1_0_rc2/DM-39132/ancillary Output for step1: HSC/runs/PDR2/{WIDE,DEEP,UDEEP}q/v24_1_0_rc2/DM-39132/step1 Inputs for step3: HSC/runs/PDR2/v24_1_0/DM-39132/step3_input,HSC/runs/PDR2/v24_1_0/DM-39132/ancillary Output for step3-6, inputs for steps 4-7: HSC/runs/PDR2/v24_1_0/DM-39132/step{3,4,5,6} Output for step7 (cumulative): HSC/runs/PDR2/v24_1_0/DM-39132/step7 (contains step3-6 outputs and ancillary skymap collection) Faro (tract-based metrics): HSC/runs/PDR2/v24_1_0/DM-39132/farotract Rollup collection: HSC/runs/PDR2/v24_1_0/DM-39132 (contains step 3-7 outputs, step3 inputs, farotract outputs, ancillary inputs (with skymap)). |
Notes | Produce pipeline quality metrics on real data, and test multi-site scalability with PanDA. |
RTN |
One time | Data Release Preview 0.2 with DC2 on IDF |
---|---|
Pilot/Co-Pilot | Huan Lin / |
channel/contact | #ops-df-proecessing / Hsin-Fang Chiang |
Reps: | Yusra AlSayyad (Pipeline) Tim Jenness (Middleware) Colin Slater (V&V) (didn't have an infrastructure rep, because execution and Hsin-Fang were on infrastructure at the time |
Schedule | |
Notes |
1 Comment
Yusra AlSayyad
Moving comment that Eric Charlesdropped in the header to a comment:
Request from EC: Could we add some fields to each table:
1. what computing resources the will be used to process the campaign,
2. what tooling (e.g., panda, bps, cm_tools, prompt processing) will be used to process the campaign?
3. naming conventions: