List of productions that the campaign management team has acknowledged as routine and accepted to oversee.
In progress
| One time | Multisite Scalability Test 0 |
|---|---|
| Pilot/Co-Pilot | |
| channel/contact | #dm-rucio-testing / James Chiang |
| Reps: | Kian-Tat Lim (Middleware) Fabio Hernandez (Fr Infrastructure) George Beckett (UK infrastructure) James Chiang (US Infrastructure) Yusra AlSayyad (Pipeline) Colin Slater (V&V) |
| Schedule: | Starting ~Oct 2023 |
| Data: | PDR2 WIDE VVDS Field. |
| Collection Names: | butler: /repo/main HSC/runs/PDR2-VVDS/w_2023_XX/DM-XXXXX |
| Notes | Multisite testing will be iterative. First iteration will be to send data (disjoint regions of sky) manually via Rucio to the sites, run step1/2a, and send calexps and preSourceTable_visits back. Testing without cm_tools this iteration. |
| On going | Monthly DRPs on RC2/DC2 datasets at USDF |
|---|---|
| Pilot/Co-Pilot | |
| channel/contact | #dm-hsc-reprocessing / Yusra AlSayyad |
| Reps: | No formal committee: Jeff, Angelo, and pipelines folks chime in on the hsc-reprocessing channel. Reviewed at the DRP Metrics Meetings |
| Schedule | Schedule |
| Campaign Names: | "the weekly XX RC2" |
| Collection Names: | butler: /repo/main HSC/runs/RC2/w_2023_XX/DM-XXXXX |
| Notes |
| On going | LATISS 10am Analysis at USDF (DRP-mode) |
|---|---|
| Pilot/Co-Pilot | Huan Lin / <open> |
| channel/contact | #rubinobs-sitcom-surveys / Erik Dennihy |
| Reps: | Pipeline: Erik Dennihy V&V: Colin Slater |
| Schedule | Every 10am after auxtel data was taken the night before |
| Collection Name: | |
| Notes | AuxTel Imaging Survey Data Descriptions |
| On going | Nightly LATISS Prompt Processing at USDF |
|---|---|
| Pilot/Co-Pilot | Hsin-Fang Chiang / <open> |
| channel/contact | #auxtel-prompt-processing / Eric Bellm |
| Reps: | Kian-Tat Lim (Middleware) Dan Speck (Infrastructure) Ian Sullivan / Krzysztof Findeisen (Pipeline) Meredith Rawls (V&V) |
| Schedule | During AuxTel imaging runs beginning May 2023 |
| Data | Butler: /repo/embargo |
| Notes | Runs may become more frequent than fortnightly as observing staff become available on Cerro Pachón. |
| Few iterations | 2023 LATISS Data Release Production at USDF |
|---|---|
| Pilot/Co-Pilot | Huan Lin / <open> |
| channel/contact | ?/ Erik Dennihy |
| Reps: | |
| Schedule | When we have sufficient data taken consistently. |
| Data | All LATISS imaging taken since May 1st (since the new observing pattern and no_RG sequencer change). Older data if necessary/processed separately? |
| Notes | Erik's Proposal |
| On going | Monthly AP Processing on HSC Cosmos/DECam HiTS at USDF |
|---|---|
| Pilot/Co-Pilot | |
| channel/contact | #dm-alert-prod / Ian Sullivan |
| Reps: | Eric Bellm , Kenneth Herner , Erin Howard Reviewed at the monthly AP QA meeting (links found here) |
| Schedule | |
| Notes |
Planned
| On going | Afternoon Calibration Production at Summit and intermittent combine/certify at USDF |
|---|---|
| Pilot/Co-Pilot | Homer Neal / <open> |
| channel/contact |
|
| Reps: | Tiago Ribeiro(ScriptQueue/Watcher) Cristián Silva (Infrastructure) Christopher Waters (Pipeline) TaxiCab? (V&V)
|
| Schedule |
|
| Data | |
| Notes | Two aspects:
|
| On going | LATISS Rapid Analysis prompt processing at the Summit |
|---|---|
| Pilot/Co-Pilot | <open>/ <open> |
| channel/contact | |
| Reps: | |
| Schedule | Earliest that pieces be in place to make it possible is April |
| Notes |
| One time | Scalability Test for USDF PanDA Server |
|---|---|
| Pilot/Co-Pilot | |
| channel/contact | |
| Reps: | |
| Schedule: | Fall 2023 |
| Data: | |
| Campaign Names: | |
| Collection Names: | butler: /repo/main |
| Notes | Goal: Find the max simultaneous cores that can be run simultaneously with the new USDF PanDA server. |
Completed
| One Time | Internal Multi-site HSC PDR2 Data Release Production with v24.0.x (USDF to start) |
|---|---|
| Pilot/Co-Pilot | |
| channel/contact | #ops-cm-team / Yusra AlSayyad |
| Reps: | Yusra AlSayyad (Pipeline) Jim Bosch (Middleware), Richard Dubois (Infrastructure) Colin Slater (V&V) |
| Schedule | To be completed July 2023 – (Steps 1-7 ready for evaluation as of Aug 29, 2023) Faro tract, rollup collection completed Sep 19, 2023. |
| Data | 2023 Internal HSC PDR2 Reprocessing at the USDF PDR2: Last rerun was in 2020: S20 HSC PDR2 Reprocessing |
| Campaign Names: | "Pilot Campaign" "DRP Test Production Campaign" |
Collection Names: | butler: /repo/main , HSC/raw/all (has 26K exposure, superset of PDR2 17K), HSC/defaults (with calibs,skymap,masks, refcats) Input for step1: HSC/raw/PDR2/{WIDE,DEEP,UDEEP} HSC/runs/PDR2/WIDE/v24_1_0_rc2/DM-39132/ancillary Output for step1: HSC/runs/PDR2/{WIDE,DEEP,UDEEP}q/v24_1_0_rc2/DM-39132/step1 Inputs for step3: HSC/runs/PDR2/v24_1_0/DM-39132/step3_input,HSC/runs/PDR2/v24_1_0/DM-39132/ancillary Output for step3-6, inputs for steps 4-7: HSC/runs/PDR2/v24_1_0/DM-39132/step{3,4,5,6} Output for step7 (cumulative): HSC/runs/PDR2/v24_1_0/DM-39132/step7 (contains step3-6 outputs and ancillary skymap collection) Faro (tract-based metrics): HSC/runs/PDR2/v24_1_0/DM-39132/farotract Rollup collection: HSC/runs/PDR2/v24_1_0/DM-39132 (contains step 3-7 outputs, step3 inputs, farotract outputs, ancillary inputs (with skymap)). |
| Notes | Produce pipeline quality metrics on real data, and test multi-site scalability with PanDA. |
| RTN |
| One time | Data Release Preview 0.2 with DC2 on IDF |
|---|---|
| Pilot/Co-Pilot | Huan Lin / |
| channel/contact | #ops-df-proecessing / Hsin-Fang Chiang |
| Reps: | Yusra AlSayyad (Pipeline) Tim Jenness (Middleware) Colin Slater (V&V) (didn't have an infrastructure rep, because execution and Hsin-Fang were on infrastructure at the time |
| Schedule | |
| Notes |
1 Comment
Yusra AlSayyad
Moving comment that Eric Charlesdropped in the header to a comment:
Request from EC: Could we add some fields to each table:
1. what computing resources the will be used to process the campaign,
2. what tooling (e.g., panda, bps, cm_tools, prompt processing) will be used to process the campaign?
3. naming conventions: