Skip to end of metadata
Go to start of metadata

List of productions that the campaign management team has acknowledged as routine and accepted to oversee.  

In progress

One timeMultisite Scalability Test 0
Pilot/Co-Pilot
channel/contact

#dm-rucio-testing / James Chiang 

Reps:

Kian-Tat Lim  (Middleware) Fabio Hernandez (Fr Infrastructure) George Beckett (UK infrastructure) James Chiang  (US Infrastructure)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:Starting ~Oct 2023
Data:PDR2 WIDE VVDS Field. 
Collection Names:

butler: /repo/main

HSC/runs/PDR2-VVDS/w_2023_XX/DM-XXXXX

NotesMultisite testing will be iterative. First iteration will be to send data (disjoint regions of sky) manually via Rucio to the sites, run step1/2a, and send calexps and preSourceTable_visits back. Testing without cm_tools this iteration.
On goingMonthly DRPs on RC2/DC2 datasets at USDF
Pilot/Co-Pilot
channel/contact
#dm-hsc-reprocessing  / Yusra AlSayyad 
Reps:

No formal committee: Jeff, Angelo, and pipelines folks chime in on the hsc-reprocessing channel.

Reviewed at the DRP Metrics Meetings

ScheduleSchedule
Campaign Names: "the weekly XX RC2" 
Collection Names:

butler: /repo/main

HSC/runs/RC2/w_2023_XX/DM-XXXXX

Notes


On goingLATISS 10am Analysis at USDF (DRP-mode)
Pilot/Co-Pilot

Huan Lin /  <open>

channel/contact

 #rubinobs-sitcom-surveys / Erik Dennihy 

Reps:

Pipeline: Erik Dennihy V&V: Colin Slater 

ScheduleEvery 10am after auxtel data was taken the night before
Collection Name:
NotesAuxTel Imaging Survey Data Descriptions
On goingNightly LATISS Prompt Processing at USDF 
Pilot/Co-Pilot
channel/contact

#auxtel-prompt-processing   / Eric Bellm 

Reps:

Kian-Tat Lim (Middleware)  Dan Speck  (Infrastructure) Ian Sullivan / Krzysztof Findeisen  (Pipeline) Meredith Rawls  (V&V)

ScheduleDuring AuxTel imaging runs beginning May 2023
Data

Butler: /repo/embargo

Notes

Runs may become more frequent than fortnightly as observing staff become available on Cerro Pachón.

Prompt Processing with AuxTel Imaging Survey Data 2023


Few iterations2023 LATISS Data Release Production at USDF
Pilot/Co-Pilot
 Huan Lin / <open>
channel/contact
Reps:


ScheduleWhen we have sufficient data taken consistently.
DataAll LATISS imaging taken since May 1st (since the new observing pattern and no_RG sequencer change). Older data if necessary/processed separately?
NotesErik's Proposal 
On goingMonthly AP Processing on HSC Cosmos/DECam HiTS at USDF
Pilot/Co-Pilot
channel/contact

#dm-alert-prod  / Ian Sullivan 

Reps:

Eric Bellm , Kenneth Herner , Erin Howard 

Reviewed at the monthly AP QA meeting (links found here)

Schedule
Notes

AP Precursor Processing Runs


Planned

On goingAfternoon Calibration Production at Summit and intermittent combine/certify at USDF 
Pilot/Co-Pilot

Homer Neal  / <open>

channel/contact
Reps:

   Tiago Ribeiro(ScriptQueue/Watcher) Cristián Silva  (Infrastructure)  Christopher Waters (Pipeline)  TaxiCab? (V&V)

  • #se-software-interact is the catch all triage channel. 
  • #auxtel-operations 
  • #dm-calibration-products


Schedule
  • Starting with LATISS May 9
  • Whenever ComCam starts routinely starts taking biases and darks at the summit. 
Data
Notes

Two aspects:

  • Afternoon: Check that the camera is working today (Summit)
  • Intermittent: Produce combined calibs for the prompt and DR productions (USDF)



On goingLATISS Rapid Analysis prompt processing at the Summit
Pilot/Co-Pilot <open>/ <open>
channel/contact
Reps:


ScheduleEarliest that pieces be in place to make it possible is April
Notes


One timeScalability Test for USDF PanDA Server
Pilot/Co-Pilot
channel/contact


Reps:


Schedule:Fall 2023
Data:
Campaign Names: 
Collection Names:

butler: /repo/main

NotesGoal: Find the max simultaneous cores that can be run simultaneously with the new USDF PanDA server. 

Completed

One TimeInternal Multi-site HSC PDR2 Data Release Production with v24.0.x  (USDF to start)
Pilot/Co-Pilot
channel/contact

#ops-cm-team / Yusra AlSayyad 

Reps:

Yusra AlSayyad (Pipeline) Jim Bosch  (Middleware),  Richard Dubois  (Infrastructure) Colin Slater   (V&V)

ScheduleTo be completed July 2023 – (Steps 1-7 ready for evaluation as of Aug 29, 2023) Faro tract, rollup collection completed Sep 19, 2023.
Data

2023 Internal HSC PDR2 Reprocessing at the USDF 

PDR2:  Last rerun was in 2020:  S20 HSC PDR2 Reprocessing

Campaign Names: "Pilot Campaign" "DRP Test Production Campaign"

Collection Names:


butler: /repo/main ,  HSC/raw/all (has 26K exposure, superset of PDR2 17K),

HSC/defaults (with calibs,skymap,masks, refcats) 

Input for step1: 

 HSC/raw/PDR2/{WIDE,DEEP,UDEEP}

HSC/runs/PDR2/WIDE/v24_1_0_rc2/DM-39132/ancillary

Output for step1:

HSC/runs/PDR2/{WIDE,DEEP,UDEEP}q/v24_1_0_rc2/DM-39132/step1

Inputs for step3:

 HSC/runs/PDR2/v24_1_0/DM-39132/step3_input,HSC/runs/PDR2/v24_1_0/DM-39132/ancillary

Output for step3-6, inputs for steps 4-7:

 HSC/runs/PDR2/v24_1_0/DM-39132/step{3,4,5,6}

Output for step7 (cumulative):

 HSC/runs/PDR2/v24_1_0/DM-39132/step7 (contains step3-6 outputs and ancillary skymap collection)

Faro (tract-based metrics): HSC/runs/PDR2/v24_1_0/DM-39132/farotract

Rollup collection: HSC/runs/PDR2/v24_1_0/DM-39132 (contains step 3-7 outputs, step3 inputs, farotract outputs, ancillary inputs (with skymap)).

NotesProduce pipeline quality metrics on real data, and test multi-site scalability with PanDA.  
RTN

http://rtn-063.lsst.io

One timeData Release Preview 0.2 with DC2 on IDF 
Pilot/Co-Pilot
channel/contact

#ops-df-proecessing / Hsin-Fang Chiang 

Reps:

Yusra AlSayyad (Pipeline) Tim Jenness  (Middleware) Colin Slater  (V&V)

(didn't have an infrastructure rep, because execution and Hsin-Fang were on infrastructure at the time

Schedule
Notes
  • No labels

1 Comment

  1. Moving comment that Eric Charlesdropped in the header to a comment:

    Request from EC:  Could we add some fields to each table: 

    1. what computing resources the will be used to process the campaign,

    2. what tooling (e.g., panda, bps, cm_tools, prompt processing) will be used to process the campaign? 

    3. naming conventions: 

    • for the production (i.e., the set of related campaigns)
    • for the campaign
    • A template for the collection names while the processing is underway, e.g. 'HSC/rc2/tests/{production}/{campaign}` 
    • A template for the final location of the output collection, if it is different