Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

List of productions that the campaign management team has acknowledged as routine and accepted to oversee.  

In progress

~3 iterationsDRP on Simulated ComCam data at USDF
Pilot/Co-Pilot
channel/contact

#ops-rehearsal-3-sim/

Reps:

(Middleware)  James Chiang  (USDF infrastructure and Raws)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:Iteration #1:Starting 22 February 2024    Iteration#2:Starting 8 March 2024    Iteration #3:Starting 22 March 2024
Data:Simulated ComCam
Pipeline:Iteration #1: Latest weekly after w_2024_08      Iteration #2: w_2024_10      Iteration #3: w_2024_12
Collection Names:

First iteration:

  • repo = /repo/ops-rehearsal-3-prep
  • collection = u/homer/htc-test1

Second iteration:

  • repo = /repo/ops-rehearsal-3-prep
  • collection = u/homer/w_2024_10/DM-43228

Third iteration:

  • repo = /repo/ops-rehearsal-3-prep
  • collection = u/homer/w_2024_12/DM-43439
NotesDRPs in advance of the ops rehearsal are needed for refining the pipeline configs and generating a collection of difference imaging templates. We expect 3 iterations


One timeMultisite Scalability Test 0
Pilot/Co-Pilot
channel/contact

#dm-rucio-testing / James Chiang 

Reps:

Kian-Tat Lim  (Middleware) Fabio Hernandez (Fr Infrastructure) George Beckett (UK infrastructure) James Chiang  (US Infrastructure)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:Starting ~Oct 2023
Data:PDR2 WIDE VVDS Field. 
Pipeline:w_2023_48 (initially)
Collection Names:

butler: /repo/main (at USDF) hsc_pdr2_multisite (at FRDF, UKDF)

 HSC/runs/PDR2-VVDS-FR/w_2023_48/DM-40654

 HSC/runs/PDR2-VVDS-UK/w_2023_48/DM-40654

 HSC/runs/PDR2-VVDS/w_2023_48/DM-40654 (at USDF)

NotesMultisite testing will be iterative. First iteration will be to send data (disjoint regions of sky) manually via Rucio to the sites, run step1/2a/2b, and send calexps and preSourceTable_visits back. Testing without cm_tools this iteration. DM-40654



On goingMonthly DRPs on RC2/DC2 datasets at USDF
Pilot/Co-Pilot
channel/contact
#dm-hsc-reprocessing  / Yusra AlSayyad 
Reps:

No formal committee: Jeff, Angelo, and pipelines folks chime in on the hsc-reprocessing channel.

Reviewed at the DRP Metrics Meetings

ScheduleSchedule
Campaign Names: "the weekly XX RC2" 
Pipelines
Collection Names:

butler: /repo/main

HSC/runs/RC2/w_2023_XX/DM-XXXXX

Notes

Request from EC:  Could we add a couple of fields to each table:  1. what computing resources the will be used to process the campaign, 2. what tooling (e.g., panda, bps, cm_tools, prompt processing) will be used to process the campaign? 

...



On goingNightly LATISS Prompt Processing at USDF 
Pilot/Co-Pilot
channel/contact

Currently during development, pre-campaign stage:

#dm-#auxtel-prompt-processing   / Ian Sullivan Eric Bellm 

Reps:

Kian-Tat Lim (Middleware)   Yee LiDan Speck  (Infrastructure)   Ian Sullivan / Krzysztof Findeisen  (Pipeline) Meredith Rawls   ? (V&V)

ScheduleExpected Start Feb 2023.  Every other week.During AuxTel imaging runs beginning May 2023
Pipelinehttps://github.com/lsst-dm/prompt_processing/blob/main/pipelines/LATISS/ApPipe.yaml  for the "AUXTEL_PHOTO_IMAGING" survey. Fallback to single frame processing only if no templates available.
Data

Butler: /repo/embargo

DataLATISS science imaging during the biweekly reruns planned in 2023

Notes

Runs may become more frequent than fortnightly as observing staff become available on Cerro Pachón

.

Prompt Processing with AuxTel Imaging Survey Data 2024, 2023 (Includes collection names)


All Oct filter and sequencer changes)Internal Multi-site HSC PDR2 Data Release Production with v24.0.x
On goingFew iterations2023 LATISS Data Release Production at USDF
Pilot/Co-Pilot
 Huan Lin / <open>
channel/contact
?

#rubinobs-sitcom-surveys / Erik Dennihy 

Reps:

? (Pipelines) Colin Slater (V&V) <No rep> (Middleware/Infrastructure)

ScheduleExpected to start January 2023Every two weeks
Data

Cumulative LATISS imaging taken since

May 1st (since the

new observing pattern and no_RG sequencer change). 

Collections

posted in:  AuxTel Imaging Survey Data Descriptions

NotesErik's Proposal One Time


On goingMonthly AP Processing on HSC Cosmos, DECam HiTS/SahaBulge, and DC2 at USDF
Pilot/Co-Pilot
 Jen Adelman-Mccarthy /<open> 
channel/contact
?/ Leanne Guy

#dm-alert-prod  / Ian Sullivan 

Reps:

Eric Bellm (Pipelines) Meredith Rawls  (V&V) ? (Middleware) ? (Infrastructure)

Reviewed at the monthly AP QA meeting (links found here)

PipelinePer-collection. See:  /sdf/group/rubin/u/elhoward/repo-main-logs/DM-XXXXX
ScheduleMonthly. Dataset chosen depends on goals. 

Notes and Collection Names:

AP Precursor Processing Runs


On goingAfternoon Calibration Production at Summit and intermittent combine/certify at USDF 
Pilot/Co-Pilot

Huan Lin  / <open>

channel/contact
Reps:

   Tiago Ribeiro(ScriptQueue/Watcher) Cristián Silva  (Infrastructure)  Christopher Waters (Pipeline)  TaxiCab

Yusra AlSayyad (Pipeline) ? (Middleware),  Richard Dubois  (Infrastructure)

? (V&V)

  • #se-software-interact is the catch all triage channel. 
  • #auxtel-operations 
  • #dm-calibration-products


Schedule
To be completed July 2023
  • Starting with LATISS January 29
Data
PDR2 or PDR3 datasets. Last rerun was in 2020:  S20 HSC PDR2 ReprocessingNotesProduce pipeline quality metrics on real data, and test multi-site scalability with PanDA. 

Notes

Two aspects:

  • Afternoon: Check that the camera is working today (Summit)
  • Intermittent: Produce combined calibs for the prompt and DR productions (USDF)


One timeScalability Test for USDF PanDA Server
Pilot/Co-Pilot
channel/contact
Reps:


Schedule:Started January 30 2024
Data:
Campaign Names: 

butler: repo/main

u/sierrav/cm/pandatesting/DM-42607_test1

Collection Names:

butler: /repo/main

Notes

Goal: Find the max simultaneous cores that can be run simultaneously with the new USDF PanDA server. 


Currently testing w/ 6000 cores, some limited PanDA iDDS issues but otherwise seems fine?


Planned

On goingLATISS Rapid Analysis prompt processing at the Summit
Pilot/Co-Pilot <open>/ <open>
channel/contact
Reps:


ScheduleEarliest that pieces be in place to make it possible is April
Notes

...

I am having trouble envisioning a scenario where campaign management looks after this


On going
Monthly DRPs on RC2/DC2 datasets
LATISS 10am Analysis at USDF  (TBD)
Pilot/Co-Pilot
Orion Eiger/ Hsin-Fang Chiang 


channel/contact
#dm-hsc-reprocessing  / Yusra AlSayyad 


Reps:


Schedule

No formal committee: Jeff, Angelo, and pipelines folks chime in on the hsc-reprocessing channel.

Reviewed at the DRP Metrics Meetings

ScheduleSchedule

Pipeline:
Collection Names:


Notes





















Completed

One TimeInternal Multi-site HSC PDR2 Data Release Production with v24.0.x  (USDF to start
NotesOn goingDaily LATISS 10am Analysis at USDF (DRP-mode
)
Pilot/Co-Pilot
Huan Lin /  <open>
channel/contact
 / Colin Slater

#ops-cm-team / Yusra AlSayyad 

Reps:

Pipeline: Erik Dennihy Middleware?

ScheduleNotesOn goingMonthly AP Processing on HSC Cosmos/DECam HiTS at USDFPilot/Co-Pilotchannel/contact

#dm-alert-prod  / Ian Sullivan 

Reps:

Eric Bellm , Kenneth Herner , Erin Howard 

Reviewed at the monthly AP QA meeting (links found here)

ScheduleNotesAP Precursor Processing Runs

...

Yusra AlSayyad (Pipeline) Jim Bosch  (Middleware),  Richard Dubois  (Infrastructure) Colin Slater   (V&V)

ScheduleTo be completed July 2023 – (Steps 1-7 ready for evaluation as of Aug 29, 2023) Faro tract, rollup collection completed Sep 19, 2023.
Data

2023 Internal HSC PDR2 Reprocessing at the USDF 

PDR2:  Last rerun was in 2020:  S20 HSC PDR2 Reprocessing

Pipelinehttps://github.com/lsst/drp_pipe/blob/v24.0.x/pipelines/HSC/DRP.yaml
Campaign Names: "Pilot Campaign" "DRP Test Production Campaign"

Collection Names:


butler: /repo/main ,  HSC/raw/all (has 26K exposure, superset of PDR2 17K),

HSC/defaults (with calibs,skymap,masks, refcats) 

Input for step1: 

 HSC/raw/PDR2/{WIDE,DEEP,UDEEP}

HSC/runs/PDR2/WIDE/v24_1_0_rc2/DM-39132/ancillary

Output for step1:

HSC/runs/PDR2/{WIDE,DEEP,UDEEP}q/v24_1_0_rc2/DM-39132/step1

Inputs for step3:

 HSC/runs/PDR2/v24_1_0/DM-39132/step3_input,HSC/runs/PDR2/v24_1_0/DM-39132/ancillary

Output for step3-6, inputs for steps 4-7:

 HSC/runs/PDR2/v24_1_0/DM-39132/step{3,4,5,6}

Output for step7 (cumulative):

 HSC/runs/PDR2/v24_1_0/DM-39132/step7 (contains step3-6 outputs and ancillary skymap collection)

Faro (tract-based metrics): HSC/runs/PDR2/v24_1_0/DM-39132/farotract

Rollup collection: HSC/runs/PDR2/v24_1_0/DM-39132 (contains step 3-7 outputs, step3 inputs, farotract outputs, ancillary inputs (with skymap)).

NotesProduce pipeline quality metrics on real data, and test multi-site scalability with PanDA.  
RTN

http://rtn-063.lsst.io


One timeData Release Preview 0.2 with DC2 on IDF 
Pilot/Co-Pilot
channel/contact

#ops-df-proecessing / Hsin-Fang Chiang 

Reps:

Yusra AlSayyad (Pipeline) Tim Jenness  (Middleware) Colin Slater  (V&V)

(didn't have an infrastructure rep, because execution and Hsin-Fang were on infrastructure at the time

Pipelinehttps://github.com/lsst/obs_lsst/blob/v23.0.x/pipelines/imsim/DRP.yaml
Schedule
Notes