List of productions that the campaign management team has acknowledged as routine and accepted to oversee.  

In progress


One timeMultisite Scalability Test 0
Pilot/Co-Pilot
channel/contact

#dm-rucio-testing / James Chiang 

Reps:

Kian-Tat Lim  (Middleware) Fabio Hernandez (Fr Infrastructure) George Beckett (UK infrastructure) James Chiang  (US Infrastructure)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:Starting ~Oct 2023
Data:PDR2 WIDE VVDS Field. 
Pipeline:w_2023_48 (initially)
Collection Names:

butler: /repo/main (at USDF) hsc_pdr2_multisite (at FRDF, UKDF)

 HSC/runs/PDR2-VVDS-FR/w_2023_48/DM-40654

 HSC/runs/PDR2-VVDS-UK/w_2023_48/DM-40654

 HSC/runs/PDR2-VVDS/w_2023_48/DM-40654 (at USDF)

NotesMultisite testing will be iterative. First iteration will be to send data (disjoint regions of sky) manually via Rucio to the sites, run step1/2a/2b, and send calexps and preSourceTable_visits back. Testing without cm_tools this iteration. DM-40654
On goingMonthly DRPs on RC2/DC2 datasets at USDF
Pilot/Co-Pilot
channel/contact
#dm-hsc-reprocessing  / Yusra AlSayyad 
Reps:

No formal committee: Jeff, Angelo, and pipelines folks chime in on the hsc-reprocessing channel.

Reviewed at the DRP Metrics Meetings

ScheduleSchedule
Campaign Names: "the weekly XX RC2" 
Pipelines
Collection Names:

butler: /repo/main

HSC/runs/RC2/w_2023_XX/DM-XXXXX

Notes
On goingNightly LATISS Prompt Processing at USDF 
Pilot/Co-Pilot
channel/contact

#auxtel-prompt-processing   / Eric Bellm 

Reps:

Kian-Tat Lim (Middleware)  Dan Speck  (Infrastructure) Ian Sullivan / Krzysztof Findeisen  (Pipeline) Meredith Rawls  (V&V)

ScheduleDuring AuxTel imaging runs beginning May 2023
Pipelinehttps://github.com/lsst-dm/prompt_processing/blob/main/pipelines/LATISS/ApPipe.yaml  for the "AUXTEL_PHOTO_IMAGING" survey. Fallback to single frame processing only if no templates available.
Data

Butler: /repo/embargo

Collection Names

LATISS/prompt/output-YYYY-MM-DD (starting from 2024-03-25)

Notes

Runs may become more frequent than fortnightly as observing staff become available on Cerro Pachón.

Prompt Processing with AuxTel Imaging Survey Data 2024, 2023 (Includes run names)


On going2023 LATISS Data Release Production at USDF
Pilot/Co-Pilot
 Huan Lin / <open>
channel/contact

#rubinobs-sitcom-surveys / Erik Dennihy 

Reps:

? (Pipelines) Colin Slater (V&V) <No rep> (Middleware/Infrastructure)

ScheduleEvery two weeks
Data

Cumulative LATISS imaging taken since May 1st (since the new observing pattern and no_RG sequencer change). 

Collections

posted in:  AuxTel Imaging Survey Data Descriptions

NotesErik's Proposal 
On goingMonthly AP Processing on HSC Cosmos, DECam HiTS/SahaBulge, and DC2 at USDF
Pilot/Co-Pilot
channel/contact

#dm-alert-prod  / Ian Sullivan 

Reps:

Eric Bellm (Pipelines) Meredith Rawls  (V&V) ? (Middleware) ? (Infrastructure)

Reviewed at the monthly AP QA meeting (links found here)

PipelinePer-collection. See:  /sdf/group/rubin/u/elhoward/repo-main-logs/DM-XXXXX
ScheduleMonthly. Dataset chosen depends on goals. 

Notes and Collection Names:

AP Precursor Processing Runs

On goingAfternoon Calibration Production at Summit and intermittent combine/certify at USDF 
Pilot/Co-Pilot

Huan Lin  / <open>

channel/contact
Reps:

   Tiago Ribeiro(ScriptQueue/Watcher) Cristián Silva  (Infrastructure)  Christopher Waters (Pipeline)  TaxiCab? (V&V)

  • #se-software-interact is the catch all triage channel. 
  • #auxtel-operations 
  • #dm-calibration-products


Schedule
  • Starting with LATISS January 29
Data
Notes

Two aspects:

  • Afternoon: Check that the camera is working today (Summit)
  • Intermittent: Produce combined calibs for the prompt and DR productions (USDF)
One timeScalability Test for USDF PanDA Server
Pilot/Co-Pilot
channel/contact
Reps:


Schedule:Started January 30 2024
Data:
Campaign Names: 

butler: repo/main

u/sierrav/cm/pandatesting/DM-42607_test1

Collection Names:

butler: /repo/main

Notes

Goal: Find the max simultaneous cores that can be run simultaneously with the new USDF PanDA server. 


Currently testing w/ 6000 cores, some limited PanDA iDDS issues but otherwise seems fine?


Planned

Ops Rehearsal 4Intermittent Cumulative DRP of the 3 nights of ops rehearsal 4 data
Pilot/Co-Pilot

channel/contact

#ops-rehearsals

Reps:

(Middleware)  James Chiang  (USDF infrastructure and Raws)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:Morning of June 28 2024
Data:3-nights of Simulated ComCam, replayed from the summit
Pipeline:?
Collection Names:


Notes


Ops Rehearsal 4Ops Rehearsal 4: Daily validation pipeline
Pilot/Co-Pilot

Homer Neal / Yusra AlSayyad (both on ET)

channel/contact

#ops-rehearsals

Reps:

(Middleware)  James Chiang  (USDF infrastructure and Raws)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:June 25, 26, 27
Data:Simulated ComCam, replayed from the summit
Pipeline:?
Collection Names:
Notes

parameters of OR4 see: https://lsstc.slack.com/archives/C064M4D6RC4/p1715274916681629

  • How to avoid visit summary issues?
  • Will rapid analysis be used and if so if an issue is seen with an input file should reverting to starting from step1 be done?
Ops Rehearsal 4Ops Rehearsal 4: Prompt Processing of ComCamSim data
Pilot/Co-Pilot
channel/contact

#comcam-prompt-processing

Reps:

Kian-Tat Lim (Middleware)  Dan Speck  (Infrastructure) Ian Sullivan / Krzysztof Findeisen  (Pipeline) Meredith Rawls  (V&V)

Schedule:During each night of ops rehearsal playback June 25, 26, 27, 2024
Data:Simulated ComCam, replayed from the summit
Pipeline:

? (version and the actual pipeline yaml that is run)

Collection Names:


Notes


On goingLATISS Rapid Analysis  at the Summit
Pilot/Co-Pilot <open>/ <open>
channel/contact
Reps:


ScheduleEarliest that pieces be in place to make it possible is April
Notes

I am having trouble envisioning a scenario where campaign management looks after this


On goingLATISS 10am Analysis at USDF  (TBD)
Pilot/Co-Pilot


channel/contact


Reps:


Schedule
Pipeline:
Collection Names:


Notes


















Completed

Ops Rehearsal 4DRP for Template Generation for Ops Rehearsal 4
Pilot/Co-Pilot
channel/contact

#ops-rehearsals

Reps:

(Middleware)  James Chiang  (USDF infrastructure and Raws)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:June 6-12 2024
Data:Simulated ComCam (to be freshly simulated and ready later in May)
Pipeline:w_2024_23, pipelines/LSSTComCamSim/DRP.yaml
Collection Names:
  • /repo/ops-rehearsal-3-prep
  • LSSTComCamSim/runs/OR4_templates/w_2024_23/DM-44718 
  • skymap='ops_rehersal_prep_2k_v1'

Notesticket: DM-44718
Calibrations Rehearsal 1USDF processing of LATISS and LSSTComCamSim calibrations data
Pilot/Co-Pilot
channel/contact
Reps:

(Calibrations)  Christopher Waters 

Schedule:28-30 May 2024
Data:3 days of LATISS and LSSTComCamSim calibrations data
Pipeline:w_2024_21, plus tickets/DM-44548 branches of cp_pipe and cp_verify
Collection Names:
  • repo = /repo/embargo
  • LATISS:  u/huanlin/PREOPS-5181/CR1_LATISS/*
  • LSSTComCamSim: u/huanlin/PREOPS-5182/CR1_LSSTComCamSim/*
Notes

Tickets providing more details:  

  • LATISS:   PREOPS-5181 - Getting issue details... STATUS
  • LSSTComCamSim:   PREOPS-5182 - Getting issue details... STATUS

Only 2 days of LSSTComCamSim data made due to some playback issues.

Ops Rehearsal 3“Intermittent Cumulative DRP” of the 3 nights of ops rehearsal  3 data
Pilot/Co-Pilot
channel/contact

#ops-rehearsal-3

Reps:

(Middleware)  James Chiang  (USDF infrastructure and Raws)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:Starting: April 12th
Data:3-nights of Simulated ComCam, replayed from the summit
Pipeline:d_2024_03_29
Collection Names:
  • repo = /repo/embargo
  • Intra-night (step1 + step2a processing): LSSTComCamSim/quickLook/24
  • Full (step3+) run: LSSTComCamSim/runs/intermittentcumulativeDRP/20240402_03_04/d_2024_03_29/DM-43865
Notes

Setup:
export LSST_VERSION=d_2024_03_29
source /sdf/group/rubin/sw/d_latest/loadLSST.bash
setup lsst_distrib -t ${LSST_VERSION}

Used:
   drp_pipe (tickets/DM-43659)

export DAF_BUTLER_CACHE_EXPIRATION_MODE=datasets=500

Yaml:
/sdf/home/h/homer/proc_comcam_sim/bps_Intermittent_Cumulative_DRP_sasquatch.yaml

Ops Rehearsal 3Ops Rehearsal 3: Daily validation pipeline
Pilot/Co-Pilot
channel/contact

#ops-rehearsal-3

Reps:

(Middleware)  James Chiang  (USDF infrastructure and Raws)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:Day after each night of ops rehearsal playback, April 2,3,4.
Data:Simulated ComCam, replayed from the summit
Pipeline:d_2024_03_29
Collection Names:
  • repo = /repo/embargo
  • Intra-night (step1 + step2a processing): LSSTComCamSim/quickLook/24
    • "quickLook" is a misnomer, this actually nightly validation. Will fix this after the ops rehearsal.
  • Full (step3+) run: LSSTComCamSim/runs/nightlyvalidation/{day_obs}/d_2024_03_29/DM-43612
Notes

Setup:
export LSST_VERSION=d_2024_03_29
source /sdf/group/rubin/sw/d_latest/loadLSST.bash
setup lsst_distrib -t ${LSST_VERSION}

Used:
   drp_pipe (tickets/DM-43659)


Submitting:
In: /sdf/home/h/homer/proc_comcam_sim/ select the appropriate nightly: bps_nightly_validation_onlybatch_sasquatch-copyForNightly20240402.yaml bps_nightly_validation_onlybatch_sasquatch-copyForNightly20240403.yaml bps_nightly_validation_onlybatch_sasquatch-copyForNightly20240404.yaml

Ops Rehearsal 3Ops Rehearsal 3: Prompt Processing of ComCamSim data
Pilot/Co-Pilot
channel/contact

#comcam-prompt-processing

Reps:

Kian-Tat Lim (Middleware)  Dan Speck  (Infrastructure) Ian Sullivan / Krzysztof Findeisen  (Pipeline) Meredith Rawls  (V&V)

Schedule:During each night of ops rehearsal playback, April 2,3,4.
Data:Simulated ComCam, replayed from the summit
Pipeline:

Apr 2, Apr 3: 2.4.0 (d_2024_03_29) 

Apr 4: 2.5.0 (d_2024_04_04)

Collection Names:

repo=/repo/embargo

LSSTComCamSim/prompt/output-<day_obs>

NotesPrompt Processing with LSSTComCamSim ops-rehearsal-3
~3 iterationsDRP on Simulated ComCam data at USDF
Pilot/Co-Pilot
channel/contact

#ops-rehearsal-3-sim/

Reps:

(Middleware)  James Chiang  (USDF infrastructure and Raws)  Yusra AlSayyad (Pipeline) Colin Slater  (V&V) 

Schedule:Iteration #1:Starting 22 February 2024    Iteration#2:Starting 8 March 2024    Iteration #3:Starting 22 March 2024
Data:Simulated ComCam
Pipeline:Iteration #1: Latest weekly after w_2024_08      Iteration #2: w_2024_10      Iteration #3: w_2024_12
Collection Names:

First iteration:

  • repo = /repo/ops-rehearsal-3-prep
  • collection = u/homer/htc-test1

Second iteration:

  • repo = /repo/ops-rehearsal-3-prep
  • collection = u/homer/w_2024_10/DM-43228

Third iteration:

  • repo = /repo/ops-rehearsal-3-prep
  • collection = u/homer/w_2024_12/DM-43439
NotesDRPs in advance of the ops rehearsal are needed for refining the pipeline configs and generating a collection of difference imaging templates. We expect 3 iterations


One TimeInternal Multi-site HSC PDR2 Data Release Production with v24.0.x  (USDF to start)
Pilot/Co-Pilot
channel/contact

#ops-cm-team / Yusra AlSayyad 

Reps:

Yusra AlSayyad (Pipeline) Jim Bosch  (Middleware),  Richard Dubois  (Infrastructure) Colin Slater   (V&V)

ScheduleTo be completed July 2023 – (Steps 1-7 ready for evaluation as of Aug 29, 2023) Faro tract, rollup collection completed Sep 19, 2023.
Data

2023 Internal HSC PDR2 Reprocessing at the USDF 

PDR2:  Last rerun was in 2020:  S20 HSC PDR2 Reprocessing

Pipelinehttps://github.com/lsst/drp_pipe/blob/v24.0.x/pipelines/HSC/DRP.yaml
Campaign Names: "Pilot Campaign" "DRP Test Production Campaign"

Collection Names:


butler: /repo/main ,  HSC/raw/all (has 26K exposure, superset of PDR2 17K),

HSC/defaults (with calibs,skymap,masks, refcats) 

Input for step1: 

 HSC/raw/PDR2/{WIDE,DEEP,UDEEP}

HSC/runs/PDR2/WIDE/v24_1_0_rc2/DM-39132/ancillary

Output for step1:

HSC/runs/PDR2/{WIDE,DEEP,UDEEP}q/v24_1_0_rc2/DM-39132/step1

Inputs for step3:

 HSC/runs/PDR2/v24_1_0/DM-39132/step3_input,HSC/runs/PDR2/v24_1_0/DM-39132/ancillary

Output for step3-6, inputs for steps 4-7:

 HSC/runs/PDR2/v24_1_0/DM-39132/step{3,4,5,6}

Output for step7 (cumulative):

 HSC/runs/PDR2/v24_1_0/DM-39132/step7 (contains step3-6 outputs and ancillary skymap collection)

Faro (tract-based metrics): HSC/runs/PDR2/v24_1_0/DM-39132/farotract

Rollup collection: HSC/runs/PDR2/v24_1_0/DM-39132 (contains step 3-7 outputs, step3 inputs, farotract outputs, ancillary inputs (with skymap)).

NotesProduce pipeline quality metrics on real data, and test multi-site scalability with PanDA.  
RTN

http://rtn-063.lsst.io

One timeData Release Preview 0.2 with DC2 on IDF 
Pilot/Co-Pilot
channel/contact

#ops-df-proecessing / Hsin-Fang Chiang 

Reps:

Yusra AlSayyad (Pipeline) Tim Jenness  (Middleware) Colin Slater  (V&V)

(didn't have an infrastructure rep, because execution and Hsin-Fang were on infrastructure at the time

Pipelinehttps://github.com/lsst/obs_lsst/blob/v23.0.x/pipelines/imsim/DRP.yaml
Schedule
Notes
  • No labels

1 Comment

  1. Moving comment that Eric Charlesdropped in the header to a comment:

    Request from EC:  Could we add some fields to each table: 

    1. what computing resources the will be used to process the campaign,

    2. what tooling (e.g., panda, bps, cm_tools, prompt processing) will be used to process the campaign? 

    3. naming conventions: 

    • for the production (i.e., the set of related campaigns)
    • for the campaign
    • A template for the collection names while the processing is underway, e.g. 'HSC/rc2/tests/{production}/{campaign}` 
    • A template for the final location of the output collection, if it is different