Robert Gruendl is leading this effort.
LDM-643 (Section 3) now has a first draft of a description for OPS rehearsal #2
The pre-operations team feel using AuxTel for this rehearsal would provide maximum end to end realism.
(Though Corvid-19 may prevent that)
Personnel needed to execute the rehearsal - Wil O'Mullane Chuck Claver Unknown User (mbutler) with DM CAMS should identify staff for the rehearsal.
- SIT-Com: Tiago, Patrick - if AUXTel with LATISS these two should be on site in Chile - and Brian (possibly Michael Reuter for EFD)
- Data Production QA : Lauren , Yusra AlSayyad since it looks like we will be defining procedures as we go, others ..
- Data Production Facility : Michelle B, Robert G ,
Participants
Agenda
- decide if we do it with simulations from the base or delay to summit reopening.
- basically set a schedule
- if we use LATISS agree ground rules with Patrick (not to interfere with engineering work)
- comcam?
- give some clarity on personnel involved and what role they play
- give some clarity on goals such as
- which procedures we might develop
- which requirements could potentially verify (Jeff Carlin)
Notes
Goals:
- Refer to LDM-643.
- “Prepare for ComCam on the sky”.
- How will observing proceed?
- How will changes be folded in?
- How will processing be incorporated into the daily operations of the system?
- Inform e.g. our approach to change control: how do you change how the DAC is read out? How do you fall back to a previous system?
- Exercise whatever systems we have available, or use shims to make the rehearsal possible.
- However, the real aim is the interaction between the various teams (summit, Tucson, LDF); reality of hardware etc is not fundamental.
- However, using hardware to make the rehearsal “hands-on” makes it more compelling.
- Previous rehearsal was based on simulated data.
- This is a commissioning rehearsal, rather than preparation for a steady state operations scenario.
- Expect to plan a twist for mid-rehearsal; assume that everything runs smoothly on the first day.
- Needs a “ringmaster” to make this happen.
Outstanding questions:
- Which instrument should we use to drive the rehearsal? Simulations, or AuxTel/LATISS, or ComCam?
- In the near term, AuxTel will be in stasis; ~months before it is back online.
- Probability ComCam will be running in La Serena at some level, but it's not clear what level of connectivity it will have.
- Wil reckons “quite a bit”; it's in the data center.
- A moving target in terms of availability.
- Could be used to generate biases, darks, flats.
- Could play fake images through the NCSA test stand. This obviously doesn't exercise ComCam hardware.
- Can inject “interesting” images, but it takes some effort.
- Decision: target ComCam in the Recinto.
- Backup plan: use test stand at NCSA if ComCam is not available. Specifically, the date takes priority over the hardware availability.
- What compute hardware do we use?
- The commissioning cluster should be available.
- Also want to exercise batch compute as part of this exercise at NCSA.
- When do we feel we need to run this rehearsal?
- Do we need to rethink ops rehearsals given our current situation? E.g. introducing more rehearsals in future, as things start coming back online.
- Note there is a pre-ops milestone which may fold in to this rehearsal.
- That milestone implies we want this done by the end of June.
- Aiming for May gives enough time to get ComCam up and running, but likely the summit will still be closed so people will be available.
- There is still some managerial work to be worked out to get ComCam going.
- As above, date takes precedence, and we use the test stand if ComCam isn't available.
- Some uncertainty about the availability of code to perform ingestion by May.
- Note DMLT meeting, DES collaboration meeting in May.
- Decision: Doodle poll key individuals for three days in May.
- But can reconsider if ingestion is unavailable on that timescale.
- How long should the rehearsal last?
- Three days last time; allows time for handovers, debriefs, etc.
- Decision: the duration will be at least three nights (four calendar days).
- If the exercise is run during the day using ComCam in Recinto, then the equivalent of three nights is fine.
- Who are the key personnel?
- Chile: Kevin Reil, Brian Stalder (likely as observing scientist; alternatives are Sandrine Thomas, Chuck Claver, Kevin Reil, Tiago Ribeiro), Cristian Silva
- Tucson: Michael Reuter, Patrick Ingraham (& Chuck as observer).
- Execution staff / NCSA: Monika Adamow, JD Maloney, Robert Gruendl, Matt Kollross
- SDQA: Lauren MacArthur, Jeff Carlin
- Depends on exactly what pipelines get run, what high-level requirements can we test?
- Pipelines support (on call): Merlin Fisher-Levine
- Camera support (on call): Tony Johnson, Htut Khine Win, Steve
- Director: Bob Blum
- Several observers.
Next steps: