Versions Compared


  • This line was added.
  • This line was removed.
  • Formatting was changed.

Robert Gruendl is leading this effort.

LDM-643 (Section 3) now has a first draft of a description for OPS rehearsal #2

The pre-operations team feel using AuxTel for this rehearsal would provide maximum end to end realism.

(Though Corvid-19 may prevent that)

Personnel needed to execute the rehearsal - Wil O'Mullane Chuck Claver Michelle Butler  with DM CAMS should identify staff for the rehearsal.

  • SIT-Com: Tiago, Patrick - if AUXTel with LATISS these two should be on site in Chile - and Brian (possibly Michael Reuter for EFD)
  • Data Production QA : Lauren , Yusra AlSayyad since it looks like we will be defining procedures as we go, others ..
  • Data Production Facility :  Michelle B, Robert G ,

OPS rehearsal II discussion (2020 March 31; 15:00 Pacific;



  • decide if we do it with simulations from the base or delay to summit reopening.
    • basically set a schedule
    • if we use LATISS agree ground rules with Patrick (not to interfere with engineering work)
    • comcam?
  • give some clarity on personnel involved and what role they play
  • give some clarity on goals such as
    • which procedures we might develop
    • which requirements  could potentially verify (Jeff Carlin)



  • Refer to LDM-643.
  • “Prepare for ComCam on the sky”.
    • How will observing proceed?
    • How will changes be folded in?
    • How will processing be incorporated into the daily operations of the system?
  • Inform e.g. our approach to change control: how do you change how the DAC is read out? How do you fall back to a previous system?
  • Exercise whatever systems we have available, or use shims to make the rehearsal possible.
  • However, the real aim is the interaction between the various teams (summit, Tucson, LDF); reality of hardware etc is not fundamental.
    • However, using hardware to make the rehearsal “hands-on” makes it more compelling.
    • Previous rehearsal was based on simulated data.
  • This is a commissioning rehearsal, rather than preparation for a steady state operations scenario.
  • Expect to plan a twist for mid-rehearsal; assume that everything runs smoothly on the first day.
    • Needs a “ringmaster” to make this happen.

Outstanding questions:

  • Which instrument should we use to drive the rehearsal? Simulations, or AuxTel/LATISS, or ComCam?
    • In the near term, AuxTel will be in stasis; ~months before it is back online.
    • Probability ComCam will be running in La Serena at some level, but it's not clear what level of connectivity it will have.
      • Wil reckons “quite a bit”; it's in the data center.
      • A moving target in terms of availability.
      • Could be used to generate biases, darks, flats.
    • Could play fake images through the NCSA test stand. This obviously doesn't exercise ComCam hardware.
      • Can inject “interesting” images, but it takes some effort.
    • Decision: target ComCam in the Recinto.
      • Backup plan: use test stand at NCSA if ComCam is not available. Specifically, the date takes priority over the hardware availability.
  • What compute hardware do we use?
    • The commissioning cluster should be available.
    • Also want to exercise batch compute as part of this exercise at NCSA.
  • When do we feel we need to run this rehearsal?
    • Do we need to rethink ops rehearsals given our current situation? E.g. introducing more rehearsals in future, as things start coming back online.
      • Note there is a pre-ops milestone which may fold in to this rehearsal.
      • That milestone implies we want this done by the end of June. 
    • Aiming for May gives enough time to get ComCam up and running, but likely the summit will still be closed so people will be available.
      • There is still some managerial work to be worked out to get ComCam going.
      • As above, date takes precedence, and we use the test stand if ComCam isn't available.
    • Some uncertainty about the availability of code to perform ingestion by May.
    • Note DMLT meeting, DES collaboration meeting in May.
    • Decision: Doodle poll key individuals for three days in May.
      • But can reconsider if ingestion is unavailable on that timescale.
  • How long should the rehearsal last?
    • Three days last time; allows time for handovers, debriefs, etc.
    • Decision: the duration will be at least three nights (four calendar days).
      • If the exercise is run during the day using ComCam in Recinto, then the equivalent of three nights is fine.
  • Who are the key personnel?
    • Chile: Kevin Reil, Brian Stalder (likely as observing scientist; alternatives are Sandrine Thomas, Chuck Claver, Kevin Reil, Tiago Ribeiro), Cristian Silva
    • Tucson: Michael Reuter, Patrick Ingraham (& Chuck as observer).
    • Execution staff / NCSA: Monika Adamow, JD Maloney, Robert Gruendl, Matt Kollross
    • SDQA: Lauren MacArthur, Jeff Carlin
      • Depends on exactly what pipelines get run, what high-level requirements can we test?
    • Pipelines support (on call): Merlin Fisher-Levine
    • Camera support (on call): Tony Johnson, Htut Khine Win, Steve
    • Director: Bob Blum
    • Several observers.

Next steps: