DMLT Meeting -2021-11-29

• Kevin Long - sent out request for schedule status.
  • Jira updated by tomorrow please (last day of the month)
• Summit power outage and stop work
  
  •  power overdraw from coating chamber - report later today or tomorrow

  • critical network systems were on the facility UPS (which failed for some reason).

    • CS: something was wrong; we were supposed to be behind the UPS. 
    • WO: electricians need to investigate
    • CS: We need protection for the coating chamber. If this happens again, it'll be bad. 

  •  still no alarms when the summit connection goes down (working opsgenie)

    • WO: need an out of bounds network. I don't know why radio wasn't used in this case. No one went on the radio and alerted them. 

  • still no summit-base connection redundancy (LTE/4G, microwave, satellite?). (on the way)

    • CS: got a quote from the vendor last week. Going to use the gemini antennae.

    • WO: haggle a bit, but get on with it because we need it in place.

  •  the procedure to bring network systems up from zero is woefully out of date-(this was the post covid unique startup but its a fair point we should work on it)

    • Lost some hardware. It was carnage for "just a power cut"
  • Chuck - cascading failures
    • its no longer a pure construction site
  • Kevin - communications poor. There was no broadcast on radio, no internet. There was a stop work called by email earlier in the week but people did not stop - (Victor) all points were well understood and a go ahead was agreed. Email is not the way to get a stop order (ther emay be hesitance for stop work also). Phrasing was "safety hazard I think we should not proceed". Then at the power cut there was no stop work order called.
    • Gessner - misunderstanding of what happened, power outage and domino effect - the summit did not think of this as critical (no visible imminent danger). But it was a circuit could have started a fire.
    • Anyone can call stop work (it does not have to be imminent danger) .. that should cause people to pause even if they continue to  safe position.
  • Frossie who is the designated authority - or on the ground ops manager - Chuck states the default person is himself. Chuck Gessner states its the site manager Eduardo or Oscar if he is not there .. but Jacques was the one who made the call on the critical lift.
    • Every time we asked, we got another answer, which wasn't confidence inspiring.
    • FE: There's needs to be a captain of the ship
  • Victor says we have an emergency list and we could ask people to work out of hours.
    • Giovanni is working on the call list
    • it has not been updated for a while
• Veronica - needs approval authority for TER.
• All hands Dec 9th .. confirmed (right after camera team all hands)
• AAS -
  • Masks are not expense-able apparently

FE: Major upgrade during thanksgiving but we'll need to do another one. Andes(sp?) did not come up from the power-outage.  Decided not to interrupt thanksgiving dinners. Yagan (sp?) is coming up today. 

CS: Andes is gone and we will never resurrect it. Convo about renaming Yagan to Andes. 

Especially any help needed..(could be TCAM )

Victor asks if everything is now down for the backup link.

CS: lost leaf switches, 3 Andes nodes, and several optic units, but have spares. 

See  #inclusion and https://www.lsst.org/about/dei

Community link: https://community.lsst.org/c/eji/45

JDOR on Wednesday, Thursday, out brief on Friday

Should be done for Status meeting Thursday.

KT: I'm going to take some time off Thurs/Friday because I worked through Thanksgiving weekend. 

as needed.

No business

WO: You get a break this week.