DMLT Meeting -2022-01-24

Wil O'Mullane

(Thursday 27th 9am PT) Jan 26,27 .. there is "Action: Allocate more time to in-depth discussions about Data Management at the next AMCR meeting. "  perhaps Leanne, Frossie, Richard(?) and Yusra could be on stand by.  Do any of you want anyone else ?

Here is the minute:"

C6.  Ops and Data Facility Update

The report about the state of the Data Management conveyed confidence that everything is on track. Relatively few facts were provided to back up this statement. This was partly because the time allocated to the DM presentation was too short to get into details.

There is some concern about the Data Release Processing (DRP) plan, where substantial fractions of the data processing will be done in the UK and France. This will require very heavy data movement over large distances. The geographic heterogeneity will create a lot of subtle issues of compatibility and coordination which require a lot more advanced at-scale testing than for a monolithic data processing facility.  Another difference between the two types of facilities is the requirement to use shared infrastructures for the production processing.  The most notable will be the transatlantic links. Some of this testing is obviously going on, but we need to hear more details at the next meeting, also about what risks are remaining and which have been eliminated.  Coordinated scale testing with other facility users can build confidence in this model.  The team needs to consider the impact and mitigation of catastrophic events, such as what happens if the transatlantic lines go down for 3 days? How quickly can they readapt to such a situation?

There was only a passing remark on the prioritization of the different data products, based upon the frequency of their use. This should be a major factor in driving the data delivery system. There was a brief discussion that the Google deployment of the IDF was good for a dynamic scale out to keep up with fluctuations in the end use of the data, but it seems that a 1-minute spin-up for a new instance is inadequate, and this should be improved."

There was also " Present a comprehensive data security plan at the next AMCR meeting.  The plan consists of identifying various ways the data (or parts of the data) can be compromised and countermeasures, including plans for recovering compromised data rather than providing an all-out perimeter defense. The plan should also identify attacks against the derived data products, and their recovery and reconstruction. "

There is an open ticket .. PREOPS-693 - Getting issue details... STATUS

See also LSST Verification & Validation Documentation and docsteady.lsst.io.

Test Plans due in the next 45 days

Milestones due in the next 45 days

• Risk dash board

Risk items needing review

Overdue risks (obligation date passed)

Risk mitigations due at the end of this month

Overdue risk mitigations

AMCR Weds/Thurs for Wil O'Mullane , some others

• Robert Lupton : some camera docs are in SLAC confluence and I am trying to move them
  
  • Leanne, Wil: put in Camera under Commissioning
  • general consensus is that it would be nice not to have this split; but since SLAC is USDF maybe we should bite the bullet and just get everyone SLAC accounts
  • Richard: could all of the SLAC spaces be migrated into project and merged?  all are moving to Atlassian Cloud sometime
  • so in the short term, anyone who wants SLAC confluence credentials can get them "easily"

Frossie Economou