DMLT Meeting -2022-01-10

Wil O'Mullane

• AMCR is most urgent activity (https://drive.google.com/drive/folders/14VRemjISCnX1W16z2itEpGGChG1K9Kvc)
  
  • standard status slides
  • DM a bit more - names of extra people to Nancy
  • for next Monday
• Travel - reduce travel and meetings
  • EPO is having a meeting this month - sanctioned as critical as only 9 months of EPO construction remains.
  • 3 month trips (i.e. to summit ok) 3 people arriving to Chile right now
  • By Feb we hope Omicron is under control but for now perhaps not a good time to plan meetings ..
    • Wil O'Mullane should send a doodle poll
• JSR responses still pending (Not DM )
• Turn schedule into a discussion .. launch from SF3 for a big picture .. communicate it better.
  • perhaps internal reviews
• Wednesday morning strategic planning meeting
  • needs a reset on participation which is an opportunity to change format and time.
  • May overlap with CAP meeting in future.
• Scope options update needed
  • Some end dates are approaching; we need to either drop or extend them
  • DM-1 and DM-7 (maybe) (Scope options: https://docushare.lsst.org/docushare/dsweb/Get/LPM-72/lpm72scopeOptions_rel7.4_20210921.pdf )
• Holiday on 17th
• Email due on Sandrine, Aaron, Steve .. (new PS and deputy Directors).
• ERs full in Tucson - so take care if you are doing anything

(Thursday 27th 9am PT) Jan 26,27 .. there is "Action: Allocate more time to in-depth discussions about Data Management at the next AMCR meeting. "  perhaps Leanne, Frossie, Richard(?) and Yusra could be on stand by.

Here is the minute:"

C6.  Ops and Data Facility Update

The report about the state of the Data Management conveyed confidence that everything is on track. Relatively few facts were provided to back up this statement. This was partly because the time allocated to the DM presentation was too short to get into details.

There is some concern about the Data Release Processing (DRP) plan, where substantial fractions of the data processing will be done in the UK and France. This will require very heavy data movement over large distances. The geographic heterogeneity will create a lot of subtle issues of compatibility and coordination which require a lot more advanced at-scale testing than for a monolithic data processing facility.  Another difference between the two types of facilities is the requirement to use shared infrastructures for the production processing.  The most notable will be the transatlantic links. Some of this testing is obviously going on, but we need to hear more details at the next meeting, also about what risks are remaining and which have been eliminated.  Coordinated scale testing with other facility users can build confidence in this model.  The team needs to consider the impact and mitigation of catastrophic events, such as what happens if the transatlantic lines go down for 3 days? How quickly can they readapt to such a situation?

There was only a passing remark on the prioritization of the different data products, based upon the frequency of their use. This should be a major factor in driving the data delivery system. There was a brief discussion that the Google deployment of the IDF was good for a dynamic scale out to keep up with fluctuations in the end use of the data, but it seems that a 1-minute spin-up for a new instance is inadequate, and this should be improved."

There was also " Present a comprehensive data security plan at the next AMCR meeting.  The plan consists of identifying various ways the data (or parts of the data) can be compromised and countermeasures, including plans for recovering compromised data rather than providing an all-out perimeter defense. The plan should also identify attacks against the derived data products, and their recovery and reconstruction. "

There is an open ticket .. PREOPS-693 - Getting issue details... STATUS

Final decision point on NCSA v SLAC for commissioning Unknown User (mbutler) said we must say if we keep hardware in NCSA in FY23 NOW. Recap - security concerns suggest commissioning data should be on a secure device and transmitted over hardware encrypted routers. SFR5 sets this up for SLAC - technically we therefore could not use NCSA for on sky commissioning data. Richard Dubois are we on track to have facilities at SLAC ?  Victor is a ware of a change in construction baseline and believes it will not be difficult to convince the agencies that they put us in this position.

• Richard Dubois emphasizes this must work, so we will plan on the transition
  • Milestones are in RTN-21

Note also USDF workshop: https://confluence.lsstcorp.org/x/RAN1Cg

Milestone LCR ....

https://docs.google.com/document/d/1LbJBg3thTHkcSMXkdUs1nhrRRKlee1cvPasr0XArghM/edit

See also LSST Verification & Validation Documentation and docsteady.lsst.io.

Test Plans due in the next 45 days

Milestones due in the next 45 days

• Risk dash board

Risk items needing review

Overdue risks (obligation date passed)

Risk mitigations due at the end of this month

Overdue risk mitigations

Frossie Economou