Page History

See 2022-04-11 DM-SST Agenda and Meeting Notes: Follow up on Alert Distribution handover to Ops & Descoping the AFS

• We have no formal requirements to verify the functioning of the Brokers, nor are we required to commission them,  so we will refer to the activities to get Brokers ready as "integration" activities, not "commissioning"
• We will send only data that can be made immediately public to Brokers during the integration activities. This reflects how they will work in Operations. This also effectively means that we will have to use alerts based on simulated or precursor data during the integration period.
• We will create alerts from the DC2 / DP0.2 data set for the purposes of integration testing. This is the easiest option for getting alerts quickly and provides a complimentary set of alerts to the DP0.2 dataset. This will also allow the CST to start to develop about workflows for time domain analyses that sit on the intersection between alerts and a data release. 
• Science validation of Brokers will need to be defined with the Brokers. 
• We do not expect to have alerts based on LSSTCam data before System First Light, but they could arrive anytime after SFL. 
• Have an alert distribution system in place that needs to be updated, many column changes, etc. Not sure it is functioning at the moment. 
• Testing the interface with the Brokers 
  
  • For stress testing, it would also be beneficial to set up dedicated campaigns rather than just an alert stream that is always running 
  • RD concerns:
    • always harder to fill the pipe than you think + self-inflicted egress problems. Everything takes longer that one thinks
    • want to find problems early to address them in good time. 
  • 17Gbps (all brokers total) out the front door will not be a problem 
  • We will start internal load testing without Brokers, then move to end-to-end testing with the Brokers in the loop
  • Hardware at SLAC - were some discussions, none since. Doable for all Brokers if they want it. 

•  Eric Bellm Leanne Guy discuss with Broker teams what it means to scientifically validate a Broker.   11 Dec 2023  2024-02-26: Only way to validate scientifically is to use real data. Engage SP-CST to produce tutorials/examples in collaboration with Brokers. Work with SP-CST to assist. Also ties in with nightly reports.  

draft for discussion: https://dmtn-226.lsst.io/v/DM-38805/index.html

• The Rubin AFS will be provided by ANTARES. Strategy is in dmtn-226 and a memo will be made to announce it formally. 
• The ANTARES-provided Rubin AFS must be commissioned and verified by Rubin during the commissioning period.
• We will verify the filtering by creating real filters (20 in ANTARES to work with Rubin alerts. Do not need to be the final science filters for testing. 
• Number of simultaneous users not relevant in the kafka world especially without a latency requirement, it's just – how many filters are loaded and how are they running. 
• Portal requirements on interfaces to the AFS could be descoped or just claimed via ANTARES/AFS 

•  Leanne Guy  talk to Gregory Dubois-Felsmann to review the original intent of the AFS-related Portal requirements before deciding on a course of action 29 May 2023 

Leanne Guy

Next meeting is: 08 May 2023