Leanne Guy

Melissa Graham

We need to converge on  RFC-600 - Getting issue details... STATUS

MLG: I think a consensus has been achieved with this version of the draft change request. If you've looked at nothing else, look at that version (it is NOT necessary to read through the comments thread of RFC-600, just look at the final product). 

MLG: NumStreams = 5 to be consistent with LDM-148. Leanne Guyasks: have we looped KT in on this estimate, is it still valid, and when was the last LDM-148 issue

Minor typos:

• 'based only only ' in  Item 1: OSS-REQ-0127 (MLG confirms fixed in her working draft)

NumStreams: 

• 5 is consistent with the sizing estimate from Kian-Tat Lim.
• Agreement that we should define a minimum number that the DMS must support both for clarity and testability. Setting a minimum number that the DMS can support does not imply that LSST operations must support this number of streams. 
• Deciding how many streams will actually be supported in operations is an operations level decision 

• Eric Bellm Organise an effort to understand what exactly what it means to support and additional 5/10 streams. Is it just a matter of allocating more bandwidth? Get  position statement from Margaret Gelman and Kian-Tat Lim26 Jul 2019

Email from Kian-Tat Lim :

Outbound bandwidth is a complicated situation.

NCSA has a certain amount of connectivity to the various science/DOE/general Internet backbones, of which LSST can use a fraction. My understanding is that the amount that LSST can use is set by policy, essentially as "something that would not be disruptive to other NCSA activities". The 10 Gbps number is based mostly on that principle, I believe.

Increasing that number is thus a matter for negotiation. The characteristics of the alert stream (bursty but using bandwidth for ~12 hours every day) may make this harder to price (as opposed to, say, a dedicated channel). In addition, NCSA outbound bandwidth only gets to an Internet peering site; connectivity the rest of the way to the broker must also be assured/obtained.

I don't think that any of the other LSST-contracted networking can be used to substitute for this bandwidth. Most of that is inbound to NCSA.

There are also presumed outbound bandwidth needs for the US Data Access Center that have been part of the sizing model but may not have been fully costed, on the assumption that they would again fall under "non-disruptive". To the extent that that bandwidth has been actually allocated, it could perhaps be
traded off against broker bandwidth.

• Leanne Guy Ensure that the community broker workshop paper includes a statement about the number of brokers we can support and the technical implications 28 Feb 2020 

Gregory Dubois-Felsmann (before meeting): As I recall, the original intent here was that an excess of difference sources in a particular visit, whether astrophysical or spurious, should not, for example, cause the entire system to crash, and that at least the otherwise required number of alerts would go out.  I think there's a lot of room to fine-tune what this means as the requirement is flowed down to the DMSR.

MLG: And in #dm-sst, K-T says: "In the original conception of the system (with a fixed number of processing "strings"), it would not have been possible to trickle alerts out over time. With current potentially more-elastic realizations, and if there's variation in the density of alerts from one CCD to another, it may be possible to trickle them out."

MLG: In RFC-600, Item 6 (in draft_rfc_[date].txt) proposes to flow-down OSS-REQ-1093, which defines nAlertVisitAvg (=10000), to the DMSR. The new proposed DMS-REQ also creates and defines nAlertVisitPeak (=40000). But it does not flow-down or clarify the last sentence of the OSS-REQ-1093 specification for nAlertVisitAvg: "Performance shall degrade gracefully beyond that limit."
Should we:
(a) keep it that way, i.e., not flow-down the 'degrade gracefully' part of the specification,
(b) flow-down that statement as is, or
(c) revise and clarify that statement in the new DMS-REQ which defines nAlertVisitAvg and nAlertVisitPeak?

• Melissa Grahamto propose a clarification to OSS-REQ-0193 to 'propose gracefully' and circulate on   SST slack channel for discussion before updating the  RFC before going to LCR 21 Jun 2019 
• Melissa Graham propose flowdown to the DMSR OSS-REQ-0193 once clarified (i.e LCR accepted)  30 Aug 2019 

• Recommendations on notebooks vs code: Simon recommends to provide a framework of policies for moving from notebooks to libraries of code, conventions, which notebooks are works in progress, which can be CI'd, how is library code stored etc. 
• Robert Lupton is looking more for guidance on tooling, e.g  CI on notebooks, a librarian who curates notebooks,  makes then orthogonal,  merging common work, advice on how to do things. 
• Jim Bosch  Notebooks lead to piles of code  - suggests if orthogonalization is required that work should be moved to libraries of code with more structure and a hierarchy 
• Keith Bechtol I expect that several of the analysis tasks will tend to need functional code combined with more interactive/plotting notebooks
  

• Robert Lupton Dig out the reference to how netflix works with and manages notebooks.   21 Jun 2019 
• Leanne Guy Organize a follow-up discussion a a future SST meeting following  DM-20168 - Getting issue details... STATUS  completion on use of notebooks and libraries of code  30 Aug 2019