No meeting 2018-12-14 due to large interest in the SQuaRE tech talk
We have quorum for a final meeting of the year on 2018-12-21
Stack club will have a final meeting of the year on 2018-12-21, 1 hour only before the SST meeting
AuxTel meeting on Monday 2018-12-03 brought together all those involved with getting the spectrograph ready for shipping to Chile. The meeting sought to put on the table all issues and work needed to be done and to define criteria to ship the spectrograph. Michael Reuter will lead this activity, all work will be put into a new Jira project: jira.lsstcorp.org/projects/CAP. A verification plan will be built in Jira.
'Mini-broker' vs Simple Filtering Service terminology. Both are used. Leanne Guy proposes to drop 'mini-broker' and only use LSST Simple Filtering Service
There are misconceptions in the community as to what 'mini' means and what 'broker' means.
Melissa Graham points out that it is not a misconception but rather that there is zero conception of either. Core documentation calls it a 'Simple Filtering Service'
DMSR : LSST provides an alert filtering service
John Swinbank prefers to more away from the term 'broker and suggests dropping term 'filter'. What is ’simple’ is subjective.
Colin Slater: Definition of 'simple' is: 'no information beyond the alert package'. It is clear in the requirements that and alert is on a single alert packet.
DM-SST recommends 'LSST Simple Alert Filtering Service' pending a check of the baseline documentation.
The DM-SST proposes the term 'LSST Simple Alert Filtering Service' pending a check of the baseline documentation. We propose to working this change back through all documents to remove the term 'mini-broker'. We do not expect there to be very many occurrences to update.
Eric Bellm Review the core baseline documentation and confirm the terminology used.
Eric Bellm Identify all occurrences of the term 'mini-broker' in DM documentation and update.
Eric Bellm Create RFC to circulate the idea and to update documents
There are formal requirements on second 2 numbers (number of alerts per visit, fraction of visits with delayed/failed alert distribution)
Add completeness and purity (2.5) and quote with appropriate threshold
Formal requirement for the 4th (fraction of alerts per visit with delayed distribution)? no one knows Should we define this?
Does the commissioning process need to demonstrate alert flow to community brokers (CBs)? There are no formal requirements on Community Brokers - they are a third-party service, but they are a key part of our service. We should do all we can to involve Community Brokers in LSST commissioning.
There are formal requirements on the LSST alerts filtering services but there is no discussion on running this service in commissioning. This is entirely data management, i.e there is no dependence on the Camera. We can test and verify the bulk of the simple alerts filtering service in Data Management acceptance testing. Commissioning should still generate alerts end-to-end and test the service but it should be mostly a box-ticking exercise for commissioning This should be discussed with commissioning.
mini-broker: alerts database latency: What is meant by query latency - L1PublicT still applies.
Alert distribution time scale - how is OTT1(60 s) defined:
Leanne Guy thinks the 60s ends when the alert is loaded in Kafka and brokers can start to pull the the alert.
Eric Bellm: thinks we should consider how CBs will access this. We don’t want CBs getting behind distributing alerts. Is it also a perception issue - we make them available in 60s but how long will it take for CB can’t get them? 1.5 - 3 mins probably will have no science impact.
Eric Bellm: bandwidth allocation suggest we can do the processing in 5 secs.
Colin Slater: is reluctant to give the up 60s budget to network transfer. Would prefer we have another budget for network transfer and tack on an extra quantity.
Eric Bellm: Agrees that network latency should not be part of OTT1. We define OTT1 as ‘available to the stream’
Clarify that this is for the last alert out of the visit.
Alert Distribution Science Validation: OTR1
OTR1 is not defined or used elsewhere (that can be found).
Has this been captured in any of the test cases so far. Eric Bellm is not aware but this is not a plan to verify this quantity. Check if it is captured in the science verification.
Colin Slater: wording is confusing. We need some text to summarise this and include reference to Melissa’s definition and ensure that verification plans are consistent with what Melissa defines.
False positives
Where does "50% of the alerts distributed with transSNR>5 are expected to be false positives" come from?
The S\spreadsheet sizing document: LSE-80/LSE-81: is not normative? It is an estimate, not a requirement
Eric Bellm : MOPS is the only place where there is a 50% purity requirement. MOPS prefers high completeness and sacrifices purity. Not everything coming from MOPS has to be an alert?
6sigma is the requirement number for transients 5 sigma is MOPS.
Melissa Graham check if there are formal requirements on the 'fraction of alerts per visit with delayed distribution'
Leanne Guy create ticket to review with commissioning team the subject of end-to-end testing of the LSST alert filtering service in commissioning.
Leanne Guy create ticket to review the role of Community Brokers in commissioning.
Eric Bellm to look in the document on the alert database latency (access) to clarify 'alerts database latency'
We propose to define OTT1 as defined as ‘the last alert of the visit is made available to the stream’
Leanne Guy create ticket for Melissa Graham to check plans to verify OTR1 are captured in science verification plans.
Jeffrey Carlin has been working on the DM acceptance test plan in Jira. He has discovered that there are requirements that are referenced in the DMSR that do not exist - 'dangling requirements'. Turns out that this in an old issue
DM-11029
-
Getting issue details...STATUS
that was never resolved. Tim has extracted the full set of dangling requirements that I think the SST should review and make a proposal in an LCR.
I have started a review page. The goals for today are to
Dominique Boutigny reports that they are investigating 3 ways:
hdf5 / parquet files accessed through Dask and/or Spark (the experts are Michael Wood-Vasey @Yao @stefplaz + probably others). There is also an interface with GCRCatalog
PostgreSQL database based on what HSC has been doing. There is a functional prototype at NERSC with Run1.2 data (the expert is Joanne bogart)
Qserv - There is a functional prototype at IN2P3 on a limited CFHT subset of data and Sabine Elles is struggling to ingest Run1.2. Our intent at IN2P3 is to ingest the full DC2 on a dedicated Qserv cluster.
Source & DIASource table review
Due date changed to push to 2019
AOB
List of SST tasks (Confluence)
Description
Due date
Assignee
Task appears on
Leanne Guy to talk to Science Pipelines (Yusra) about when do this transfer
Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN. Update 2022-02-09: Meeting to resolve this on 2022-02-21
Gregory Dubois-Felsmann / Leanne Guy Create a TN that outlines the science use cases and options for compressed PVIs from AP. Present the issue at DMLT VF2F and then the PST.
Leanne Guy Prepare to consult the PST on the question of providing compressed PVIs for AP outputs, to cover the period before the data become available in a DR.
Parker FagreliusPatrick Ingraham how long will it take to do a scan as described? No need to scan the whole WL range but will require additional points outside nominal lambda range.