Date & Time 

  11:00 PDT

Location

Browser

Room System

Phone Dial-in

https://bluejeans.com/103664856

  1. Dial: 199.48.152.152 or bjn.vc
  2. Enter Meeting ID: 103664856 -or- use the pairing code

Dial-in numbers:

  • +1 408 740 7256
  • +1 888 240 2560 (US Toll Free)
  • +1 408 317 9253 (Alternate Number)

Meeting ID: 103664856

Attendees

Regrets


Discussion items


ItemWhoNotesConclusions and Action Items
Project/Science Updates

Leanne Guy

No PST meeting last week. No meeting this week either.


Summary of DESC meeting

Robert Lupton Jim Bosch Leanne Guy

Summary of DESC meeting and any key takeaways for DM:

  • General feeling that talks were well received and that DESC are receptive to and interested in working with us.
  • Collaborations with Piff folk and WL team seem to be proceeding well. WL has picked up 1/3 Arun's time.  
  • For next meeting we should present numbers demonstrating how how well we are doing
  • Robert Lupton Leanne GuyEnsure that the next presentation to the DESC presents a quantitative assessment of how we are progressing.   
Ideas on using Kafka streams in DM 

Eric Bellm & Spencer Nelson

Discussion of current and potential alternate approaches to alert distribution.

Slides

One major question: if there was an approach that let us distribute a larger number of streams, would we short-circuit the broker selection process?


  • Eric and Spencer presented ideas for  a split messaging (notification) and data distribution architecture for the alert streams.
  • Currently the alert packets contain all the information about a DIASource in the alert packet and consequently are very big (~100kb). This is a problem for scaling the alert stream and serving it to a larger audience.
  • The new proposal involves separating the distribution of the alerts from the notification notification the alert packets thus reducing drastically the size of the alert packet and enabling distribution to a wider audience, potentially thousands.  
  • Users could potentially access the stream on their laptops running a small kafka client. 

  • The alert database is meant to be permanent, this system means that we’d have to keep these URLs functional for decades.  

  • This new transport mechanism does not also need to replace the alerts database - but then this will result in 3 systems needing to be kept in referential integrity. 
  • Should present these ideas to the broker proposers - some initial enthusiastic feedback from Antares. 
  • Much discussion around data rights and the community expectations: 
    • This system puts URLs in alerts packets instead of pixels and data. The public access the alert packets via brokers but then only data rights holders can access the data via the URL. 
    • The project has never promised a subscription to anyone other than the brokers – DPOL-503: “the complete alert stream will not be distributed directly to individual investigators, but rather will be delivered to a selected set of community brokers”.
    • This potentially results in a situation whereby the public can access the alert packets but only data rights holders can access the data. Even if this was technically always the case - this is not what the community are expecting 
  • What, if any, are the implications for  OTT1? Is the interpretation that the whole packet must be delivered? The DMSR clearly states deliverables within OTT1 but this is not true at the SRD level. 
  • What, if any, are the implications for the broker selection process? If this is a useful model, can we skip the broker proposal process or have a more lightweight one – need to work with the SAC.
  • Eric Bellm Present these ideas at the PCW to broker proposing teams and the community to get community perspective  
  • Leanne Guy Schedule follow-up DM-SST discussion on emergent topics around community access to data.    : 2021-01-25:Will discuss at DMLT 2021-02-23 and take to PST if necessary.
Algorithms follow-up workshop at PCW

Leanne Guy

The PCW is in 2 weeks.  We have a follow-up session organized to address questions from the algorithms workshop. There is a form for submission of questions and we have several questions that came from the workshop itself.  No complaints
AOB

Leanne Guy




List of SST tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN.  Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Gregory Dubois-Felsmann check if SDM standardization is adequately represented in project documents, and whether DMTN-067 should be required.
31 Mar 2022Gregory Dubois-Felsmann2022-02-14 DM-SST Virtual F2F Agenda and Meeting notes
28 Feb 2023Leanne Guy2023-01-23 DM-SST Agenda and Meeting Notes
  • Leanne Guy talk to Steve R about presenting plans for the ShearObject table to PST and SciCollab chairs   
20 Mar 2023Leanne Guy2023-02-27 DM-SST Agenda and Meeting Notes
31 Mar 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
  • Leanne Guy  talk to Gregory Dubois-Felsmann to review the original intent of the AFS-related Portal requirements before deciding on a course of action  
29 May 2023Leanne Guy2023-05-01 DM-SST Focus Meeting - Brokers in Commissioning
  • Leanne Guy Prepare to consult the PST on the question of providing compressed PVIs for AP outputs, to cover the period before the data become available in a DR.  
02 Jun 2023Leanne Guy2023-03-27 DM-SST Agenda and Meeting Notes
  • Jim Bosch Incorporate 30-60 day period for raws on disk into the strawman proposal and present to KT  
26 Jun 2023Jim Bosch2023-05-08 DM-SST Agenda and Meeting Notes
  • Parker Fagrelius Patrick Ingraham  how long will it take to do a scan as described? No need to scan the whole WL range but will require additional points outside nominal lambda range.  
30 Jun 2023Parker Fagrelius2023-03-27 DM-SST Agenda and Meeting Notes
31 Jul 2023Colin Slater2023-07-10 DM-SST Agenda and Meeting Notes
  • Eli Rykoff , Leanne Guy  Develop a proposal for what calibration processing, hardware, data we actually need and what will be needed for DR1. This has implications for the ORR and for prioritisation of work in commissioning  
31 Jul 2023Eli Rykoff2023-01-30 DM-SST Agenda and Meeting Notes
  • Yusra AlSayyad will look to see if there is any effort to help on option 1  
28 Aug 2023Yusra AlSayyad2023-08-14 DM-SST Agenda and Meeting Notes
  • Jim Bosch  Provide a physical example of that a  up on cell table would look like fo the Colin Slater / DAX team to review  
31 Aug 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
  •  "What is the pathway to defining the data products that are required to meet DMS-REQ-0266" Jeffrey Carlin   
30 Nov 2023Jeffrey Carlin2023-10-23 DM-SST vF2F Agenda and Meeting Notes
30 Nov 2023Gregory Dubois-Felsmann2023-10-23 DM-SST vF2F Agenda and Meeting Notes
30 Nov 2023Leanne Guy2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  • Jeffrey Carlin follow up with KT on DMS-REQ-0176 and DMS-REQ-0315 to update/disaggregate this for latest base/summit infrastructure split.  
30 Nov 2023Jeffrey Carlin2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  • Jim Bosch Follow up on the possibility of investigating further the ability to process 2 collections in parallel.   
31 Jan 2024Jim Bosch2023-12-04 DM-SST Agenda and Meeting Notes
31 Jan 2024Jeffrey Carlin2023-12-04 DM-SST Agenda and Meeting Notes
Gregory Dubois-Felsmann2023-10-23 DM-SST vF2F Agenda and Meeting Notes