Skip to end of metadata
Go to start of metadata

Date & Time 

  11:00 PDT

Location

Browser

Room System

Phone Dial-in

https://bluejeans.com/927260081

  1. Dial: 199.48.152.152 or bjn.vc
  2. Enter Meeting ID: 103664856 -or- use the pairing code

Dial-in numbers:

  • +1 408 740 7256
  • +1 888 240 2560 (US Toll Free)
  • +1 408 317 9253 (Alternate Number)

Meeting ID: 927260081

Attendees

Regrets

Discussion items

 ​

ItemWhoPre-Meeting NotesNotes and  Action Items
Project/Science Updates
  • Excellent talk from Eli at last week's PST meeting on calibration
  • SCOC workshop on cadence this week
  • Document written by Chuck, Leanne, Robert, Brian, etc on missing scope. Many entries relevant to DM/DM-SST 
  • Rubin-Euclid DDP report. Where/Who will be serving them?  Out of scope for this report, this is the next step. Rubin is not expected to produce/host/serve them as part of it's current scope. 
News from Broker Technical Workshop 

Eric Bellm (on SOC)

  • Meeting website
  • No major reactions to Spencer's outline of Alert Distribution interfaces (auth, etc.)
  • Brokers continue to be very unsure about practical limitations of access to data products through the Science Platform & it's clear this is hindering their planning
  • Interest in bulk access to historical alerts (~tarballs) in addition to present DMTN-183 design
  • Discussion among brokers about potentially using a shared alert archive
  • Francisco Forster strongly arguing for larger cutouts–an interesting proposal to make them multi-resolution to keep the overall size (in bytes) similar.  Followup experiments on ZTF confirm (moderate) improvement in image-based classification (slides).
  • Spencer is getting alert distribution set up at IDF right now. 
  • Cutouts, access to PPDB. Eric pointed them to sizing model document but community still not sure how to plan.
    • Access to alert archive has not been confronted yet 
    • Community concerned about volume limitations, 1, 10, more cutouts.
    • Cloning the PPDB, dump of the diff? 
    • What is not working about the standard cutout service? Science platform was not designed for this. Careful about pushing the RSP into a role it was not designed for. 
    • Also supplemental queries to what is in the packet, cutouts larger than alerts, 
    • Should we give larger cutouts in the first place. 
    • Collect a list of issues ahead of time to think about our responses. 
  • Also uncertainty on our side. 
  • Spencer has a  prototype for the alert archive, focus on small searches, pick out individual alerts.
  • Alerce have a deep NN classifier on the ZTF stamps , variable, AGN, SN classification. FF concern is that the current cutout size prevents classification.  Proposal is a hierarchical resolution - larger spatial area but same overall number of pixels. 
    • Is this the right approach?
    • How do we represent the data?
    • How do we decide the scale?
  • Little concern from other brokers. Alerce is doing the most stamp processing.
  • GPDF: No real concerns, good experiment.
  • Are the coarser pixels for the central regions included? FF used annuli to avoid duplication and for size concerns.  
  • Can pack this into a FITS file but an image viewer will not recognize it as an image. 
  • Could also adopt a stricter lossy type compression. Most compression algorithms are not designed around small things. 
  • How much are they looking for in the template images vs difference images. 
    • Orig. pres is about host association → template 
    • What about a low resolution as well?
    • Could we have both options?
  • Do brokers need to take everything? Can throw away what people want to keep. Indicated that we can remove some parts of the alerts, e.g remove the history and have just the Object/DIASource record. No team has committed to this yet. 
  • Eric Bellm organize an informal round-table with the brokers to understand their needs.  
  • Eric Bellm will follow up on the topic of larger/multi-resolution cutouts with Francisco F and report back in ~ 4 months  
Descoping the Alert Filtering Service 
  • Proposal to descope the alert filtering service 
  • strong ensemble of community brokers lessens the need for an on-project backstop service
  • Required system has challenging implementation characteristics and only iffy performance
  • DMTN-165 hybrid alert approach remains viable & interesting but hard to prioritize it over other work
  • staffing challenges
  • Requirements to strike:
    • DMS-REQ-0348
    • DMS-REQ-0343
    • DMS-REQ-0342
    • LSR-REQ-0025 is missing?  flows down to DMS-REQ-0342
    • LSR-REQ-0026 ?
  • POC on AFS done, claimed the milestone but a lot of work to do to build it. 
  • Formally retire the risk on "no brokers" 
  • KSK: This was the only interface that promised RT access to the alert. No broker is under any requirement to turn around the alerts in any given time. AFS was for the 
  • EB: No latency requirement on the AFS, OTT1. We would aim to but we have no requirement. 
  • Brokers will be motivated to make the turnaround fast. We can follow up. 
  • AFS: Also only place where the whole DR community could put their filters. 
  • CS: latency could be backfilled with PPDB broadly addressing some use cases - can the latency be reduced. Significant implications for the architecture. 
  • AP would be better served focusing on algorithms and pipelines. 
  • GPDF: Want to be careful about descoping a promised service. 
  • ZI: put the filtering service into the SRD, there was not guarantee that any broker would be funded. If antares is the broker at the US national level, we could descope. Need some agreement btw NL, NSF, Rubin. 
LCR for Three-stamp LCR  Alerts
  • LCR still being drafted, to combine with above descope
    • change would be to DMS-REQ-0274
    • (is an LCR required here?  Additional contents don't violate the requirements...)
    • Alert size is only specified in the DPDD
  • Eric Bellm Create RFC for three-stamp alerts, to communicate and garner feedback  
  • Two-stamp approach is mentioned in change-controlled design documents (LDM-151 and LDM-612) in addition to the DPDD; these would need to be changed in any event.
Alert Distribution handover to Ops
  • The alert distribution system is reaching maturity and we would like to consider handing over to ops. What do we need to demonstrate to deliver on the construction requirements? What can be passed to ops (eg P2/3 requirements).
    • DMS-REQ-0002 (1b)
    • DMS-REQ-0391 (2)
    • DMS-REQ-0393 (2) ?
    • DMS-REQ-0094 (1b) ?
  • Planning for an acceptance test campaign.
  • Prompt Processing system adjacent ot the AD system. Had nover one without handing over the other. 
  • Can we really test and verify AD without having any online processing. 
  • Decouple the verification of requirements from the handover to ops. Look to put together a test plan now and test on precursor datasets at the IDF. Then later look to rerun at SLAC later with LSSTCam data. 


  • Eric Bellm Leanne Guy  to work on a plan to hand over the Alert Distribution System to Ops  
AOB

Next meeting is   



List of SST tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Eric Bellm Create RFC for three-stamp alerts, to communicate and garner feedback  
31 Dec 2021Eric Bellm2021-11-15 DM-SST Agenda and Meeting notes
  • Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN.  Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Robert Lupton and Leanne Guy — review all columns in the Source & DIASource table and decide whether to remove or move to a different table. The decisions on justification should be recorded in these tables. Proposed changes will be reviewed at a future SST meeting before going to LCR. Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Gregory Dubois-Felsmann Submit an RFC for the compressed-PVI requirements. Make clear that the requirement on quality may require further discussion can be postponed, and should not delay moving forwards on the 6 functional requirements, which need to be approved soon.   
27 May 2022Gregory Dubois-Felsmann2021-01-11 DM-SST Agenda and Meeting notes