Logistics

Date , 10:00 PT

BlueJeans (note the new number)https://bluejeans.com/583102117

Attendees 

Regrets


Weekly meeting of the DM-SST to discuss scientific aspects of the Data Management System

Discussion items


ItemWhoNotesConclusions and action items
Updates
  • Date and location for algorithms workshop is now set:
    • 4-6 Dec 2019, Princeton area (i.e not necessarily on the Princeton campus).
  • Leanne Guysend save-the-date notes out  
Suggestions from the DM-SST for PCW sessions

Melissa Graham  is leading the SOC and Colin Slater  is on the POC. Some possible ideas (MLG seeded draft session proposals below, please update them): 

  1. User-Generated Data Products
    • We concluded that there is not enough concrete implementation of a system for UG data products yet to warrant a session on this topic. 
    • We have presented the requirements many times but the interfaces are not ready
    • WebDav is maturing but nothing concrete yet
    • Proposal & implementation  from CADC on providing via TAP service
    • Does not cover all use cases but in moving in the right direction
    • No DB backend yet
    • Basically nothing demo-able
    • Conclusion: too early to  show any sort of demo on this yet 

  2. Photometric redshifts (MLG will liaise with DESC PZ WG)
    • Conclusion: Submit a proposal for a session. 
    • Melissa Graham will lead this from the DM side, Jim Bosch Colin Slater Leanne Guy Zeljko Ivezic should also plan to attend. Robert Blum will be invited as well. 
      • DM will communicate the current baseline and issues as we see it such as: 
        • Do we just take a published algorithm or someone's table and publish it?
        • Do we train models or do we take a trained model from the community?
        • How will photo-Zs be validated in commissioning?
  3. Run a session during the time-domain day-of-focus on working with time series in the LSP using precursor data 
    • We have stripe83 and WISE catalogs but no time series model to show  
    • IVOA have done a lot on TS data models – will be a focus at the Paris meeting next week
    • What do users expect in terms of a time-series data model and associated attributes on time series?
    • Feedback from the LSP review is that they want not just aggregated information and time series, but also ready access to postage stamps for the time series. They did not like the idea of batch service for retrieving a bunch of objects. 
    • Conclusion:  we are not ready to show any sort of demo on this yet but will participate in a requirements gathering session around working with time series as part of the time-domain data products session by Michael Wood-Vasey
    • Gregory Dubois-Felsmann Melissa Graham Eric Bellm Leanne Guy at least should be present. All welcome.

  4. Team up with stack-club folk to run a tutorial session on doing LSST science with the LSP 
    • Gregory Dubois-Felsmann is worried about creating expectations that we cannot meet. 
    • The stack-club model is good in that people who use the platform contribute something  (analysis notebooks) - it is not just a free-for-all
    • Phil  is open to extending this to include LSP usage and not just the stack. Perhaps as one team or 2 separate (LSP/Stack)
    • Conclusion is that we will wait until we have HSC and Gaia data and re-consider this again next year

  5. A deep-dive into the LSST algorithms would be great but that is what the workshop later this year is for and I don’t want to do the same thing twice. We should have a good idea of the status of 
    • No disagreement
  • Leanne Guyto submit proposal for session Photometric Redshifts in the Data Release Data Products (Melissa Grahamis on holiday)  
  • Leanne Guy make sure Bob Blum is apprised of plans for a photo-z session at the 2019 PCW   
Ops rehearsal

Summary of what happened 

  • Transferred raft scale data from base to NCSA each morning, ingested into a Gen2 butler and ran through processCCD (ISR) 
  • rsync with a file sent every 20sec. total throughput 2MB/s
  • More like a normal observing run and not commissioning - things will change more frequently in commissioning. 
  • latency for base → NCSA transfers:  1 sec base 
  • Colin Slater added that many lessons were learnt on the science side

    • It is important to have good QA and visualization to see how the night proceeds rather than a big results file to go away and look at. This should be raised to the successor of the QAWG

    • There were something like a few dozen failures in a thousand (failure == fail to process an image to produce photometry)

    • Lauren understood the causes  immediately. Bugs were already known and Jira tickets exist already

    • Robert Lupton points out that the correct process is to process all the images to completion w/o fail and apply science criteria at the end

Following related discussion on data transfers for AuxTel data, the topic of latency and access to AuxTel data will be raised at the upcoming DMLT-F2F

  • John SwinbankAdd agenda item for AuxTel data transfers, ingestion and access via LSP to agenda  
Algorithm classification

Colin Slater has made a start on the  algorithm classification

Some updates made during the meeting - SST folk are requested to add their inputs before  . The goal is to identify which of these the SST can contribute to,  assessing options to identify a course of action. 

Colin Slater and Leanne Guy will use this to build an agenda for the F2F meeting.

  • Colin SlaterAdd  connection to LDM-151  to the algorithms classification page   
Review open tickets 

AOB
No meeting next week due to absences

Draft Session Proposals to Submit (please feel free to edit/reword)


Title: Photometric Redshifts in the Data Release Data Products

Session Abstract: The goals of this session are: (1) to hear from a diverse cross-section of the science community, and LSST EPO, about their science use-cases for photo-z in the DR DP; (2) to discuss the various options for photo-z generation and storage in the DR catalogs; and (3) for DM to describe their constraints on generating/storing photo-z, and/or any additional input needed from the science communities, and to present a timeline for making decisions about photo-z in the DR DP.

Discussion:  Leanne Guy discussed with Jim Bosch and Colin Slater the possibility of including the photo-z community in choice of photo-z algorithm and validation of data products prior to a DR via one or more pre-DR sprints. 

Suggested Audience: scientists, data management team, students
Category: Science, Data Management
To Be Submitted ByLeanne Guy on behalf of Melissa Graham


Title: User Generated Data Products

Session Abstract: The goals of this session are to: (1) inform the science community about the scope of the Prompt and Data Release data products, and where User-Generated data products might be needed; (2) allow the science community to share their identified use-cases and needs for UGDP, and potentially identify cross-cutting opportunities for collaboration; (3) have DM illustrate a path forward for incorporating UGDP (and/or their codes) into the data products.

Suggested Audience: scientists, data management team, students
Category: Science, Data Management
To Be Submitted By(name)


Title: LSST Science Platform (LSP) Tutorial: Working with Time Series Precursor Data

Session Abstract: This session will be a prepared by data management staff. The goal will be for the DM team to demonstrate to the science community how to work with time series data products from precursor data available through the LSP. (More details here!!)

Discussion: Feedback from  the LSP review: `It was felt all users would expect that a time-domain survey such as LSST should support fundamental time-domain functions such as the ability to visualize and analyze light curve data.`

Suggested Audience: scientists, data management team, students, stack club members
Category: Science, Data Management
To Be Submitted By(name)


Title: LSST Science Platform (LSP) Tutorial: How to do science in the LSP

Session Abstract: This session will be a jointly prepared by data management staff and LSST Stack Club members. The goal will be to demonstrate, in a hands-on tutorial-like setting, how to do LSST-like science with the codes and precursor data available in the LSP. (This session will be suitable for newcomers and include a brief onboarding instructions?)

Suggested Audience: scientists, data management team, students, stack club members
Category: Science, Data Management
To Be Submitted By(name)


List of SST tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN.  Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Gregory Dubois-Felsmann check if SDM standardization is adequately represented in project documents, and whether DMTN-067 should be required.
31 Mar 2022Gregory Dubois-Felsmann2022-02-14 DM-SST Virtual F2F Agenda and Meeting notes
28 Feb 2023Leanne Guy2023-01-23 DM-SST Agenda and Meeting Notes
  • Leanne Guy talk to Steve R about presenting plans for the ShearObject table to PST and SciCollab chairs   
20 Mar 2023Leanne Guy2023-02-27 DM-SST Agenda and Meeting Notes
31 Mar 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
  • Leanne Guy  talk to Gregory Dubois-Felsmann to review the original intent of the AFS-related Portal requirements before deciding on a course of action  
29 May 2023Leanne Guy2023-05-01 DM-SST Focus Meeting - Brokers in Commissioning
  • Leanne Guy Prepare to consult the PST on the question of providing compressed PVIs for AP outputs, to cover the period before the data become available in a DR.  
02 Jun 2023Leanne Guy2023-03-27 DM-SST Agenda and Meeting Notes
  • Jim Bosch Incorporate 30-60 day period for raws on disk into the strawman proposal and present to KT  
26 Jun 2023Jim Bosch2023-05-08 DM-SST Agenda and Meeting Notes
  • Parker Fagrelius Patrick Ingraham  how long will it take to do a scan as described? No need to scan the whole WL range but will require additional points outside nominal lambda range.  
30 Jun 2023Parker Fagrelius2023-03-27 DM-SST Agenda and Meeting Notes
31 Jul 2023Colin Slater2023-07-10 DM-SST Agenda and Meeting Notes
  • Eli Rykoff , Leanne Guy  Develop a proposal for what calibration processing, hardware, data we actually need and what will be needed for DR1. This has implications for the ORR and for prioritisation of work in commissioning  
31 Jul 2023Eli Rykoff2023-01-30 DM-SST Agenda and Meeting Notes
  • Yusra AlSayyad will look to see if there is any effort to help on option 1  
28 Aug 2023Yusra AlSayyad2023-08-14 DM-SST Agenda and Meeting Notes
  • Jim Bosch  Provide a physical example of that a  up on cell table would look like fo the Colin Slater / DAX team to review  
31 Aug 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
30 Nov 2023Leanne Guy2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  •  "What is the pathway to defining the data products that are required to meet DMS-REQ-0266" Jeffrey Carlin   
30 Nov 2023Jeffrey Carlin2023-10-23 DM-SST vF2F Agenda and Meeting Notes
30 Nov 2023Gregory Dubois-Felsmann2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  • Jeffrey Carlin follow up with KT on DMS-REQ-0176 and DMS-REQ-0315 to update/disaggregate this for latest base/summit infrastructure split.  
30 Nov 2023Jeffrey Carlin2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  • Jim Bosch Follow up on the possibility of investigating further the ability to process 2 collections in parallel.   
31 Jan 2024Jim Bosch2023-12-04 DM-SST Agenda and Meeting Notes
31 Jan 2024Jeffrey Carlin2023-12-04 DM-SST Agenda and Meeting Notes
Gregory Dubois-Felsmann2023-10-23 DM-SST vF2F Agenda and Meeting Notes


Overdue or Undated DM Science Team tickets

Key Summary T Created Updated Due Assignee Reporter P Status Resolution
Loading...
Refresh

LIT tickets of interest to DM Science

T Key Summary Assignee Reporter P Status Resolution Created Updated Due
Loading...
Refresh