(back to the list of all DM-SST meeting minutes)

This DM-SST vF2F  meeting precedes the DMLT meeting and replaces the DM-SST meeting on 2023-10-16

Time: 

9:00 - 13:00 PT

Location

Browser

Room System

Phone Dial-in

Short : ls.st/dmsst 

Zoom: https://noirlab-edu.zoom.us/j/97839121776?pwd=K1JPeUpSMXFvSXJSa2xORGkyVk5zdz09

Zoom Meeting ID: 978 3912 1776 
Zoom Password: 512314

Dial closest IP: 162.255.36.11 (east coast) and 162.255.37.11 (west coast) then use the Zoom meeting ID 978 3912 1776  as the dialing extension.

For example: 978 3912 1776 @162.255.37.11 or: 162.255.37.11##978 3912 1776  Password: 512314

Dial-in numbers:

+1 346 248 7799 (US Toll)

+1 669 900 6833 (US Toll) Meeting ID: 936 2540 1560 International numbers available: https://gemini.zoom.us/u/adcUNrbXzS

Attendees

Regrets


Agenda


Time (Pacific)TopicCoordinatorPre-meeting notesRunning notes and actions 

Moderator: Leanne Guy 

Note taker: Colin Slater 

09:00 - 9:15

Welcome, project science updates

Slides

Leanne Guy 

Updates to key Project milestones

  • ComCam on TMA: June 2024
  • ComCam on-sky: July 2024
  • LSSTCam ready for  on-sky (First Photon with LSSTCam) : mid December  2024 
  • LSSTCam on-sky (System First Light): end January 2025 
  • SV Surveys complete: mid-late May 2025
  • ORR complete: end May 2025
  • Re: comcam, AOS testing requires intra/extrafocal images that reduces the time available for collecting in-focus images.
  • System first light: milestone being used as a timepoint to key other subsequent events off of, e.g. SVV surveys, survey operations start.
  • What time could we imagine a DP1 if it were to be based on comcam? Comcam nominally finished in September 2024, DP1 "Late 2024".
    • Contingent on whether there's scientifically-useful data? Still in discussion.
    • Want to have a discussion with everyone involved in the data preview publishing process before making a commitment (February timescale).
    • Principle issue for RSP is being ready for the scaling step-up. "No field will be useless, and no (single) field will be maximally useful to everyone"
9:15 – 9:30

Overall status of verification

P1

Jeffrey Carlin 

Where are we now?

  • Slide 7, pipelines
    • More useful to have a JIRA ticket for implementing the functionality in the list
    • Proposal to drop 0034 to a P1b, out of sequence to require an Object catalog in commissioning prior to having a DR/DP
    • 0266: Questions around how data will flow from the pipelines into the CDB
    • 0266: What pathway will satisfy this requirement, many exposure catalogs in different places → does this refer to the DRP produced catalogs, parquet, CAOM, etc ...0297:  
  •  "What is the pathway to defining the data products that are required to meet DMS-REQ-0266" Jeffrey Carlin   
  • Slide 8, reporting
    • Eric has item on the DMLT agenda about reporting.
    • SITCOM has related interests.
  • Slide 9, data facility
    • DMS-REQ-0298: need to enumerate the mapping from the requirement to the services we have. Provenance is a known weakness.
      • Might verify the majority of this requirement and leaving some small piece for later.
      • Or split requirement into a 1a aspect and 1b aspect.
      • Calexp, is it easy to determine the dark/flat/bias, yes.
  • Slide 10
    • Jeffrey Carlin follow up with KT on DMS-REQ-0176 and DMS-REQ-0315 to update/disaggregate this for latest base/summit infrastructure split.  
  • Slide 11
    • Disaggregate 1a and non-1a priority aspects.
  • Slide 12
    •  DMS-REQ-0003
    • GPDF: Some of these requirements were meant to be in a hierarchy, so that some high-level requirements are "rollups" of lower level requirements. Ideally these could have been claimed as "verified by children", but hierarchy hasn't been maintained for the past few years.
    • GPDF: "Science Data Archive" means science-user-facing.
    • DMS-REQ-0398, Why is the user-facing EFD a 1a? Probably was interpreted as internal-facing at the time.
      • Leanne Guy drop DMS-REQ-0398 (public facing EFD data) to priority 1b.
  • DMS-REQ-0066,
    • Does this require user batch? IF so, may end up having to defer this to operations.
    • "Virtual data products"
    • Split this into 1a: archive all of the data products, and the path to recreating products is feasible (does not mean the ideal provenance system, but reasonable-for-commissioning level of functionality)
    • Jim: all of this discussion is referring to different levels of ease-of-recreation. Several different levels of this. Some can be priority 2.
    • Priority 1 requirement proposal: "The DMS service shall record information to support re-creation"
    • Priority 2 requirement proposal: "The DMS service shall provide services for re-creation"
  • DMS-REQ-0339: make this a priority 2 requirement. No reason for it to be 1a.
  • Snaps: Can these be 1a's or not?
    • Eric: Need these to be 1a's so we're ready to go for commissioning. Auxtel is taking data in snaps-mode already.
    • Will schedule an SST meeting about criteria for snaps vs single visits. Not SST's final call, but we want to have DM's input ready to go.
  • DMS-REQ-0362: ellipticity residuals.

Jeffrey Carlin disaggregate P1a components of all requirements  from other components for all requirements discussed. Create an RFC and push through an LCR  

9:30 -- 10:30Path to P1a completion  to support commissioning then (if time) P1b to ORR

Jeffrey Carlin 

slides

This session will focus on the path to completion for LDM-503-19a (All P1a DM requirements verified)

The related SITCOM ticket is SITCOM-114

10:30 - 11:00Break

Moderator: Leanne Guy 

Note Taker: Colin Slater 
11:00 – 11:45RSP functionality + prioritization 

Gregory Dubois-Felsmann 

Slides: (PDF)
  • Data Delivery
    • DMLT agenda item on Thursday for AP to RSP.
    • Moving window of compressed AP image storage is a storage upscope, doesn't require new software.
  • Databases
  • Data Services
11:45 --12:00Definition of dataset selection requirements: DMS-REQ-293 (P1a) – what degree of dataset-selectability is required to claim this requirement?

Colin Slater 
Robert Lupton Gregory Dubois-Felsmann 

Specification: A Dataset may consist of one or more pixel images, a set of records in a file or database, or any other grouping of data that are processed or produced as a logical unit. The DMS shall be able to identify and retrieve complete, consistent datasets for processing.

Discussion: Logical groupings might be pairs of Exposures in a Visit, along with supporting metadata and provenance information, or might be groupings defined in the context of Level3 processing

From 2023-07-10 DM-SST Agenda and Meeting Notes

What is sufficient to claim this requirement?

Deferred to a future meeting - need Robert Lupton 

12:00-- 12:30 Closeout + (remaining) task review + future

Leanne Guy 

Next meeting is next week: Alex Drlica-Wagner will present his idea for a TGAS-like DECam+LSST proper motion sample. 

DM Science team meetings in FY24 given that most of us will be > 50% Ops.  Transition to Operations. The Ops PO/Science team. 

Summary 

  • A lot of proposals for de-prioritization of P1a requirements, or de-aggregating P1a requirements that contain elements that are complete and elements that are P1b/2/3 Jeffrey Carlin will put together a summary to go to RFC/LCR
  • Interaction between science platform and middleware regarding user storage for butler purposes - Jim Bosch Gregory Dubois-Felsmann
  • Large amount of remaining RSP work needs a science-driven  prioritization  - Gregory Dubois-Felsmann and Leanne Guy . Will be presented at a future SST meeting
  • A number of topics identified for future follow up  - these will be scheduled
12:30Close 
13:00 – 14:00 

Ops Rehearsal for Commissioning Meeting


List of SST tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN.  Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Gregory Dubois-Felsmann check if SDM standardization is adequately represented in project documents, and whether DMTN-067 should be required.
31 Mar 2022Gregory Dubois-Felsmann2022-02-14 DM-SST Virtual F2F Agenda and Meeting notes
  • Steve Ritz Work with DM and Camera to develop plans for delivering a shutter position-vs-time model  
16 May 2022Steve Ritz2022-03-14 DM-SST Agenda and Meeting Notes
  • Gregory Dubois-Felsmann  / Leanne Guy Create a TN that outlines the science use cases and options for compressed PVIs from AP. Present the issue at DMLT VF2F and then the PST.   
13 Jun 2022Gregory Dubois-Felsmann2022-05-09 DM-SST Agenda and Meeting Notes
  • Eli Rykoff Report on what information DES used for their decision on using compression?  
25 Jul 2022Eli Rykoff2022-06-06 DM-SST VF2F Agenda and Meeting notes
  • Melissa Graham to draft a proposal for including pz in alert packets  
12 Sep 2022Melissa Graham2022-08-15 DM-SST Agenda and Meeting Notes
26 Sep 2022Steve Ritz2022-09-12 DM-SST Agenda and Meeting Notes
  • Gregory Dubois-Felsmann  start a technote to write down the details for science end user interface to SED corrections in catalogs  
31 Oct 2022Gregory Dubois-Felsmann2022-08-22 DM-SST Agenda and Meeting Notes
31 Dec 2022Leanne Guy2022-06-06 DM-SST VF2F Agenda and Meeting notes
28 Feb 2023Leanne Guy2023-01-23 DM-SST Agenda and Meeting Notes
  • Leanne Guy talk to Steve R about presenting plans for the ShearObject table to PST and SciCollab chairs   
20 Mar 2023Leanne Guy2023-02-27 DM-SST Agenda and Meeting Notes
31 Mar 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
  • Leanne Guy  talk to Gregory Dubois-Felsmann to review the original intent of the AFS-related Portal requirements before deciding on a course of action  
29 May 2023Leanne Guy2023-05-01 DM-SST Focus Meeting - Brokers in Commissioning
  • Leanne Guy Prepare to consult the PST on the question of providing compressed PVIs for AP outputs, to cover the period before the data become available in a DR.  
02 Jun 2023Leanne Guy2023-03-27 DM-SST Agenda and Meeting Notes
  • Jim Bosch Incorporate 30-60 day period for raws on disk into the strawman proposal and present to KT  
26 Jun 2023Jim Bosch2023-05-08 DM-SST Agenda and Meeting Notes
  • Parker Fagrelius Patrick Ingraham  how long will it take to do a scan as described? No need to scan the whole WL range but will require additional points outside nominal lambda range.  
30 Jun 2023Parker Fagrelius2023-03-27 DM-SST Agenda and Meeting Notes
31 Jul 2023Colin Slater2023-07-10 DM-SST Agenda and Meeting Notes
  • Eli Rykoff , Leanne Guy  Develop a proposal for what calibration processing, hardware, data we actually need and what will be needed for DR1. This has implications for the ORR and for prioritisation of work in commissioning  
31 Jul 2023Eli Rykoff2023-01-30 DM-SST Agenda and Meeting Notes
  • Yusra AlSayyad will look to see if there is any effort to help on option 1  
28 Aug 2023Yusra AlSayyad2023-08-14 DM-SST Agenda and Meeting Notes
  • Jim Bosch  Provide a physical example of that a  up on cell table would look like fo the Colin Slater / DAX team to review  
31 Aug 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes