2023-12-04 DM-SST Agenda and Meeting Notes

Short : ls.st/dmsst 

Zoom: https://noirlab-edu.zoom.us/j/97839121776?pwd=K1JPeUpSMXFvSXJSa2xORGkyVk5zdz09

Zoom Meeting ID: 978 3912 1776 
Zoom Password: 512314

Dial closest IP: 162.255.36.11 (east coast) and 162.255.37.11 (west coast) then use the Zoom meeting ID 978 3912 1776  as the dialing extension.

For example: 978 3912 1776 @162.255.37.11 or: 162.255.37.11##978 3912 1776  Password: 512314

Dial-in numbers:

+1 346 248 7799 (US Toll)

+1 669 900 6833 (US Toll) Meeting ID: 936 2540 1560 International numbers available: https://gemini.zoom.us/u/adcUNrbXzS

No PST meeting last week due to AMCR

Slides

Colin and Robert on summit this week and will try to come up with a proposal

Proposal from Colin: 

• argues that 'dataset' is a Butler dataset only to define a scope that we can deliver - "so one or more butler datasets"
• mechanisms 
  • tagged collections based on Butler dimensions
    • this is implemented
  • open-ended - pass a list of data IDS to the Butler generated somehow (e.g EFD ...) 
    • this is possible but a bit hacky and the Butler UI could be improved, but it is possible. 
    • E.g. a CDB query that can be turned into a tagged collection would satisfy this. 
    • File with a list of dataIDs would be nice to have
• Consistent has a definition in the upper OSS -  - e.g. arising from a coherent set of inputs 
  • Chaining together of collections in the right way is the mechanism for superseding.  
  • Tagged collection is independent of a chain and is only a list of data IDs - does it guarantee that there are no duplicates?
  • See CM  and the reprocessing of dome IDs
• Butler requirement to process multiple input datasets in parallel is DMS-MWBT-REQ-0082 satisfies this in a notebook with analysis code, but not in a pipetask, The requirement is about querying, not process. 
• There is a feature request in DMTN-220 for dataId set upload that is the same as this request: https://dmtn-220.lsst.io/#middleware-feature-requests (3.1). DMTN-220 is the source for all these types of Butler requests. 

• Jim Bosch Follow up on the possibility of investigating further the ability to process 2 collections in parallel.  31 Jan 2024 
• Jeffrey Carlin Gregory Dubois-Felsmann Colin Slater Define verifications for the consistency of collections part of this requirements - e.g. OSS-118 31 Jan 2024 
• Gregory Dubois-Felsmann Create ticket for new Butler functionality to take a (simple text)  file of data IDs and pass them to the Butler and retrieve the associated data. Follow up with Jim Bosch  08 Dec 2023  : DM-42298

Status of RFC-970 proposed changes - have all the proposals to re-prioritize and/or de-aggregate requirements discussed at the vF2F meeting been addressed? 

RFC has been approved. LCR-3643 has been submitted.

After consultation with K-T, RFC-977 has also been submitted with some minor changes to "facilities"-related requirements.

Closeout and review of Test Report: DMTR-401 / Test Plan: LVV-P106

slides with updates since vF2F