Logistics

Date , 11:00 PT

Teleconference coordinates: https://bluejeans.com/50789407

Attendees

Regrets

Goals

Weekly meeting of the DM-SST to discuss scientific aspects of the Data Management System

Discussion items


ItemWhoNotesConclusions and action items
Updates from project, science collaborations, and other meetings.

Jim Bosch, Robert Lupton, Colin Slater

  • Jim & Robert L attended the Aspen meeting. Jim explained DM plans for shear estimation and coadds, and issues around edges to the community. There was general understanding and agreement
  • Colin attended Stellar Halos 2018,
    • Lots of users of HSC data for both resolved stellar populations and diffuse light measurements, all using the LSST pipeline. This often involves custom post-processing of images, sometimes to make up for cases our code doesn't handle well (e.g. points sources on top of diffuse light). Will need to keep an eye on how this usage evolves to see what we need to fix/augment in the stack.
  • Leanne attended the Solar System SC readiness sprint at UW. 3 important topics:
    • The idea of computing the orbits of solar system objects at the MPC and not in the LSST daily MOPs pipeline is under discussion. There are pros and cons; this puts the MPC on the LSST critical path and represents a risk. Currently the MPC are not resourced for the required uptime, although they claim they are scaling up to the required need. On the other hand if we do not do this, we risk having bi-furcated orbits. Discussion is ongoing
    • Review of the SSObject table, DM-12092: An productive afternoon was spent by several people reviewing this ticket, Mario will update the ticket when complete
    • Request for an LSST DM presence at DPS: It is difficult and costly to host SSSC meetings. As all members of this SC attend the annual DPS meeting, a request was made to have an LSST presence including a presentation on the status of LSST, MOPS and possibly a tutorial session. Jim pointed out that if we commit to this we must prioritise Mario's time.
  • Leanne Guy follow up with project about LSST presence at DPS Outcome: LSST will have a presence at DPS (again) from 2019. We will possibly start running LSP tutorials. Ranpal is coordinating
Update on LDM-639 and next steps for JSR

Michael Wood-Vasey

  • RFC-495 
  • Test Items/Objectives need to be updated to provide concise one line sentences of the goal of the test item
  • Definition of a common vocabulary is needed and should be added to the document, e,g verify that, show that and added to the document
  • Pending updates from Gregory Dubois-Felsmann should be added after the JSR

Next-to-DB Use CasesColin Slater
  • DMTN-086
  • Background: The DB group need use cases for the next to database processing system they are building, incl. what data from the DB is most relevant for this type of processing.
  • Robert Lupton: The document contains science use case, there is a gap in scope; data served to the jupyter notebook aspect, ORM layers. Can next-to-db-processing replace aspects of the pipelines? Should be able to use the pipeline tools to understand the data we process.
  • Jim Bosch raises a longer term issue; parts of the pipeline look like next-to-db processing. e.g can we do photometric calibration in the DB. Can we commission parts of the pipelines as NTDBP
  • Colin Slater: Goal is to design the system for users first, envisages a different document to cover these aspects.
  • The next to DB prototyping work is described in Epic: S18 Qserv Next-to-data Processing Prototype
  • Colin Slater Forward technote to Wil
  • Colin Slater Organise an informal presentation to the SST on the current state of the DAX prototyping activities for a next-2-db processing system
Interaction with DESC (and subsequently all SCs)

Leanne Guy


At the DESC CMU meeting (2018-07-23 - 2018-07-27),  Katrin, Phil & Leanne Guy will get together to discuss high level coordination plans regarding DC2.  I would like to go prepared with suggestions as to how DM can work with DESC, especially during DC2 to our mutual benefit? E.g:

  • How can we leverage DESC resources to implement the DPDD?
  • Identify resources who will work on DESC and ensure the benefits come back to us
    • Put effort into educating DESC on LSST tools.
  • Using DESC DC2 Qserv instance (~200TB) to advance DAX milestones
  • Handling of image metadata
  • SuperTask & G3 Butler; what is the timeline?
  • DESC members fixing documentation/bugs on the DM stack and issuing pull requests which are subsequently reviewed by DM.
  • ... other suggestions

Should we try to formalize our interactions with DESC?

  • Semi-regular information sessions?



Camera team request to use DM pipelines and LSP for Camera I&T

Leanne Guy

  • Steve Ritz is interested in using the LSP and DM science pipelines for Camera AI&T.

Deferred to next meeting

Open action itemsLeanne Guy
  • See list below


Overdue or dateless JIRA tickets

Leanne Guy

  • Review status and assign a new due date and plan of work


List of SST tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Leanne Guy to talk to Science Pipelines (Yusra) about when do this transfer  
19 Oct 2021Leanne Guy2021-09-28 Science Metric Development Agenda and Meeting notes
  • Leanne Guy arrange to disucss at a future meeting if there are metrics from PDR3 & this paper that we might want to include in faro.   
26 Oct 2021Leanne Guy2021-08-31 Science Metric Development Agenda and Meeting notes
  • Colin to ask about capturing ideas for improvement to the stellar locus algorithm   
30 Nov 2021 2021-11-09 Science Metric Development Agenda and Meeting notes
  • Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN.  Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Gregory Dubois-Felsmann check if SDM standardization is adequately represented in project documents, and whether DMTN-067 should be required.
31 Mar 2022Gregory Dubois-Felsmann2022-02-14 DM-SST Virtual F2F Agenda and Meeting notes
  • Steve Ritz Work with DM and Camera to develop plans for delivering a shutter position-vs-time model  
16 May 2022Steve Ritz2022-03-14 DM-SST Agenda and Meeting Notes
  • Gregory Dubois-Felsmann  / Leanne Guy Create a TN that outlines the science use cases and options for compressed PVIs from AP. Present the issue at DMLT VF2F and then the PST.   
13 Jun 2022Gregory Dubois-Felsmann2022-05-09 DM-SST Agenda and Meeting Notes
  • Eli Rykoff Report on what information DES used for their decision on using compression?  
25 Jul 2022Eli Rykoff2022-06-06 DM-SST VF2F Agenda and Meeting notes
  • Melissa Graham to draft a proposal for including pz in alert packets  
12 Sep 2022Melissa Graham2022-08-15 DM-SST Agenda and Meeting Notes
26 Sep 2022Steve Ritz2022-09-12 DM-SST Agenda and Meeting Notes
  • Gregory Dubois-Felsmann  start a technote to write down the details for science end user interface to SED corrections in catalogs  
31 Oct 2022Gregory Dubois-Felsmann2022-08-22 DM-SST Agenda and Meeting Notes
31 Dec 2022Leanne Guy2022-06-06 DM-SST VF2F Agenda and Meeting notes
28 Feb 2023Leanne Guy2023-01-23 DM-SST Agenda and Meeting Notes
  • Leanne Guy talk to Steve R about presenting plans for the ShearObject table to PST and SciCollab chairs   
20 Mar 2023Leanne Guy2023-02-27 DM-SST Agenda and Meeting Notes
31 Mar 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
  • Leanne Guy  talk to Gregory Dubois-Felsmann to review the original intent of the AFS-related Portal requirements before deciding on a course of action  
29 May 2023Leanne Guy2023-05-01 DM-SST Focus Meeting - Brokers in Commissioning
  • Leanne Guy Prepare to consult the PST on the question of providing compressed PVIs for AP outputs, to cover the period before the data become available in a DR.  
02 Jun 2023Leanne Guy2023-03-27 DM-SST Agenda and Meeting Notes
  • Jim Bosch Incorporate 30-60 day period for raws on disk into the strawman proposal and present to KT  
26 Jun 2023Jim Bosch2023-05-08 DM-SST Agenda and Meeting Notes
  • Parker Fagrelius Patrick Ingraham  how long will it take to do a scan as described? No need to scan the whole WL range but will require additional points outside nominal lambda range.  
30 Jun 2023Parker Fagrelius2023-03-27 DM-SST Agenda and Meeting Notes
31 Jul 2023Colin Slater2023-07-10 DM-SST Agenda and Meeting Notes

Open DM SST RFCs 

T Key Summary Assignee Reporter P Status Resolution Created Updated Due
Refresh


Open DM SST LITs 

T Key Summary Assignee Reporter P Status Resolution Created Updated Due
Refresh


Overdue or undated DM SST JIRA Tickets

(JIRA DM tasks with label "dm-sst")

T Key Summary Assignee Reporter P Status Resolution Created Updated Due
Refresh