(back to the list of all DM-SST meeting minutes)

Time: 

 11:00 PT

Location

Browser

Room System

Phone Dial-in

Short : ls.st/dmsst 

Zoom: https://noirlab-edu.zoom.us/j/97839121776?pwd=K1JPeUpSMXFvSXJSa2xORGkyVk5zdz09

Zoom Meeting ID: 978 3912 1776 
Zoom Password: 512314

Dial closest IP: 162.255.36.11 (east coast) and 162.255.37.11 (west coast) then use the Zoom meeting ID 978 3912 1776  as the dialing extension.

For example: 978 3912 1776 @162.255.37.11 or: 162.255.37.11##978 3912 1776  Password: 512314

Dial-in numbers:

+1 346 248 7799 (US Toll)

+1 669 900 6833 (US Toll) Meeting ID: 936 2540 1560 International numbers available: https://gemini.zoom.us/u/adcUNrbXzS

Attendees

Regrets

Discussion items

 ​

ItemWhoPre-Meeting NotesNotes and  Action Items
Project/Science Updates

Leanne Guy

  • PST meeting 2022-11-30: 
    • EPO closeout:
      • Completed successfully last week. EPO is now 100% Operations, this will allow them to start doing EPO, which was not allowed under Construction.
      • There are some remaining EPO-DM interface requirements to verify, these will be completed when DM is ready. 
    • Schedule 
      • No negative feedback received from the JOG nor science collaborations.
      • Final version of the revised Construction schedule was expected for last Friday. 
      • Details of the Data Previews are still being worked on - there will be changes. Planning to make a statement at the AAS meeting in January
  • TMA is moving! "Soak tests" where the TMA is run for 8-10 hrs to demonstrate that it slews and tracks correctly are ongoing. Many issues uncovered .. 
  • SCOC still on track to deliver a report in December. 
    • DESC have expressed concerns to the SCOC about the no-uniformity of coadds and the impact of rolling cadence. Leanne, Jim, Robert will meet with a small number of DESC people on Friday to understand their concerns
  • Schedule finalization has been delayed until the end of this week
  • JC: Galaxies SC also expressed some concern about non-uniformity of coadds. Worth pulling them into the discussion, too?
  • LPG: Yes, we will be talking to all science collaborations 
  • SR: Working on a better definition of system first light.
Catchup/retry processing and database structure

Eric Bellm 

We have validity ranges on DIAObjects but not on DIASources--Discussion on DM-36792 - Getting issue details... STATUS . Do we need to have the capacity to mark any record in the APDB/PPDB as invalid

slides

  • JB: Similar "linear range"  things in the Butler for calibration validity - we use a PostgreSQL call for that, defining a unique constraint. Need a DB that is indexing =. 
  • CTS: Don't understand the replay issue - how is processing in the morning any different?
  • EB: Second bullet covers some breakage - Some DIASource records in the APDB but not all, or some kind of inconsistent state where it would be best to just wipe it and start over than try to patch the records. 
  • EB: If processing just fails then there will be no records in the APDB
  • GPDF: technical question - we are not overriding the scientific semantics of the record but rather updating the DB record 
  • KT: Consider the alternatives 1) have a version number rather than a range - adds query complexity but removes the update mentioned by Gregory 2) when we need to replace a record - just replace it. this has obvious consistency and reproducibility issues.
  • CTS: DIAObect use case:  what did the DB look like at a specific time - is a "this record is bad" flag column sufficient? 
  • EB: We have a requirement to be able to reconstruct the state at any given time. If we send out bad alerts and a user comes back looking for supplemental information,  is it sufficient to just mark them as bad? 
  • CTS: If a user queries a DIAObject they will get a reference to a lot of DIASources  - those linkages would still be preserved. But if they 
  • GPDF: 2 different use cases here: the use case that led to the design on slide 1 is the normal course of ops where DIAObjects will evolve and we want to reconstruct any state in the past. The second use case is about fixing mistakes; this is new and valid. Are we overloading the same mechanism to try to deal with both in the same way? Playback goes back through all data, good or bad.  Is something superseded because we got more data or because it was bad. Should we rethink the data mode to  make sure users never see bad data?
  • GPDF: Alerts once distributed cannot be recalled and will have IDs. Bad or not, we want to associate an ID with data in our system. Alert DB is underspecified in terms pf the semantic capabilities . Can we flag an alert in the Alert DB with a withdrawn flag?
  • KT: Object stores have metadata - could set a flag in that MD 
  • GPDF: Need a rethink and a more concrete proposal
  • GPDF: Is it Ok to do a delete for bad data?
  • ER: Provenance makes sense - get an alert, need to see what it was .. leads to a red box, alert has been withdrawn
  • ER: need some sort of versioning or time indicator - not just a flag. Cannot replay the DB at a given time or know when it went bad
  • ER: Expect to update calibrations frequently - fine grained time stamps
  • EB: Think about how this interacts through the night - bad DIASources will lead to bad DIAObjects → how do we unfold. 
  • JB: DIAObject needs both the new thing and the thing before - separate field
  • Eric Bellm Will go away and consolidate into a new proposal   
Calibration Strategy Paper overview

Eli Rykoff 

PSTN-026


LCR to remove schemas from DPDD and project-level change controlEric Bellm 

Making even minor naming changes to the baseline schema is made difficult by the requirement for project-level LCR.  Propose to make sdm_schemas the source of truth and have the DPDD direct users there.


(included in slides above)

  • Requires a LCR to make changes 
  • CTS: RFC-807 changed a bunch of names in the baseline schema - DM-CCB and no LCR. Semantic changes require an LCR
  • EB: Schema browser baseline schema is not actually updated  
  • CTS: Sorry
  • GPDF: Are you proposing to remove the tables from the DPDD. What would we lose if we change this?
    • Original proposal was that the the relationship between the DPDD and concrete DM be visible 
    • Internal consistency of all the different data models - still missing a lot of automated tooling - still very manual. need more time invested in felis, etc  tooling
  • YAS: DM-22078 - epic to upstream changes to be reflected in the baseline. 
  • EB: Concerned that we have a strong divergence between the DPDD and current plan
  • ER: InSim is under CI  - keep everything under regular testing - guarantee that what imsim.yaml defines is what the pipelines produce
  • https://github.com/lsst/pipe_tasks/blob/main/schemas/Object.yaml
  • JB: logical vs physical schemas - important 
  • Generate LDM-153 from imsim.yAml TO REFLECT current state. 
AOB

Leanne Guy

Next meeting is:  



List of SST tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN.  Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Gregory Dubois-Felsmann check if SDM standardization is adequately represented in project documents, and whether DMTN-067 should be required.
31 Mar 2022Gregory Dubois-Felsmann2022-02-14 DM-SST Virtual F2F Agenda and Meeting notes
  • Eli Rykoff Report on what information DES used for their decision on using compression?  
25 Jul 2022Eli Rykoff2022-06-06 DM-SST VF2F Agenda and Meeting notes
26 Sep 2022Steve Ritz2022-09-12 DM-SST Agenda and Meeting Notes
31 Dec 2022Leanne Guy2022-06-06 DM-SST VF2F Agenda and Meeting notes
28 Feb 2023Leanne Guy2023-01-23 DM-SST Agenda and Meeting Notes
  • Leanne Guy talk to Steve R about presenting plans for the ShearObject table to PST and SciCollab chairs   
20 Mar 2023Leanne Guy2023-02-27 DM-SST Agenda and Meeting Notes
31 Mar 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
  • Leanne Guy  talk to Gregory Dubois-Felsmann to review the original intent of the AFS-related Portal requirements before deciding on a course of action  
29 May 2023Leanne Guy2023-05-01 DM-SST Focus Meeting - Brokers in Commissioning
  • Leanne Guy Prepare to consult the PST on the question of providing compressed PVIs for AP outputs, to cover the period before the data become available in a DR.  
02 Jun 2023Leanne Guy2023-03-27 DM-SST Agenda and Meeting Notes
  • Jim Bosch Incorporate 30-60 day period for raws on disk into the strawman proposal and present to KT  
26 Jun 2023Jim Bosch2023-05-08 DM-SST Agenda and Meeting Notes
  • Parker Fagrelius Patrick Ingraham  how long will it take to do a scan as described? No need to scan the whole WL range but will require additional points outside nominal lambda range.  
30 Jun 2023Parker Fagrelius2023-03-27 DM-SST Agenda and Meeting Notes
31 Jul 2023Colin Slater2023-07-10 DM-SST Agenda and Meeting Notes
  • Eli Rykoff , Leanne Guy  Develop a proposal for what calibration processing, hardware, data we actually need and what will be needed for DR1. This has implications for the ORR and for prioritisation of work in commissioning  
31 Jul 2023Eli Rykoff2023-01-30 DM-SST Agenda and Meeting Notes
  • Yusra AlSayyad will look to see if there is any effort to help on option 1  
28 Aug 2023Yusra AlSayyad2023-08-14 DM-SST Agenda and Meeting Notes
  • Jim Bosch  Provide a physical example of that a  up on cell table would look like fo the Colin Slater / DAX team to review  
31 Aug 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
30 Nov 2023Gregory Dubois-Felsmann2023-10-23 DM-SST vF2F Agenda and Meeting Notes
30 Nov 2023Leanne Guy2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  •  "What is the pathway to defining the data products that are required to meet DMS-REQ-0266" Jeffrey Carlin   
30 Nov 2023Jeffrey Carlin2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  • Jeffrey Carlin follow up with KT on DMS-REQ-0176 and DMS-REQ-0315 to update/disaggregate this for latest base/summit infrastructure split.  
30 Nov 2023Jeffrey Carlin2023-10-23 DM-SST vF2F Agenda and Meeting Notes