Versions Compared

Old Version 19

changes.mady.by.user Leanne Guy

Saved on

compared with

New Version Current

changes.mady.by.user Eric Bellm

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

ItemWhoPre-Meeting NotesNotes and  Action Items
Project/Science Updates

Leanne Guy

  • PST meeting 2022-11-30: 
    • EPO closeout:
      • Completed successfully last week. EPO is now 100% Operations, this will allow them to start doing EPO, which was not allowed under Construction.
      • There are some remaining EPO-DM interface requirements to verify, these will be completed when DM is ready. 
    • Schedule 
      • No negative feedback received from the JOG nor science collaborations.
      • Final version of the revised Construction schedule was expected for last Friday. 
      • Details of the Data Previews are still being worked on - there will be changes. Planning to make a statement at the AAS meeting in January
  • TMA is moving! "Soak tests" where the TMA is run for 8-10 hrs to demonstrate that it slews and tracks correctly are ongoing. Many issues uncovered .. 
  • SCOC still on track to deliver a report in December. 
    • DESC have expressed concerns to the SCOC about the no-uniformity of coadds and the impact of rolling cadence. Leanne, Jim, Robert will meet with a small number of DESC people on Friday to understand their concerns
  • Schedule finalization has been delayed until the end of this week
  • JC: Galaxies SC also expressed some concern about non-uniformity of coadds. Worth pulling them into the discussion, too?
  • LPG: Yes, we will be talking to all science collaborations 
  • SR: Working on a better definition of system first light.
Catchup/retry processing and database structure

Eric Bellm 

We have validity ranges on DIAObjects but not on DIASources--Discussion on

Jira
serverJIRA
serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
keyDM-36792
. Do we need to have the capacity to mark any record in the APDB/PPDB as invalid

slides

  • JB: Similar "linear range"  things in the Butler for calibration validity - we use a PostgreSQL call for that, defining a unique constraint. Need a DB that is indexing =. 
  • CTS: Don't understand the replay issue - how is processing in the morning any different?
  • EB: Second bullet covers some breakage - Some DIASource records in the APDB but not all, or some kind of inconsistent state where it would be best to just wipe it and start over than try to patch the records. 
  • EB: If processing just fails then there will be no records in the APDB
  • GPDF: technical question - we are not overriding the scientific semantics of the record but rather updating the DB record 
  • KT: Consider the alternatives 1) have a version number rather than a range - adds query complexity but removes the update mentioned by Gregory 2) when we need to replace a record - just replace it. this has obvious consistency and reproducibility issues.
  • CTS: DIAObect use case:  what did the DB look like at a specific time - is a "this record is bad" flag column sufficient? 
  • EB: We have a requirement to be able to reconstruct the state at any given time. If we send out bad alerts and a user comes back looking for supplemental information,  is it sufficient to just mark them as bad? 
  • CTS: If a user queries a DIAObject they will get a reference to a lot of DIASources  - those linkages would still be preserved. But if they 
  • GPDF: 2 different use cases here: the use case that led to the design on slide 1 is the normal course of ops where DIAObjects will evolve and we want to reconstruct any state in the past. The second use case is about fixing mistakes; this is new and valid. Are we overloading the same mechanism to try to deal with both in the same way? Playback goes back through all data, good or bad.  Is something superseded because we got more data or because it was bad. Should we rethink the data mode to  make sure users never see bad data?
  • GPDF: Alerts once distributed cannot be recalled and will have IDs. Bad or not, we want to associate an ID with data in our system. Alert DB is underspecified in terms pf the semantic capabilities . Can we flag an alert in the Alert DB with a withdrawn flag?
  • KT: Object stores have metadata - could set a flag in that MD 
  • GPDF: Need a rethink and a more concrete proposal
  • GPDF: Is it Ok to do a delete for bad data?
  • ER: Provenance makes sense - get an alert, need to see what it was .. leads to a red box, alert has been withdrawn
  • ER: need some sort of versioning or time indicator - not just a flag. Cannot replay the DB at a given time or know when it went bad
  • ER: Expect to update calibrations frequently - fine grained time stamps
  • EB: Think about how this interacts through the night - bad DIASources will lead to bad DIAObjects → how do we unfold. 
  • JB: DIAObject needs both the new thing and the thing before - separate field
  •  Eric Bellm Will go away and consolidate into a new proposal   
Calibration Strategy Paper overview

Eli Rykoff 

PSTN-026


LCR to remove schemas from DPDD and project-level change controlEric Bellm 

Making even minor naming changes to the baseline schema is made difficult by the requirement for project-level LCR.  Propose to make sdm_schemas the source of truth and have the DPDD direct users there.


(included in slides above)

  • Requires a LCR to make changes 
  • CTS: RFC-807 changed a bunch of names in the baseline schema - DM-CCB and no LCR. Semantic changes require an LCR
  • EB: Schema browser baseline schema is not actually updated  
  • CTS: Sorry
  • GPDF: Are you proposing to remove the tables from the DPDD. What would we lose if we change this?
    • Original proposal was that the the relationship between the DPDD and concrete DM be visible 
    • Internal consistency of all the different data models - still missing a lot of automated tooling - still very manual. need more time invested in felis, etc  tooling
  • YAS: DM-22078 - epic to upstream changes to be reflected in the baseline. 
  • EB: Concerned that we have a strong divergence between the DPDD and current plan
  • ER: InSim is under CI  - keep everything under regular testing - guarantee that what imsim.yaml defines is what the pipelines produce
  • https://github.com/lsst/pipe_tasks/blob/main/schemas/Object.yaml
  • JB: logical vs physical schemas - important 
  • Generate LDM-153 from imsim.yAml TO REFLECT current state. 
AOB

Leanne Guy

Next meeting is:  


...