Logistics

Date & Time

Date : , 09:50 - 13:30 PT

This precedes the DMLT vF2F (19,20,21 Oct) 

Location

Cyberspace: https://noirlab-edu.zoom.us/j/94036703741?pwd=N0NaRUNjclQxUCtBQ0tNdytVN0NFUT09


Participants


@Agenda


Time (PST)TopicCoordinatorPre-meeting notesRunning notes and actions 

Moderator: Leanne Guy 

Notetaker: Colin Slater 

09:50 - 9:55WelcomeLeanne Guy 

9:55-10:00Middleware PO role
  • Who is the point of contact for usability issues in middleware? Middleware support channel is the first stage of triage (for project-internal users), then a linear combination of Gregory and Jim. Product owner role is to prioritize tickets, e.g. in order to balance user needs vs needs of pipelines execution, etc.
  • From #dm-sst channel. Agreement that the scope of the middleware PO covers interfaces to the underlying execution/production system, but not the execution/production system itself (e.g PanDA, HTCondor).  
10:00 - 10:30

Requirements on user batch   

Review of final draft of dmtn-202.lsst.io to present at DMLT-vF2F 
  • Does the requirement on "once and only once" processing mean that we supply some software framework, or is it met by providing compute resources plus "cookbook" of recipes that guides people towards the "right way" of accessing the data once-and-only-once.
  • RHL: HSC experience was that users needed to reprocess images at the position of their candidate objects. Users had to write their own code to take those catalog and then activate batch processing of their image processing code. Would like users to not have to write their own slurm submissions to do this type of processing.
  • LPG: What have we promised for DP1? Unlikely that we will have anything like this on that short of a timescale.
  • RHL: we did say that we would provide a solution for this. GPDF: did not say we would have a cradle-to-grave solution for this, we said we would provide the tools to enable users.
  • CTS: need to be more specific about what an MVP solution would look like in order to converge on whether we could build and support that.
  • LPG: What are commissioning requirements on this? Treat commissioning as using project systems, doesn't need to be "user batch".
    • LPG: I am thinking about the AO / community contributions to the commisioning data processing and analysis and not the standard project processing of the data. 
10:30 - 10:40

Special Programs staffing needs




Deferred.

10:40 - 11:00Break

Moderator:

Notetaker: Colin Slater 

11:00 - 12:00Commissioning needs for cross-match tables

Follow on from discussions at  2021-08-30 DM-SST

LIT-552 - Plan for and provide cross-identification tables between LSST objects and external catalogs (1f) MANAGER APPROVED

LPG: Ideally the outcome would be a TN describing the needs. 

Slides with discussion points

  • Eric: Solar systems objects will come "for free" from the SSP pipeline, because the data comes from the Minor Planet Center.
  • Eric: don't expect transient/variable catalogs to be sufficient to verify completeness and purity at the faint end, will need to depend on injected sources or other technical means to verify. Might be feasible at the bright end.
  • Why UV and IR? Keith: UV helpful for calibration of u band, IR for photo-z and S/G sep.
  • How much should we standardize the refcat columns on input vs. apply processing when they're loading? Experience with the Gaia catalogs was that we tried to standardize them to be SDSS-like, but that made it eventually harder to use.
    • Agreement was that standardizing on refcat creation is a bad idea, instead should focus on analysis code that can handle heterogenous inputs.
  • Are catalogs provided via qserv in the RSP in-scope for this discussion? In-scope for the LIT ticket that triggered this, but focusing on commissioning needs right now. Will need a follow-up on the RSP catalogs.
12:00 - 12:30Complete reconciliation of DPDD with pipelinesLeanne Guy Colin Slater Yusra AlSayyad 

Reconciling the DPDD with the required information for pipeline QA.

  • Will we provide an ObjectId in the Source table - need a clear statement on this. 
  • Can we remove detailed schema information from the DPDD so it is no longer serving as source of truth (painful after RFC-807, for instance).  Point instead to https://dm.lsst.org/sdm_schemas/browser/ and/or LDM-153.

DM-22078 - Reconciling the DPDD with the required information for pipeline QA. IN PROGRESS

  • ObjectID in the Source table:
    • We want an ObjectId in the Source table but we can't guarantee that every Source will have a (good) match. Provide a set of flags that tell the quality of the match.
    • Can Sources match to Solar System Objects. Yes, must be possible. (Then we definitely cannot partition by that match)
    • Need a follow-up to address the Solar System topic more thoroughly.
  • Ran out of time, remaining topics deferred to a later SST meeting.
12:30 - 13:00 Break

Moderator: Eric Bellm 

Notetaker: Colin Slater 

13:00 - 13:30
  • Verification plan through end of construction

Incl.

  • Slide 16, the P2 and P3 dates seem stronger than the definitions of P2 and P3 (in that they could be deferred much later than the dates in the bullet point).
  • Could we make one of the auxtel runs one of the ops rehearsals? Sounds like yes.
  • Integration milestones have explicit prerequisites, but haven't written those down for ops rehearsals. Robert would like the ops rehearsals to only use components that have already been demonstrated/verified by construction.
  • Need to add the RSP's Level 2 milestones, LDM-503-RSPa and LDM-503-RSPb, and clarify whether the latter is a predecessor of "DM Ready for Science Verification" (as I've been assuming) or a peer of it.
  • Question of whether the construction requirements are sufficiently related to the milestones to make them verifiable as part of claiming the milestone, or whether a separate acceptance campaign would provide a better connection to requirements.
  • Eric: sense that the upcoming milestones are large and vague, making it difficult to tie to development specifics.
13:30Close



Pre-meeting suggested topics 


Topic Who NotesVote for topic
Special programsWe have many tickets open on special programs and a draft TN. Much of special programs can be left to ops but there are some topics that need to be resolved in construction. We could use this time to conclude on open topics
Requirements on user batchRequirements on user batch will be presented at DMLT : https://dmtn-202.lsst.io/ . Short review of final TN 

Wil O'Mullane also on DMLT

Sources in the prompt data products

At PCW2021 this topic was raised. For some science cases (e.g., searching for Solar System objects where there are no templates) even severely paired down source catalogs (just ra, dec, psfMag) would be extremely useful .

DM-15307 - Include Sources in Prompt Products Database TO DO


Commissioning needs for cross-match tables

Keith available from 10:15 PT

Follow on from discussions at  2021-08-30 DM-SST

LIT-552 - Plan for and provide cross-identification tables between LSST objects and external catalogs (1f) MANAGER APPROVED

LPG: Ideally the outcome would be a TN descrining the needs. 

Complete reconciliation of DPDD with pipelines

Reconciling the DPDD with the required information for pipeline QA.

  • Will we provide an ObjectId in the Source table - need a clear statement on this. 
  • Can we remove detailed schema information from the DPDD so it is no longer serving as source of truth (painful after RFC-807, for instance).  Point instead to (on-project) https://fritzm.github.io/sdm_schemas/browser/baseline and/or LDM-153.

DM requirements gap analysis

DM Requirements Gap Analysis


Verification plan through end of construction

Incl.