You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 28 Next »




Logistics

Date 

2021 October 19 and 20

Location

This meeting will be virtual;

Join Zoom Meeting

https://noirlab-edu.zoom.us/j/98768200109?pwd=cDVWTzJCVzFuaDRBU0tqb0hEeEFCdz09

Meeting ID: 987 6820 0109
Password: 161803




Day 1, Tuesday October 19 2021

Time (Project)TopicCoordinatorPre-meeting notesRunning notes

Moderator: Leanne Guy 

Notetaker: Simon Krughoff 

09:00Welcome
Wil O'Mullane



09:15Project news and updates
Wil O'Mullane

  • Frossie: (re: Chuck's effort to identify missing scope) we don't have spare people at the moment
  • Wil: we have several potential pools of in kind, pre-ops, etc.
  • Frossie: just trying to manage expectations because prioritizing one thing means de-prioritizing another
  • Robert: some of these things will preclude us going on sky, so these hard prioritizations need to be done
  • Wil: a sticky issues is that the project was scoped for nominal operations, but commissioning is not nominal operations, so scope was definitely missed in tooling in that area


  • Leanne: we have requirements on the base.  Does handing off the base mean we have to do a test campaign to verify? Requirements in DMSR and in the ICD documents. 
  • Wil: We have already accepted the base, but we should go through any requirements and verify we have met them


  • Tim: Have we got a plan to off board people from github etc?
  • Wil: Update of DM team on regular basis?
  • K-T: Working on a one time script, but that's not a plan.
  • Wil: We don't always want to remove people when they go off project.  Sometimes they are willing to keep contributing
  • Tim: We could do what Eric suggested and just do a yearly walk through
  • Frossie: Kick people off the org and convert them to outside collaborators if they are going to continue contributing.  This should be left up to the T-CAM
  • Wil: Do we want to capture this on the off boarding list as a check box?
  • Frossie: Sure
  • Wil O'Mullane ask for a new entry in the off boarding form to make sure people are removed or moved to external collaborator status in github when they leave  
09:30Low hanging fruit milestones and how can we claim them 

Frossie Economou 

We have a bunch of milestones that can be closed with a bit of work that keeps getting put off or for lack of the formal testing being completed. Let's go through them and come up with a plan. 


Wil's google sheet for milestones

  • Robert: But what if these milestones don't actually reflect what work we actually need to do
  • Frossie: There may be a couple of those, but by and large that doesn't seem to be the case with the current set of milestones
  • Michelle: Can we try to organize the completion process a little?  E.g. put them all in a spreadsheet and identify next steps and responsible teams so that we can get a better handle on how to make progress
  • Wil: I have a sheet like that, though there are probably new ones.  I can resurrect that document
  • Gregory: Looking at overdue ones in my areas, they are all in active development.
  • Frossie: Those are tractable.  LDM-503-EFDc is a case where there is no one team who can retire it on their own.
  • Gregory: Exactly.  Maybe we should look for ones that haven't been started and that we may not even have architecture for


  • Fritz: My work is under represented by milestones relative to work left.
  • Frossie: I'm worried about user databases which does have a milestone and is a problem since it's one of these that takes many teams
  • Frossie: Maybe we should have each T-CAM classify their late milestones into: not relevant, mostly done, need to be moved to another location, etc.
  • Frossie Economou with T-CAMs will do some taxonomy to try to categorize late and soon to be due milestones  
  • Wil O'Mullane will update the spreadsheet to remove done ones and add new ones  
  • Yusra: Most milestones can be completed internally.  Exception is AP-15 and DRP-24.  We can't do anything more without precursor data
  • Fritz: We did a big walkthrough and filed LCRs and moved the needle.  Maybe we just need another one of those at the next F2F
  • Wil: We could do that
  • Gregory: Is there low hanging fruit?
  • Yusra: The ones in orange in Wil's spreadsheet are first guesses at the lowest hanging fruit
  • Frossie Economou will run a milestone "parade" for a time box starting 09:00 project on Thursday  

The output of the parade should be a fleshed out version of the spreadsheet.

10:05Data IDs

Frossie Economou 


  • Frossie: There is a way to uniquely identify a piece of data.  We need a way to hand these things around through services and processing etc.
  • Tim: If it's just coming from a service, we can just pass around opaque UUIDs.  These UUIDs are specific to a butler.
  • Jim: If you are getting them from the database, then UUIDs are the thing you want.  If the user needs to generate them, UUIDs are not going to work
  • Tim: We can't put the UUID in the metadata at put time because we don't know it and not all data types have a sense of metadata
  • Gregory: Service should translate obscreatorid (our UUID) to datasettype, collection and dataId and then pass that on to the processing
  • Tim: We could write a task that takes UUIDs and then does the query to unpack them and continue on
  • Simon: If the service is responsible for unpacking the UUIDs, it will also have to know which pipeline task to execute for each dataset type.
  • Jim: We can do a lot with dimensions we have, but we could add new ones through a schema migration.  It's painful, but if necessary, we can do it.
  • Frossie: As SQuaRE can I just care about UUIDs?
  • Gregory: No.  Your service will need to know how to turn UUIDs into DataRefs
  • Tim: getDataset is what you want
  • Frossie Economou will schedule a focused follow on meeting to discuss data identifiers with the relevant parties: e.g. Simon, Tim, Gregory, Yusra, Jim  
10:30Break

Moderator: Wil O'Mullane 

Notetaker: Cristián Silva 

11:00Verification plan through end of construction

Leanne Guy 

Picking up what I should have presented at an earlier F2F meeting (but got ill) 










12:30Break

Moderator: Wil O'Mullane 

Notetaker: Ian Sullivan 

1:00

Consolidated DB or equivalent

Frossie Economou



1:45

Sizing

Kian-Tat Lim 



14:00 Close

Day 2 October 20 2021

Moderator: 

 Notetaker:
9:00

Requirements on user batch

Leanne Guy 

Gregory Dubois-Felsmann



9:30RSP policy issues

Frossie Economou 



10:00Terminology 

Kian-Tat Lim 



10:15Focus Friday
Ian Sullivan 


10:30   Break

Moderator: Wil O'Mullane 

Notetaker: 

11:00


Risks 

Frossie Economou Yusra AlSayyad 



11:30Status part I


12:30

Break

Moderator: Robert Lupton Notetaker:

1:00


Status part II



1:45Wrap up 


2:00Close



Proposed Topics

TopicRequested byTime required (estimate)Notes
Requirements on user batch

Leanne Guy 

Gregory Dubois-Felsmann

30DM Science will present requirements on user batch. dmtn-202.lsst.io
How can we finalise Data IDs?Frossie Economou 30One of the biggest obstacles we have to putting VO services into (Rubin data) production is the fact that we do not have an agreed format for Data IDs uniquely identifying our data products. This information exists in a dict, but we don't have a scheme for converting it to a string. Let's discuss the complications and come up with a plan. 
Low hanging fruit milestones and how can we claim them Frossie Economou 60We have a bunch of milestones that can be closed with a bit of work that keeps getting put off or for lack of the formal testing being completed. Let's go through them and come up with a plan. 
Consolidated DB or equivalentFrossie Economou 45

We need a plan on how to get observation metadata somewhere where VO services can get them - this means probably the Consolidated DB (though, crazy idea, Butler registry seems to know most of this stuff?) Right now nobody seems to own this and be working it so we have to come up with an actionable plan. 

KTL: Isn't this DM-30853?

RSP policy issuesFrossie Economou 30User data guarantees, backwards compatibility promises, hybrid model impacts etc. 
Verification plan through oend of construction

Leanne Guy 

60Picking up what I should have presented at an earlier F2F meeting (but got ill) 
Focus FridayIan Sullivan 15

I am getting more feedback from developers getting frustrated by some aspects of Focus Friday. Note that these concerns are mostly addressed by the open support channels and by instructing them on how to use Slack's "Schedule for later" feature.

Sizing

Kian-Tat Lim 

15What changes in data products or pipeline step complexity or memory usage are known or anticipated?  What is the plan for DM-22082?
Terminology

Kian-Tat Lim 

15What else should we change while changing the default branch?


Attached Documents


Action Item Summary

DescriptionDue dateAssigneeTask appears on
  • Frossie Economou Will recommend additional Level 3 milestones for implementation beyond just the DAX-9 Butler provenance milestone.   
15 Mar 2022Frossie EconomouDM Leadership Team Virtual Face-to-Face Meeting, 2022-02-15 to 17
  • Kian-Tat Lim Convene a meeting with Colin, Tim, Robert, Yusra to resolve graph generation with per-dataset quantities (likely based on Consolidated DB work).  
18 Mar 2022Kian-Tat LimDM Leadership Team Virtual Face-to-Face Meeting, 2022-02-15 to 17
  • Frossie Economou Write an initial draft in the Dev Guide for what "best effort" support means  
17 Nov 2023Frossie EconomouDM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24
  • Convene a group to redo the T-12 month DRP diagram and define scope expectations Yusra AlSayyad 
30 Nov 2023Yusra AlSayyadDM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24
11 Dec 2023Gregory Dubois-FelsmannDM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24
02 May 2024Frossie EconomouDMLT Meeting - 2024-04-22
22 May 2024 DMLT Meeting - 2024-04-22
22 May 2024Fabio HernandezDMLT Meeting - 2024-04-22
  • Richard Dubois USDF part in data facilities for PSTN-017 and distrib processing ? 
22 May 2024Richard DuboisDMLT Meeting - 2024-04-22
  • Tim Jenness - section on middleware for PSTN-017  
22 May 2024Tim JennessDMLT Meeting - 2024-04-22
  • Cristián Silva - section on summit/data acquisition  for PSTN-017  
22 May 2024Cristián SilvaDMLT Meeting - 2024-04-22
Richard DuboisDMLT Meeting - 2024-04-29
  • Cristián Silva if you come to SLAC lets have plenty of photos of locks and racks etc ..
 Cristián SilvaDMLT Meeting - 2024-04-29


  • No labels