(back to the list of all DMLT meeting minutes)

Location

ZOOM link in bookmarks of #dm-camelot on Slack

Time

10:15am PT

Attendees

Regrets

  • Leanne travel to DC

Discussion Items

ItemWhoNotes
NotesYusra AlSayyad last time Ian
Project Updates
  • NSF arriving in La Serena today
  • Victor here in Chile
    • Wants to check on doing mirror work earlier if needed
  • AMCR uneventful
    • still would like an end date we can live with
    • by next AMCR - run a couple of scenarios
    • AMCR visit in November
  • Strategic planning (9am PT) will be about camera delivery and M1M3 
    • If we do M1M3 in November we can not have Camera on summit before March.
    • baseline charge should be done now - 3552. We know it will potentially change after schedule workshop (OCT4)
    • need to know what prep is needed and which NOIRLab and off project resources can be deployed to help over next nine months.
  • No all hands meeting in Wednesday - Oct 6th (but clash with JOG) 10 Pacific (would avoid JOG a bit)
  • Crane - Meloni need to authorize local tech to visit and fix. Holding up M@ and ComCam work.
  • Dome cable. Cristián Silva needs to send communications to Jeff Barr.
  • More contractors for camera arrival - desire to have it done and waiting. Vacuum jacket lines in place .. some have long mead time (2-3 months)
requisitions

WOM: Working to get Reqs in by the end of the week

UW - have we converged ?

NCSA updates pending

SLAC - converged ?

Princeton - draft budget - sow ..


DEI


12:00 Pacific/MST   https://www.ted.com/talks/asif_sadiq_diversity_equity_inclusion_learning_how_to_get_it_right

See  #inclusion and https://www.lsst.org/about/dei

Community link: https://community.lsst.org/c/eji/45

UWS

Which team owns UWS or is willing to - it needs work on TTS right now .. who do we have ....

  • Universal Worker Service on TTS. This is a DM task.
  • FE: UWS is a standard, no one owns it. Are we talking about a particular implementation. K-T: Implementation by NCSA originally. FE: so USDF now. 
  • CTS: What is it running? K-T: The OCPS. It allows DM pipelines to be run in the summit environment from the OCS (by scriptqueue) 
  • WOM: Indeed its software that no one owns anymore. There's a question on whether we ought to have it or replace it. 
  • K-T: If you want script to trigger something. Batch isn't going to do it. Prompt Processing isn't going to do it. The back end can be replaced, the front end is necessary. 
  • SRP: What does it do again?
  • K-T: The OCPS EDC received a command from scriptqueue over DDS and executes a pipeline using this UWS backend service. 
  • WOM: Questions is where it ends up in operations. It needs to be owned by some group. We couuld dump it on T&S, but that doesn't feel quite right since its an opening to run DM pipelines on the control system
  • FE: We have to organizationally solve the problem that we have a missing team... that does Python level components between the DM layer. DM doesn't have any free cycles. You can use this as an argument to keep the software teams together. This becomes and easier problem if we keep it together.  But I don't want it. It's too close to observing and the dome, and I didn't have a say in it.
  • FE: the question is who takes the OCPS that drives pipelines at the summit.
  • K-T: It is currently in use. 
  • TJ: Robert had a dream where he'd tweak the CBP and take some data and then tweak the whatever and take some data. 
  • KTL: The only data-driven framework we have at the summit now is Rapid Analysis.
  • CTS: We don't have anyone who can make the call that we don't need UWS here right now. 
  • FE: We can discuss as people who know how to structure software, or we find someone who can answer "Do we need OCPS?" or can we use RapidAnalysis instead. 
  • KTL: These are two distinct things with two distinct use cases. While you can make one do the other, it becomes a weird fit. Driving processing via labels is different thatn driving processing via explicit responses. 
  • FE: I find it hard to believe that we've come up with all these orthogonal frameworks.
  • CTS: If I take 50 biases, and then combine them.  That last one is a command. Putting a command in the label works around not having a commanding system. 
  • WOM: Why can't this be a batch command that says "process these 50 biases" 
  • KTL: The backend can be batch, but the frontend is needed. The OCPS CSD is 500 lines of Python most of which is CSC boilerplate.
  • FE: Do all frameworks that DM come up with belong to one group? If we don't collect them under one umbrella, then we just perpetuate them. If DM owns frameworks that drive processing, then they need to come under one umbrella to make architectural decisions. 
  • FE: I do not own processing frameworks, My recommendation is that Wil find a single organizational umbrella. Perhaps middleware. 
  • WOM: I would like to replace the backend with batch, but that means installing condor.
  • TJ: Allegedly I'm supporting the prompt processing framework (SteveP). I don't think header service is a DM thing anymore. Camera has taken over. There's nothing DM about it. 
  • WOM: it's something we built in DM, (KT: because it fed our stuff) but is not really DM anymore. If we don't look after header service, I'm afraid of what we'll get out of it. 


3 things:

(1) Batch Service. Carlos is building htcondor. Can it be deployed on TTS? CS: It could be deployed but could be some porbles the low resources. 

(2) Need to make a case to Bob that between Andy, Tiago they need a new software group.

  • Wil O'Mullane writes up a proposal (consulting with Tiago, Andy, and Frossie) about what to do with header service and OCPS. 

(3) Short term we need someone to help with Michael R. KTL: I can help him with the front end, but not the backend. Sounds like I can't help him. Is this an emergency? K-T will talk to Michael R. No one at Princeton knows K8s, Some people at UW do. Frossie can help if its very time boxed. 

  • Report

https://docs.google.com/document/d/1jPUBfGJv4RjirA68GXHDL6-UzWY9wPXaEYZ2M1gzNzw/edit#heading=h.np561ki2n5yk
Review of Project Milestones


https://www.lsst.org/about/project-status

Level 2 Milestones


See also LSST Verification & Validation Documentation and LDM-692 (VCD).

Test Plans due in the next 45 days

Key Summary T Created Updated Due Assignee Reporter P Status Resolution
Loading...
Refresh

Milestones due in the next 45 days

Key Summary T Created Updated Due Assignee Reporter P Status Resolution
Loading...
Refresh

Risk Review



Risk items needing review

Key Summary T Created Updated Due Assignee Status days since review
Loading...
Refresh

Overdue risks (obligation date passed)

T Key Summary Assignee Reporter P Status Resolution Created Updated Due
Loading...
Refresh

Risk mitigations due at the end of this month

Key Summary T Created Updated Due Assignee Reporter P Status Resolution
Loading...
Refresh

Overdue risk mitigations

 

T Key Summary Assignee Reporter P Status Resolution Created Updated Due
Loading...
Refresh

DMLT Travel & Availability this week


Wil in summit Thursday again (not regular day).
Any Other Business



TCAM Stamdupas needed
Action review



DMLT Action Items (Confluence quick tasks)

Confluence action items on DMLT meeting minutes pages are meant to be mainly low-level actions that can be completed quickly.  More substantial tasks should appear in JIRA.  The label "DMLT" can be used (with any component) to mark a task as resulting from a DMLT action item.

DescriptionDue dateAssigneeTask appears onLabel(s)
  • Frossie Economou Will recommend additional Level 3 milestones for implementation beyond just the DAX-9 Butler provenance milestone.   
15 Mar 2022Frossie EconomouDM Leadership Team Virtual Face-to-Face Meeting, 2022-02-15 to 17meeting-notesdmltdmlt-f2f
  • Kian-Tat Lim Convene a meeting with Colin, Tim, Robert, Yusra to resolve graph generation with per-dataset quantities (likely based on Consolidated DB work).  
18 Mar 2022Kian-Tat LimDM Leadership Team Virtual Face-to-Face Meeting, 2022-02-15 to 17meeting-notesdmltdmlt-f2f
  • Frossie Economou Write an initial draft in the Dev Guide for what "best effort" support means  
17 Nov 2023Frossie EconomouDM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24dmltdmlt-f2fmeeting-notes~ebellm:favourite~mrawls:favourite~gpdf:favourite
  • Convene a group to redo the T-12 month DRP diagram and define scope expectations Yusra AlSayyad 
30 Nov 2023Yusra AlSayyadDM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24dmltdmlt-f2fmeeting-notes~ebellm:favourite~mrawls:favourite~gpdf:favourite
11 Dec 2023Gregory Dubois-FelsmannDM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24dmltdmlt-f2fmeeting-notes~ebellm:favourite~mrawls:favourite~gpdf:favourite
02 May 2024Frossie EconomouDMLT Meeting - 2024-04-22dmltmeeting-notes
22 May 2024 DMLT Meeting - 2024-04-22dmltmeeting-notes
22 May 2024Fabio HernandezDMLT Meeting - 2024-04-22dmltmeeting-notes
  • Richard Dubois USDF part in data facilities for PSTN-017 and distrib processing ? 
22 May 2024Richard DuboisDMLT Meeting - 2024-04-22dmltmeeting-notes
  • Tim Jenness - section on middleware for PSTN-017  
22 May 2024Tim JennessDMLT Meeting - 2024-04-22dmltmeeting-notes

DMLT Action Items (JIRA)

JIRA tasks with label "DMLT" but not TESTPLANS:

Key Summary T Created Updated Due Assignee Reporter P Status Resolution
Loading...
Refresh