Date & Time 

  11:00 PDT

Location

Browser

Room System

Phone Dial-in

https://bluejeans.com/927260081

  1. Dial: 199.48.152.152 or bjn.vc
  2. Enter Meeting ID: 103664856 -or- use the pairing code

Dial-in numbers:

  • +1 408 740 7256
  • +1 888 240 2560 (US Toll Free)
  • +1 408 317 9253 (Alternate Number)

Meeting ID: 927260081

Attendees

Regrets

Discussion items

 ​

ItemWhoPre-Meeting NotesNotes and  Action Items

Project/Science Updates & Task review 

Leanne Guy

Zeljko Ivezic

  • PST recieved a note from DESC on "Note from DESC on pipelines". Read and discuss on #dm-sst-coord
  • Data security discussion with Livermore. Main remaining open issue is dealing with potential impactors. For objects that exceed 30deg/day and are determined by Livermore to not be sensitive assets, JPL/Earth impact people need to be alerted ASAP to run their own analysis for impactors. 
  • Cybersecurity, NSF subcontracted the JASONS  on security. Wil gave a report to the PST on the JASONS cybersecurity meeting at the ned of June. Discussion on implications for the summit - we need remote access.  Will probably have a VPN. 
  • Wil will propose to treat the Camera as a black box for security and ringfence it. Any implementation of security in the Camera will lead to a lot of delays.
  • September for a PST SciCollab talk on time sereis features? Works for Eric, (except 1 Sept)
  • Zeljko Ivezic :
    • Tiger Team with Bob to decide what to do about satellite trails - need an assessment of the impact of trails. This is on the horizon and comes from the NSF and delivered by the JASONS. 
    • Is there a DM session at PCW? Community facing - want to give a message in a controlled manner to update them on the status of pipelines. 
    • Thanks to Jeffrey Carlin , Melissa Graham , Leanne Guy
    • : notebooks from DP0 - felt like doing real science, truth table matching, etc.  Very mature and better than expected. 
User batch requirements

Gregory Dubois-Felsmann

As discussed at DMLT-vF2F - 2021-06-22/23, we have been requested to  provide an interpretation of the user batch requirements that will define the minimum viable system we need to deliver. 

See Level 3 Definition and Traceability for the collected relevant requirements (summary: they don't constrain this very well).


Output should be a referenceable document (DMTN)

  • Need to provide users with the capability to run ppl task based processing over images. 
  • For catalogs - will be available in parquet format now so a DASK/Spark- like f/w and s/w to assist users to run over the collection of our catalogs 
  • Variance quotas for different people - resource allocation mechanism to apply for more space beyond the baseline – Users Committee.  Possibly automatically assign a small increment to avoid allocating space that is not used. 1 → 5 but to 20 has to go to an allocation committee. Limit up to which more could be allocated but beyond, might have to comtrinute money for resources. 
    • Proposal process can take sci. merit and relase of results to teh whole community available as criteria. Promose not emforceable. 
    • E.g new pz measure - publish as a table to federate would get extra points - have said this publicly.
  • In public, we have said:
    • used the word batch and have said their would be a batch system. i.e that users could submit batch jobs from a nb to slum, etc .....
    • said we would not constrain people to run entirely in our f/w. MCMC, forward modelling of catalogs that might not fit in our f/w. Implied they could run those
    • Democratize access to sky and replace the need to compete for telescope time. This had to come with compute resources for people with little access to resoources. These resources were not designed for DESC. They would saturate.
  • RHL: moset DESC questions about about can they do their science with our s/w - 10% is not enough
  • Stuck: 
    • Batch is close to h/w - involves the USDF people. 
    • Not standard in the RSP in a portable way. 
  • GPDF: conventional batch + DASK/Spark system + support for m/w 
    • CADC 2nd gen canfar - container based distributed batch-like system 
  • RHL: classical codes - do not forclose on supporting a posix files system, e.g s-xtracter
  • GPDF: Difficult to escape providing posix if we are going to let people provide their own code. To not do so would not be in conflict with requirements but there isa nexpectation, we have said this, be  a descope
  • CS:  Posix file system. does the env have to match, type of data access, is data access via APIs or posix itself? 
  • DMTN shoudl address these vague requirements in a quasi-technical manner. 
  • GPDF: Reluctant to say that our files should be available via posix, e.g all image files would be available via a filesystem. But have a posix like fs for outputs is sustainable. 
  • Use cases: time dependent lenses, high-redshift quasars, Andromeda - can download the data, bullk operations on catalogs. 
  • Will write a DMTN (not just an RFC). Quasi technical language, others to provide some use cases to support this and make clear what is out of scope. 
  • This can evolve over operations. Would be good to have for DP1.  Not for DP0.2 to get feedback.  Will we have user batch in the IDF at all? This will constrain the technical base. 
  • HFC: Might use different tools to fulfill all the requirements. 
  • Chase up whether DP1/2 will be on the IDF. What will be run at USDF/IDF?   Update: Yes, DP1/2 will be on the IDF. 
  • Gregory Dubois-Felsmann  to summarise requirements and user expectations on user batch in a short DMTN.  
10 oldest DM-SST issuesLeanne Guy
  1. DM-13226 - Getting issue details... STATUS
  2. DM-10110 - Getting issue details... STATUS
  3. DM-15654 - Getting issue details... STATUS
  4. DM-20300 - Getting issue details... STATUS
  5. DM-9046 - Getting issue details... STATUS
  6. DM-20766 - Getting issue details... STATUS
  7. DM-12584 - Getting issue details... STATUS
  8. DM-1016 - Getting issue details... STATUS
  9. DM-20563 - Getting issue details... STATUS
  10. DM-20311 - Getting issue details... STATUS



AOB

Leanne Guy

  • Do we want to follow DMLT's lead and have a break in August?
  • Next meeting is  2020-07-26
 We will break for the month of August and resume on 2021-09-20  (2021-09-06 is labour day holiday). 


List of SST tasks (Confluence)

DescriptionDue dateAssigneeTask appears on
  • Robert Lupton Clarify the meaning of time in the object table. 1 sentence description in sdm_schemas, can link to a short DMTN.  Update 2022-02-09: Meeting to resolve this on 2022-02-21  
28 Feb 2022Robert Lupton2018-11-05 DM SST F2F Agenda and Meeting notes
  • Gregory Dubois-Felsmann check if SDM standardization is adequately represented in project documents, and whether DMTN-067 should be required.
31 Mar 2022Gregory Dubois-Felsmann2022-02-14 DM-SST Virtual F2F Agenda and Meeting notes
28 Feb 2023Leanne Guy2023-01-23 DM-SST Agenda and Meeting Notes
  • Leanne Guy talk to Steve R about presenting plans for the ShearObject table to PST and SciCollab chairs   
20 Mar 2023Leanne Guy2023-02-27 DM-SST Agenda and Meeting Notes
31 Mar 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
  • Leanne Guy  talk to Gregory Dubois-Felsmann to review the original intent of the AFS-related Portal requirements before deciding on a course of action  
29 May 2023Leanne Guy2023-05-01 DM-SST Focus Meeting - Brokers in Commissioning
  • Leanne Guy Prepare to consult the PST on the question of providing compressed PVIs for AP outputs, to cover the period before the data become available in a DR.  
02 Jun 2023Leanne Guy2023-03-27 DM-SST Agenda and Meeting Notes
  • Jim Bosch Incorporate 30-60 day period for raws on disk into the strawman proposal and present to KT  
26 Jun 2023Jim Bosch2023-05-08 DM-SST Agenda and Meeting Notes
  • Parker Fagrelius Patrick Ingraham  how long will it take to do a scan as described? No need to scan the whole WL range but will require additional points outside nominal lambda range.  
30 Jun 2023Parker Fagrelius2023-03-27 DM-SST Agenda and Meeting Notes
31 Jul 2023Colin Slater2023-07-10 DM-SST Agenda and Meeting Notes
  • Eli Rykoff , Leanne Guy  Develop a proposal for what calibration processing, hardware, data we actually need and what will be needed for DR1. This has implications for the ORR and for prioritisation of work in commissioning  
31 Jul 2023Eli Rykoff2023-01-30 DM-SST Agenda and Meeting Notes
  • Yusra AlSayyad will look to see if there is any effort to help on option 1  
28 Aug 2023Yusra AlSayyad2023-08-14 DM-SST Agenda and Meeting Notes
  • Jim Bosch  Provide a physical example of that a  up on cell table would look like fo the Colin Slater / DAX team to review  
31 Aug 2023Jim Bosch2023-02-27 DM-SST Agenda and Meeting Notes
30 Nov 2023Gregory Dubois-Felsmann2023-10-23 DM-SST vF2F Agenda and Meeting Notes
30 Nov 2023Leanne Guy2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  •  "What is the pathway to defining the data products that are required to meet DMS-REQ-0266" Jeffrey Carlin   
30 Nov 2023Jeffrey Carlin2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  • Jeffrey Carlin follow up with KT on DMS-REQ-0176 and DMS-REQ-0315 to update/disaggregate this for latest base/summit infrastructure split.  
30 Nov 2023Jeffrey Carlin2023-10-23 DM-SST vF2F Agenda and Meeting Notes
  • Jim Bosch Follow up on the possibility of investigating further the ability to process 2 collections in parallel.   
31 Jan 2024Jim Bosch2023-12-04 DM-SST Agenda and Meeting Notes
31 Jan 2024Jeffrey Carlin2023-12-04 DM-SST Agenda and Meeting Notes
Gregory Dubois-Felsmann2023-10-23 DM-SST vF2F Agenda and Meeting Notes

DM-SST-relevant Jira Tickets

Key Summary Reporter Assignee Created Due
Refresh