DMLT photo at Connect
Logistics Date 01 Nov 2017 - 03 Nov 2017 (Main meeting Nov 2 and 3.) Location
Bluejeans using DMLT BJ meeting
293724745. Connect, 33 S. 5th Avenue, Tucson. This is a coworking location in downtown Tucson. Large meeting room capable of holding 14 people. 1 minute walk from suggested hotel. Wifi connections are: Network: Connect Devices password: superman Network: Connect Guest password: connectwithus Hotel / Per Diem Travel Erin checked flight departures and Latest flight from Tucson to Seattle NONSTOP is 1140 am weekly; later flights are available with layovers/plane changes (615 pm from Tucson is latest) May need to overnight until Saturday for East Coast travelers as latest flight is about 340 pm Flight schedules are subject to change at any time so please get your Travel Request in ASAP! Planning to attend NB Directors Ops Review Oct 30 - Nov 1: means Donald Petravick, Margaret Gelman, Wil O'Mullane will be unavailable on 1 November. Remote Jeff Kantor (not attending most of meeting, will be in Chile, on Summit on 11/2 all day) Eric Bellm (will call in both days; remote due to a commitment on Friday) Robert Gruendl (will call in Thursday for Working Group Reports session) Regrets
Day 1: 2017-11-02 Time Topic Chair Notes 09:00 Welcome Review outstanding actions Review & adjust agenda 09:30 F17 Milestones
The first three replanned level 2 milestones are due at the end of F17, ie at the end of November. These are:
LDM-503-1: Science Platform with WISE data in PDAC LDM-503-2: HSC reprocessing LDM-503-3: Alert generation validation
Topics for discussion:
Are the milestones fully and properly defined? Are we on track to meet them? How will we capture the results of this process? Slides.
We agreed to appoint the following milestone "champions":
We agreed that these individuals will coordinate primarily with the managers to arrange the work that needs to be done, rather than directly issuing instructions to individual developers. However, managers are expected to do everything possible to facilitate the execution of the test protocols.
We agreed that work should start as soon as possible, preferably this month, on defining test specifications and plans for future milestones.
We acknowledged some uncertainty over the AuxTel schedule and how it will feed into the milestones.
We agreed that future milestones should not need champions: they will be planned in advance and handled through the regular line of management.
10:30 ADASS & IVOA debrief ADASS The "HIPS discussion": Gregory Dubois-Felsmann This is not an a replacement for any existing data products. However, it suggests that we could use HEALPIX internally for all "sky binning" applications. And/or for spatial indexing in DAX. The DRP group is planning to flesh out our plans for completeness maps in the S18 cycle which will feed into this discussion. From equity breakout: We should pass our job ads through an external service which will flag them for . Ranpal Gill has details. We should ask the project for more help in identifying new avenues for hiring and recruitment. There is unconscious bias training on Lynda.com. We should investigate the "Safe Agile" development framework. Science Platform breakout: It's clear that the whole community is moving in the direction of providing next-to-data analysis services; the term "Science Platform" has caught on. Wil O'Mullane has provided notes which are available on GitHub. Issuing DOIs for data is a hot topic; we need to understand how to integrate this with our plans. IVOA A lot of constructive work being reported; not dominated by VO theory. Discussion of registering authenticated services; got a consensus on a proposal that allows the VO Registry to show the existence of authenticated services. Preliminary discussion of alternate forms for VOTables: Support for FITS tables because they are blind indexable. Little interest in SQLite format tables. TAP: Interesting work on the semantics of tabular data, generating a complex proposal. LSST should aim to solve some simple issues. VOEvent: Discussion of the Williams et al. VOEvent evolution. We should aim to make a concrete proposal for VOEvent at the next InterOp meeting. VOSpace is now a well-adopted standard; we should use it. STC2 is close to standardization, but there is no plan for serialization. Tim Jenness is pressing on this. 11:00 Break 11:15 Documentation
At the July DM review, we talked about a variety of test specifications, design documents, requirements documents, etc which were works in progress or (in some cases) not yet started. Are they now complete? If not, how do we complete them?
LDM-230 — Data Facility ConOps. There will be some copy-editing work and filling in of blank sections carried out by Donald Petravick & Margaret Gelman. Then they will submit this for an RFC. LDM-554 — Science Platform Requirements. Security requirements in this document need to be "indirected" by request of Donald Petravick. There are no requirements on the Notebook & API aspects yet. LDM-556 — Middleware Requirements. SuperTask requirements exist in the document, but need editing. Decision: wait to baseline this document until the Butler requirements are ready, in part because we expect some revision to the SuperTask requirements to harmonize them with the ButlerWG outputs. Further discussion of Butler requirements postponed to this afternoon's discussion. LDM-542 — Science Platform Design. Will be discussed at the early-December Science Platform meeting. Can then be RFCed. LDM-502 — Measurement and Validation of the Science Pipelines KPMs. This will be on hold waiting for Generalised Mario. LDM-141 — Storage Sizing and I/O model & LDM-129 — Infrastructure Design Data Backbone Requirements. No handle issued yet. Draft Word document has been circulated by Donald Petravick to some people. Alert Production Requirements. Does this cover all prompt processing or just the AP payload? The latter. This includes the generation and distribution of alerts, by explicitly the work carried out at UW. We will start with a direct flow down from DMSR, then Eric Bellm will edit and update as required. Data Release Production Requirements Will baseline a version including Jointcal in the next two weeks. Flowdown from DMSR will follow in a future update. LSE-61 — DM System Requirements Various cleanups and flowdowns are waiting to be added from the SST, OSS, Science Platform work, LDM-135. We will timebox creation of a new draft for baselining by the end of December, even if not all of the above is completed. Donald Petravick: Propose LDM-230 (Concept of Operations for the LDF Services) for RFC by mid-December 15 Dec 2017 Gregory Dubois-Felsmann: Propose LDM-554 (LSP Requirements) for RFC by mid-December 15 Dec 2017 Gregory Dubois-Felsmann: Propose LDM-542 (LSP Design) for RFC 19 Mar 2018 Kian-Tat Lim & Donald Petravick: Get together to start work on updates for LDM-141 (Storage Sizing and I/O Model), LDM-129 (Infrastructure Design) 31 Dec 2017 Tim Jenness: and flow down requirements from DMSR to AP Requirements. 15 Dec 2017 Tim Jenness: Baseline Jointcal requirements in the DRP Requirements. 16 Nov 2017 Tim Jenness: Propose new version of LSE-61 (DMSR) for baselining (timeboxed activity; move forward even if not all updates have been finished). 31 Dec 2017 12:00 LSST Training
A short presentation from the LSST training officer introducing lynda.com.
DMLT requested S/MIME certificate for training emails with links.
Lynda.com has implemented DMARC and I hear LSST will also be implementing DMARC in the near future. Is this a sufficient way to authenticate potential spam emails? 12:30 Lunch (on your own) 13:30 Data Compression WG report
Update on experiments and recommendation for compression, process on compressed images or not.
PPT and PDF formats. Significant interest in the results of this group beyond DM: Robert Gruendl will be invited to present the results of the WG to the PST. Although the WG end date is constrained, we are cognisant of the manpower limitations on the WG. Given this, an extension of the end date is likely possible given approval from Wil O'Mullane. 13:50 Obs Package WG report The intention is not to hold the WG until the new-Butler-driven obs package converges, but rather to kick off and at least support the creation of obs_ztf in the short term. 14:10 Butler WG report
Lead us through the
important use cases, requirements and the rough design. Bosch slides The completion of this Working Group will be marked by a formal review of the delivered documentation. We expect the documents to be delivered over the next ~month. Kian-Tat Lim: Set up review group for Butler WG outputs. 01 Dec 2017 15:15 Break 15:30 Calibration Where are we with calibration products documentation? Do we have a a calibration database schema and an API (fairly simple find me calibration products relevant for time x — as opposed to "butler this for me"), Who is involved in calibration activities? When do we intend to have a plan with named people responsible for specific activities? Which things do we need out of commissioning when? Slides
We agreed that:
Designs for “the calibration database” belong in documentation curated by the DAX team. Detailed definition of the database schema for calibration products is pending AuxTel and/or ComCam; it's ~1 year off. Gaia GDR2 will be made available as RefCats for use in the Pipeline in ~April 2018. Loading them to Qserv is a separate job of work. We need to provide user focused documents (technotes?) organized along algorithmic lines covering e.g. astrometry for distribution to science collaborations. We agreed that the science focused documentation may be a deliverable from the DM Science WBS (under the direction of the — new — DM Subsystem Scientist), rather than (or as well as) Pipelines. 16:30 Subsystem show & tell Wil O'Mullane 5min or less per team including SST (T/CAM or product owner) to showcase highlights of work done since the last F2F and main goal(s) for S18. 17:30 Finish 18:30 Group dinner at Downtown Kitchen, 135 South 6th Avenue ( 3 minutes walk) Day 2: 2017-11-03 08:30 IHS ticket handling & procurement at NCSA
Overview of IHS ticket tracking procedures and the procedure for out of band procurement at NCSA. Perhpas touching on communication channels and expediters.
09:00 Transition to Operations
Short session on operations and how we get there - transition team.
Donald Petravick will cover the inclusion of DOE staff. 09:30 PVIs and raw data storage 10:00 Break 10:05 SUIT Show & Tell Gregory Dubois-Felsmann Postponed from yesterday. 10:10 Planning for FY18
Plans for S18 - We should understand and agree on all of the work that will be undertaken by the various teams during the S18 cycle.
Each team 5 minutes max on milestones and what is expected from other teams:
Other work potentially not covered:
Supporting stack releases (
RFC-188 Getting issue details...
Test specs and test reports (morning topic) Working group output — Data Access and Task Framework updates Completing the L3 milestone schedule We will ask Jim Bosch to act as product owner for Butler Gen 3. Fritz Mueller will act as T/CAM for the Butler Gen 3 effort (at least during S18). We agreed that many of our L3 milestones are actually more correctly regarded as L4 (ie, they are internal to particular DM teams). Only deployed services or other functionality which is intended for consumption by other teams should be recorded as L3. It's likely that almost all of these milestones will actually be delivered by the LDF as they stand up services. Wil O'Mullane and John Swinbank will audit the existing list of L3 milestones, determine which should actually be L4, and identify services which are not unambiguously covered by one or more milestones. 06 Nov 2017 11:30 Pipeline output dataset Gregory Dubois-Felsmann
Discussions arising from
RFC-243. Gregory Dubois-Felsmann: (work with Unknown User (xiuqin)) to schedule work to provide information on the metadata which should be included in pipeline outputs to make them useful for SUIT. 15 Nov 2017
DM-12813 Getting issue details...
Kian-Tat Lim: produce a technical proposal for including database ingest and archiving of data for use by SUIT et al. in CI 15 Dec 2017 12:00 Review actions & date of next meeting 12:30 Working Lunch (provided) with continued FY18 planning 13:30 Planning for FY18 Continuation of earlier discussion. 14:30 Close Pre-Meeting Planning Suggested topics for discussion
Topic Requested by Time required (estimate) Notes Local Group Topic 45min Chance for local non DMLT members to share a topic of interest/worry and discuss directly with DMLT - could be one of the decision making topics or something else. If nothing is offered then this would not be on the agenda. Perhaps not so obviously needed in Tucson .. but at other locations I would like to allow a channel for DM people to quasi formally discuss issues that worry them or that they think we should be looking into. Science Website 1 hr In the August 2017 presentation on DM/Community Interaction Strategy to the SAC, we stressed the need for having a maintained website geared towards scientists with key LSST numbers, documents, notes, and quantities (see slide #12 in the presentation). I think we need to understand how we assign responsibility for organization and maintenance of such a website, its initial setup, and the interaction and overlap with the overall lsst.org website that the Communications Team maintains. May be good to have this discussion F2F with the comm's team while you're in Tucson. Attached Documents