Skip to end of metadata
Go to start of metadata

DMLT photo at Connect

Logistics

Date

  -    (Main meeting Nov 2 and 3.)

Location

Bluejeans using DMLT BJ meeting 293724745.

  • Connect, 33 S. 5th Avenue, Tucson. This is a coworking location in downtown Tucson.
  • Large meeting room capable of holding 14 people.
  • 1 minute walk from suggested hotel.
  • Wifi connections are: 
    • Network:  Connect Devices    password: superman 
    • Network:  Connect Guest       password:  connectwithus


Hotel / Per Diem

Travel

  • Erin checked flight departures and
    • Latest flight from Tucson to Seattle NONSTOP is 1140 am weekly; later flights are available with layovers/plane changes (615 pm from Tucson is latest)
    • May need to overnight until Saturday for East Coast travelers as latest flight is about 340 pm 
  • Flight schedules are subject to change at any time so please get your Travel Request in ASAP!

Planning to attend

NB Directors Ops Review Oct 30 - Nov 1: means Donald Petravick, Margaret Gelman, Wil O'Mullane will be unavailable on 1 November.

Regrets

Agenda          PDF Agenda for expense reports

Day 1: 2017-11-02

TimeTopicChairNotes
09:00Welcome
  • Review outstanding actions
  • Review & adjust agenda
09:30F17 Milestones

The first three replanned level 2 milestones are due at the end of F17, ie at the end of November. These are:

  • LDM-503-1: Science Platform with WISE data in PDAC
  • LDM-503-2: HSC reprocessing
  • LDM-503-3: Alert generation validation

Topics for discussion:

  • Are the milestones fully and properly defined?
  • Are we on track to meet them?
  • How will we capture the results of this process?

Slides.


We agreed to appoint the following milestone "champions":

We agreed that these individuals will coordinate primarily with the managers to arrange the work that needs to be done, rather than directly issuing instructions to individual developers. However, managers are expected to do everything possible to facilitate the execution of the test protocols.

We agreed that work should start as soon as possible, preferably this month, on defining test specifications and plans for future milestones.

We acknowledged some uncertainty over the AuxTel schedule and how it will feed into the milestones.

We agreed that future milestones should not need champions: they will be planned in advance and handled through the regular line of management.

10:30ADASS & IVOA debrief

ADASS

  • The "HIPS discussion":
    • Gregory Dubois-Felsmann proposes that we should promptly RFC the adoption of HIPS/MOC as a standard LSST data products.
    • This is not an a replacement for any existing data products.
    • However, it suggests that we could use HEALPIX internally for all "sky binning" applications.
    • And/or for spatial indexing in DAX.
    • The DRP group is planning to flesh out our plans for completeness maps in the S18 cycle which will feed into this discussion.
  • From equity breakout: 
    • We should pass our job ads through an external service which will flag them for unconscious bias. Ranpal Gill has details.
    • We should ask the project for more help in identifying new avenues for hiring and recruitment.
    • There is unconscious bias training on Lynda.com.
  • We should investigate the "Safe Agile" development framework.
  • Science Platform breakout:
    • It's clear that the whole community is moving in the direction of providing next-to-data analysis services; the term "Science Platform" has caught on.
    • Wil O'Mullane has provided notes which are available on GitHub.
  • Issuing DOIs for data is a hot topic; we need to understand how to integrate this with our plans.

IVOA

  • A lot of constructive work being reported; not dominated by VO theory.
  • Discussion of registering authenticated services; got a consensus on a proposal that allows the VO Registry to show the existence of authenticated services.
  • Preliminary discussion of alternate forms for VOTables:
    • Support for FITS tables because they are blind indexable.
    • Little interest in SQLite format tables.
  • TAP:
    • Interesting work on the semantics of tabular data, generating a complex proposal.
    • LSST should aim to solve some simple issues.
  • VOEvent:
    • Discussion of the Williams et al. VOEvent evolution.
    • We should aim to make a concrete proposal for VOEvent at the next InterOp meeting.
  • VOSpace is now a well-adopted standard; we should use it.
  • STC2 is close to standardization, but there is no plan for serialization. Tim Jenness is pressing on this.
11:00Break
11:15Documentation

At the July DM review, we talked about a variety of test specifications, design documents, requirements documents, etc which were works in progress or (in some cases) not yet started. Are they now complete? If not, how do we complete them?


  • LDM-230 — Data Facility ConOps.
    • There will be some copy-editing work and filling in of blank sections carried out by Donald Petravick & Margaret Gelman.
    • Then they will submit this for an RFC.
  • LDM-554 — Science Platform Requirements.
    • Security requirements in this document need to be "indirected" by request of Donald Petravick.
    • There are no requirements on the Notebook & API aspects yet.
  • LDM-556 — Middleware Requirements.
    • SuperTask requirements exist in the document, but need editing.
    • Decision: wait to baseline this document until the Butler requirements are ready, in part because we expect some revision to the SuperTask requirements to harmonize them with the ButlerWG outputs.
    • Further discussion of Butler requirements postponed to this afternoon's discussion.
  • LDM-542 — Science Platform Design.
    • Will be discussed at the early-December Science Platform meeting.
    • Can then be RFCed.
  • LDM-502 — Measurement and Validation of the Science Pipelines KPMs.
    • This will be on hold waiting for Generalised Mario.
  • LDM-141 — Storage Sizing and I/O model & LDM-129 — Infrastructure Design
  • Data Backbone Requirements.
    • No handle issued yet.
    • Draft Word document has been circulated by Donald Petravick to some people.
  • Alert Production Requirements.
    • Does this cover all prompt processing or just the AP payload? The latter.
    • This includes the generation and distribution of alerts, by explicitly the work carried out at UW.
    • We will start with a direct flow down from DMSR, then Eric Bellm will edit and update as required.
  • Data Release Production Requirements
    • Will baseline a version including Jointcal in the next two weeks.
    • Flowdown from DMSR will follow in a future update.
  • LSE-61 — DM System Requirements
    • Various cleanups and flowdowns are waiting to be added from the SST, OSS, Science Platform work, LDM-135.
    • We will timebox creation of a new draft for baselining by the end of December, even if not all of the above is completed.
  • Donald Petravick: Propose LDM-230 (Concept of Operations for the LDF Services) for RFC by mid-December   
  • Gregory Dubois-Felsmann: Propose LDM-554 (LSP Requirements) for RFC by mid-December  
  • Gregory Dubois-Felsmann: Propose LDM-542 (LSP Design) for RFC  
  • Kian-Tat Lim & Donald Petravick: Get together to start work on updates for LDM-141 (Storage Sizing and I/O Model), LDM-129 (Infrastructure Design)  
  • Tim Jenness: Generate handle and flow down requirements from DMSR to AP Requirements.  
  • Tim Jenness: Baseline Jointcal requirements in the DRP Requirements.  
  • Tim Jenness: Propose new version of LSE-61 (DMSR) for baselining (timeboxed activity; move forward even if not all updates have been finished).  
12:00LSST Training

A short presentation from the LSST training officer introducing lynda.com.

DMLT requested S/MIME certificate for training emails with links. Lynda.com has implemented DMARC and I hear LSST will also be implementing DMARC in the near future. Is this a sufficient way to authenticate potential spam emails?

12:30Lunch (on your own)
13:30Data Compression WG report

Update on experiments and recommendation for compression, process on compressed images or not.

Presentation in PPT and PDF formats.


  • Significant interest in the results of this group beyond DM: Robert Gruendl will be invited to present the results of the WG to the PST.
  • Although the WG end date is constrained, we are cognisant of the manpower limitations on the WG. Given this, an extension of the end date is likely possible given approval from Wil O'Mullane.
13:50Obs Package WG report
  • The intention is not to hold the WG until the new-Butler-driven obs package converges, but rather to kick off and at least support the creation of obs_ztf in the short term.
14:10Butler WG report

Lead us through the important use cases, requirements and the rough design.

Bosch slides


  • The completion of this Working Group will be marked by a formal review of the delivered documentation. We expect the documents to be delivered over the next ~month.
  • Kian-Tat Lim: Set up review group for Butler WG outputs.  
15:15Break
15:30Calibration


  • Where are we with calibration products documentation?
  • Do we have a a calibration database schema and an API  (fairly simple find me calibration products relevant for time x — as opposed to "butler this for me"),
  • Who is involved in calibration activities?
  • When do we intend to have a plan with named people responsible for specific activities?
  • Which things do we need out of commissioning when?

Slides


We agreed that:

  • Designs for “the calibration database” belong in documentation curated by the DAX team.
  • Detailed definition of the database schema for calibration products is pending AuxTel and/or ComCam; it's ~1 year off.
  • Gaia GDR2 will be made available as RefCats for use in the Pipeline in ~April 2018. Loading them to Qserv is a separate job of work.
  • We need to provide user focused documents (technotes?) organized along algorithmic lines covering e.g. astrometry for distribution to science collaborations.
  • We agreed that the science focused documentation may be a deliverable from the DM Science WBS (under the direction of the — new — DM Subsystem Scientist), rather than (or as well as) Pipelines.
16:30Subsystem show & tellWil O'Mullane

5min or less per team including SST (T/CAM or product owner) to showcase highlights of work done since the last F2F and main goal(s) for S18.

17:30Finish
18:30Group dinner at Downtown Kitchen, 135 South 6th Avenue (3 minutes walk)

Day 2: 2017-11-03

08:30IHS ticket handling & procurement at NCSA

Overview of IHS ticket tracking procedures and the procedure for out of band procurement at NCSA. Perhpas touching on communication channels and expediters.

09:00Transition to Operations

Short session on operations and how we get there - transition team. Donald Petravick will cover the inclusion of DOE staff.

09:30PVIs and raw data storage
10:00Break
10:05SUIT Show & TellGregory Dubois-FelsmannPostponed from yesterday.
10:10Planning for FY18

Plans for S18 - We should understand and agree on all of the work that will be undertaken by the various teams during the S18 cycle.

Each team 5 minutes max on milestones and what is expected from other teams:

Other work potentially not covered:

  • Supporting stack releases ( RFC-188 - Getting issue details... STATUS )
  • Test specs and test reports (morning topic)
  • Working group output — Data Access and Task Framework updates
  • Completing the L3 milestone schedule

  • We will ask Jim Bosch to act as product owner for Butler Gen 3.
  • Fritz Mueller will act as T/CAM for the Butler Gen 3 effort (at least during S18).
  • We agreed that many of our L3 milestones are actually more correctly regarded as L4 (ie, they are internal to particular DM teams). Only deployed services or other functionality which is intended for consumption by other teams should be recorded as L3. It's likely that almost all of these milestones will actually be delivered by the LDF as they stand up services.
  • Wil O'Mullane and John Swinbank will audit the existing list of L3 milestones, determine which should actually be L4, and identify services which are not unambiguously covered by one or more milestones.  
11:30Pipeline output datasetGregory Dubois-Felsmann

Discussions arising from RFC-243.


  • Gregory Dubois-Felsmann: (work with Unknown User (xiuqin)) to schedule work to provide information on the metadata which should be included in pipeline outputs to make them useful for SUIT.   DM-12813 - Getting issue details... STATUS
  • Kian-Tat Lim: produce a technical proposal for including database ingest and archiving of data for use by SUIT et al. in CI  
12:00Review actions & date of next meeting
12:30Working Lunch (provided) with continued FY18 planning
13:30Planning for FY18Continuation of earlier discussion.
14:30Close


Pre-Meeting Planning

Suggested topics for discussion

TopicRequested byTime required (estimate)Notes
Local Group Topic45minChance for local non DMLT members to share a topic of interest/worry and discuss directly with DMLT - could be one of the decision making topics or something else. If nothing is offered then this would not be on the agenda. Perhaps not so obviously needed in Tucson .. but at other locations I would like to allow a channel for DM people to quasi formally discuss issues that worry them or that they think we should be looking into.
Science Website1 hrIn the August 2017 presentation on DM/Community Interaction Strategy to the SAC, we stressed the need for having a maintained website geared towards scientists with key LSST numbers, documents, notes, and quantities (see slide #12 in the presentation). I think we need to understand how we assign responsibility for organization and maintenance of such a website, its initial setup, and the interaction and overlap with the overall lsst.org website that the Communications Team maintains. May be good to have this discussion F2F with the comm's team while you're in Tucson.


Attached Documents

  File Modified
Microsoft Powerpoint Presentation Gruendl_DMLT_F2F_20171102.pptx Nov 02, 2017 by Robert Gruendl
PDF File Gruendl_DMLT_F2F_20171102.pdf Nov 02, 2017 by Robert Gruendl
PDF File 20171102-dmlt_milestones.pdf Nov 02, 2017 by John Swinbank
PDF File Generation 3 Butler Design.pdf Nov 02, 2017 by Jim Bosch
PDF File Arch DMLT 2017-11.pdf Nov 02, 2017 by Kian-Tat Lim
PDF File dmlt2f2_2017_11.pdf Nov 02, 2017 by Frossie Economou
PDF File DMLT_F2F_20171102_DAX_F17_Highlights.pdf Nov 02, 2017 by Fritz Mueller
PDF File 20171102-dmlt_f17highlights.pdf Nov 02, 2017 by John Swinbank
PDF File lupton.pdf Nov 02, 2017 by Robert Lupton
Microsoft Powerpoint Presentation DMLT-LDF Recap & Nearterm Goals.pptx Nov 02, 2017 by Margaret Gelman
PDF File dmlt2f2_2017_11._s18.pdf Nov 03, 2017 by Frossie Economou
PDF File DMLT_F2F_20171102_DAX_S18_Planning.pdf Nov 03, 2017 by Fritz Mueller
Microsoft Powerpoint Presentation LSST Lynda Slides.pptx Nov 03, 2017 by Chris Montgomery
PDF File LSST Lynda Slides.pdf Nov 03, 2017 by Chris Montgomery
PDF File 20171103-dmlt_s18.pdf Nov 03, 2017 by John Swinbank
PDF File Agenda_DMLT_F2F-Nov2017.pdf Nov 06, 2017 by lpetrick
PDF File DMLT-LDF Service Delivery Processes.pdf Nov 27, 2017 by Margaret Gelman
PDF File DMLT-LDF Procurement Process.pdf Nov 27, 2017 by Margaret Gelman

4 Comments

  1. Unless we are missing too many critical people, this may be a good time to hash out how to divide up the previously-unplanned work that will (IMO almost inevitably) be identified by the close of the Butler WG. Doing that will take more time than the Butler WG report already requested above.

    1. added to 14:30 Fy18 planning part .. if not covered by the teams ..

  2. Is there a location to upload talks (e.g. especially for remote presenters)?


    1. You can attach it to this confluence page.