March 2018 DM All Hands Meeting

Meeting Website

This Confluence page is being used for planning and coordinating the 2018 DM-AHM. For finalized arrangements, please refer to the meeting website.

Dates

Note the draft agenda for the meeting is now available here; as it's finalized, it will appear on the main meeting website.

Dates	Description	Who should be there	Rooms available
2018-03-05 PM	DM Subsystem Science Team meeting.	DM-SST members	KS410, KS415
2018-03-06	DM All Hands.	All DM staff	Hameetman, KS410, KS415, KS300
2018-03-07	DM All Hands.	All DM staff	Hameetman, KS410, KS415, KS300 (not 11am–noon)
2018-03-08 AM	DM All Hands.	All DM staff	Hameetman, KS410, KS415, KS300 (starting at10am)
2018-03-08 PM	DM Leadership Team meeting.	DMLT members	Hameetman, KS410, KS415, KS300 starting at10am)
2018-03-09 AM	DM Leadership Team meeting.	DMLT members	Hameetman, KS410, KS415
2013-03-09	Optional hack day.	Anybody who is interested in helping out & learning new things	Hameetman, KS410, KS415, MR214, KS300 (noon to 5pm)

Location

Meeting will take place at IPAC, Pasadena, California.

Hotel Info:

Hilton Pasadena, 168 South Los Robles Avenue, Pasadena, California 91101; Telephone: 626-577-1000.

To make a reservation on-line: http://ls.st/43x Reservations must be made by February 12!! (Link is active until 11:59pm PST on February 11)

Room Rate per night: $173

Group Name: LSST 2018 Data Management and Systems Engineering Meeting Group Code: LSST

Check-in: 04-MAR-2018; Check-out: 10-MAR-2018

Organizing Committee

Logistics

Program

Attendees

Please register here.

Goals

Get as many DM people together as possible
Discuss issues across DM
Show progress and share ideas from all over DM
Arrange specific side meetings with other subsystems (e.g. Camera) aligned with interfaces

Proposed Topics

This section is maintained for historical interest only; refer to the meeting agenda for details on what's actually happening.

Joint Sessions with SysEng

Topics Suggested by DM/On This Page

(Wil O'Mullane) Commissioning with Chuck and Brian .. rehearsal ?
(Frossie Economou) Since System Engineering will be there and since they are responsible for integration and facility-level validation and characterisation, could we have a presentation from them on timelines and what kind of tests they will be doing when? Yes I know there are docs but it would be good for the DM team to get the summary
(Kian-Tat Lim) SysEng viewing of DM testing
(Chuck Claver) Roles and Responsibilities between DM and Commissioning
(Chuck Claver) Differentiating DRP Verification (DM) and Validation (Commissioning)
(Chuck Claver) Differentiating Alert Production Verification (DM) and Validation (Commissioning)
(Simon Krughoff mentioned on behalf of Patrick Ingraham) the CBP needs to be commissioned and this will be a big effort because it's new technology. We should look at timelines and next steps. If we need more personnel, that should be addressed ASAP.
(Fritz Mueller via Kian-Tat Lim) Organization and processes (and systems?) in Commissioning and Operations.

Suggested by SysEng

Roles and Responsibilities in Verification/Validation
Planned DM tooling for Verification/Validation
Tooling extensions for commissioning
DM support during commissioning
Delineate DRP Verification (DM) and Validation (Commissioning)
Delineate Alert Processing Verification (DM) and Validation (Commissioning
LCR-1203: Searchable - Interactive Nightly Reports
Visualization tools for images (I think you all have already been thinking about this)
Mechanism for tagging and/or bundling sets of data to be processed for various tests during commissioning
Tools for creating survey coverage maps at fine angular resolution and for tracking metadata at the survey level (e.g., seeing in the set of exposures used to create a coadd as a function of sky position)
Who is responsible for assembling and validating external datasets that would be used during commissioning?
Tools for realtime data quality assessment to inform operations
Schedule of DM development during commissioning and how this interacts with verification/validation activities
Demo and intro to Jupyterlab as a development environment (this is requested to be a half day session)

DM-Focused Meeting Sessions

(Wil O'Mullane) Project status to kick off on Tuesday with Victor Krabbendam
(Wil O'Mullane) Stack releases going forward, support for previous releases and how will it all work in commissioning (also some SysEng interest, of course)
(Eric Bellm) The mini-broker: Capabilities, use cases, user interfaces, requirements development...
(Eric Bellm) obs package working group check-in
(John Swinbank) QA tooling and plans for moving forward.
- What are the requirements for instrumenting running pipelines (during development, during commissioning, during operations)?
- What are the requirements for QA “afterburners” (during development, during commissioning, during operations)?
- What are the long term plans for SQuaSH?
- What are the long term plans for the work carried out at Princeton? How does it relate to SQuaSH?
- What is the future of lsst.verify?
- Which datasets are we running in CI (or equivalent, e.g. regular manual runs)? Who is responsible for maintaining them?
- How do we distinguish ad-hoc developer requested processing from formal CI systems? How do make it possible for developers to test algorithms at scale?
- How do we unify ap_verify, validate_drp, ci_hsc, lsst_dm_stack_demo, etc, etc, etc.
- How do we provide the “drill down” workflow that is so frequently requested? On which datasets?
(Jeff Kantor) Debrief on status of Summit, Base, and Networks
(Jonathan Sick) Science Pipelines User Guide: status briefing and planning
- Highlight our new guidelines for writing Python docstrings.
- The user documentation style guide initiative.
- How to create a new-style user guide for your package and build it (hopefully via Jenkins by the time of the meeting).
- Discussion: how do we document our Pybind11 APIs?
- Gather feedback and address questions on the pipelines.lsst.io project from the teams.
(Jim Bosch) Gen. 3 Middleware: status and overview of upcoming changes (audience is all of DM)
(Jim Bosch) Gen. 3 Middleware: dev team design session (audience is people working on Gen. 3 Middleware this cycle and major stakeholders).
(Frossie Economou) Computational metrics way forward? (square, ldf, pipelines) - 30-45 min
(Kian-Tat Lim) Overall DM status: progress of each team
(Kian-Tat Lim) LDM-503 milestone definitions, test plan structure, test execution plans
(Kian-Tat Lim) Multi-core programming and efficiently working in limited memory
(Kian-Tat Lim) Current thinking on next-to-database processing
(Ian Sullivan) DCR-aware coadds and matched templates for image differencing
- I would like to discuss my preliminary implementation
- Would need to be on Tuesday or Wednesday
(Wil O'Mullane) Code of conduct refresher & update.
(Wil O'Mullane / Margaret Gelman / Unknown User (mbutler)) Updates on developer support (round robin?)
(Beth Willman) Orientation with DM communication tools, developer best practice.
(Margaret Gelman) Introduction to the service monitoring systems being built at LDF.
(Unknown User (xiuqin)) Follow-up status report on the action items generated during the LSP workshop, maybe a hack session on some specific items.
(Unknown User (xiuqin)) Demo of Firefly apabilities in display HiPS map images and conneciton to the underlying FITS images when zooming in/out.

Hack Day Projects

(Wil O'Mullane) Hack day .. build a mountain monitoring system with different portals for say DM manger and Commissioning manager. How do we integrate data from EFD, various monitoring systems and log files to present warning of interest to different individuals or groups ? How do we drill down to say " Show me all log messages around this event" How can that be made sensible (some logs may be very large and have many entries per second). What is the best electronic log book for operators to use and how will it tie in with procedure execution ? Would want Chuck Claver around for this
(Kian-Tat Lim) Hack day: make Header Service configurable
(John Parejko) Technical debt cleanup
- Finishing obs_base tests for decam and subaru (and porting them to the remaining obs packages): DM-8444
- Implement RFC-322 (flux → instFlux)
- Implement RFC-333 (*Sigma → *Err)
TAGGED! (Simon Krughoff) A (hack) session to add comment strings describing all the entries in obs_base.
- To which Jim Bosch adds: "Could we do that in a separate file (e.g. YAML mapping name -> description), maybe, so it'd be easier to machine-read and load into whatever form those definitions take in Gen3?"
TAGGED! (John Parejko) Implement a deprecation system, and document it, following RFC-213.
TAGGED! (Jim Bosch) (hack) Convert Task documentation to Sphinx/reStructuredText (would address a lot of DM-13161 - Getting issue details... STATUS ).
TAGGED! (Jonathan Sick) Science Pipelines Documentation Hacks. Let's get the Stack into https://pipelines.lsst.io. Participants will pick their favorite package and modernize its documentation:
- Add Sphinx documentation boilerplate based on our new project template.
- Learn how to build the docs locally and on Jenkins.
- Convert and write docstrings in the Numpydoc format.
(Simon Krughoff) Document repositories in /datasets. See DM-13508 - Getting issue details... STATUS and RFC-447 - Getting issue details... STATUS .
TAGGED! (John Parejko) Add Readme.rst files for all packages!
TAGGED! (Tim Jenness) Clean up afw:
- flake8 clean
- clang-tidy c++
- ndarray API change
- enable flake8 in travis, master protection, and up-to-date checks for afw pull requests.