Meeting Website

This Confluence page is being used for planning and coordinating the 2018 DM-AHM. For finalized arrangements, please refer to the meeting website.


Note the draft agenda for the meeting is now available here; as it's finalized, it will appear on the main meeting website.

DatesDescriptionWho should be thereRooms available
2018-03-05 PM

DM Subsystem Science Team meeting.

DM-SST membersKS410, KS415
2018-03-06DM All Hands.All DM staffHameetman, KS410, KS415, KS300
2018-03-07DM All Hands.All DM staffHameetman, KS410, KS415, KS300 (not 11am–noon)
2018-03-08 AMDM All Hands.All DM staffHameetman, KS410, KS415, KS300 (starting at10am)

2018-03-08 PM

DM Leadership Team meeting.DMLT membersHameetman, KS410, KS415, KS300 starting at10am)
2018-03-09 AMDM Leadership Team meeting.DMLT membersHameetman, KS410, KS415
2013-03-09Optional hack day.Anybody who is interested in helping out & learning new thingsHameetman, KS410, KS415, MR214, KS300 (noon to 5pm)


Meeting will take place at IPAC, Pasadena, California.

Hotel Info: 

Hilton Pasadena, 168 South Los Robles Avenue, Pasadena, California  91101;  Telephone:  626-577-1000

To make a reservation on-line:    Reservations must be made by February 12!!  (Link is active until 11:59pm PST on February 11)

Room Rate per night:  $173

Group Name:   LSST 2018 Data Management and Systems Engineering Meeting     Group Code:  LSST

Check-in:   04-MAR-2018;  Check-out: 10-MAR-2018

Organizing Committee





  • Get as many DM people together as possible
  • Discuss issues across DM
  • Show progress and share ideas from all over DM
  • Arrange specific side meetings with other subsystems (e.g. Camera) aligned with interfaces

Proposed Topics

This section is maintained for historical interest only; refer to the meeting agenda for details on what's actually happening.

Joint Sessions with SysEng

Topics Suggested by DM/On This Page

  • (Wil O'Mullane) Commissioning with Chuck and Brian .. rehearsal ?
  • (Frossie Economou) Since System Engineering will be there and since they are responsible for integration and facility-level validation and characterisation, could we have a presentation from them on timelines and what kind of tests they will be doing when? Yes I know there are docs but it would be good for the DM team to get the summary
  • (Kian-Tat Lim) SysEng viewing of DM testing
  • (Chuck Claver) Roles and Responsibilities between DM and Commissioning
  • (Chuck Claver) Differentiating DRP Verification (DM) and Validation (Commissioning)
  • (Chuck Claver) Differentiating Alert Production Verification (DM) and Validation (Commissioning)
  • (Simon Krughoff mentioned on behalf of Patrick Ingraham) the CBP needs to be commissioned and this will be a big effort because it's new technology.  We should look at timelines and next steps.  If we need more personnel, that should be addressed ASAP.
  • (Fritz Mueller via Kian-Tat Lim) Organization and processes (and systems?) in Commissioning and Operations.

Suggested by SysEng

  • Roles and Responsibilities in Verification/Validation
  • Planned DM tooling for Verification/Validation
  • Tooling extensions for commissioning
  • DM support during commissioning
  • Delineate DRP Verification (DM) and Validation (Commissioning)
  • Delineate Alert Processing Verification (DM) and Validation (Commissioning
  • LCR-1203: Searchable - Interactive Nightly Reports
  • Visualization tools for images (I think you all have already been thinking about this)
  • Mechanism for tagging and/or bundling sets of data to be processed for various tests during commissioning
  • Tools for creating survey coverage maps at fine angular resolution and for tracking metadata at the survey level (e.g., seeing in the set of exposures used to create a coadd as a function of sky position)
  • Who is responsible for assembling and validating external datasets that would be used during commissioning?
  • Tools for realtime data quality assessment to inform operations
  • Schedule of DM development during commissioning and how this interacts with verification/validation activities
  • Demo and intro to Jupyterlab as a development environment (this is requested to be a half day session)

DM-Focused Meeting Sessions

  • (Wil O'Mullane) Project status to kick off on Tuesday with Victor Krabbendam
  • (Wil O'Mullane) Stack releases going forward, support for previous releases and how will it all work in commissioning (also some SysEng interest, of course)
  • (Eric Bellm) The mini-broker: Capabilities, use cases, user interfaces, requirements development...
  • (Eric Bellm) obs package working group check-in
  • (John Swinbank) QA tooling and plans for moving forward.
    • What are the requirements for instrumenting running pipelines (during development, during commissioning, during operations)?
    • What are the requirements for QA “afterburners” (during development, during commissioning, during operations)?
    • What are the long term plans for SQuaSH?
    • What are the long term plans for the work carried out at Princeton? How does it relate to SQuaSH?
    • What is the future of lsst.verify?
    • Which datasets are we running in CI (or equivalent, e.g. regular manual runs)? Who is responsible for maintaining them?
    • How do we distinguish ad-hoc developer requested processing from formal CI systems? How do make it possible for developers to test algorithms at scale?
    • How do we unify ap_verify, validate_drp, ci_hsc, lsst_dm_stack_demo, etc, etc, etc.
    • How do we provide the “drill down” workflow that is so frequently requested? On which datasets?
  • (Jeff Kantor) Debrief on status of Summit, Base, and Networks
  • (Jonathan Sick) Science Pipelines User Guide: status briefing and planning

    • Highlight our new guidelines for writing Python docstrings.

    • The user documentation style guide initiative.
    • How to create a new-style user guide for your package and build it (hopefully via Jenkins by the time of the meeting).
    • Discussion: how do we document our Pybind11 APIs?
    • Gather feedback and address questions on the project from the teams.
  • (Jim Bosch) Gen. 3 Middleware: status and overview of upcoming changes (audience is all of DM)
  • (Jim Bosch) Gen. 3 Middleware: dev team design session (audience is people working on Gen. 3 Middleware this cycle and major stakeholders).
  • (Frossie Economou) Computational metrics way forward? (square, ldf, pipelines) - 30-45 min
  • (Kian-Tat Lim) Overall DM status: progress of each team
  • (Kian-Tat Lim) LDM-503 milestone definitions, test plan structure, test execution plans
  • (Kian-Tat Lim) Multi-core programming and efficiently working in limited memory
  • (Kian-Tat Lim) Current thinking on next-to-database processing
  • (Ian Sullivan) DCR-aware coadds and matched templates for image differencing
    • I would like to discuss my preliminary implementation
    • Would need to be on Tuesday or Wednesday
  • (Wil O'Mullane) Code of conduct refresher & update.
  • (Wil O'Mullane / Margaret Gelman / Unknown User (mbutler)) Updates on developer support (round robin?)
  • (Beth Willman) Orientation with DM communication tools, developer best practice.
  • (Margaret Gelman) Introduction to the service monitoring systems being built at LDF.
  • (Unknown User (xiuqin)) Follow-up status report on the action items generated during the LSP workshop, maybe a hack session on some specific items.
  • (Unknown User (xiuqin)) Demo of Firefly apabilities in display HiPS map images and conneciton to the underlying FITS images when zooming in/out.

Hack Day Projects

  • (Wil O'Mullane) Hack day .. build a mountain monitoring system with different portals for say DM manger and Commissioning manager. How do we integrate data from EFD, various monitoring systems and log files to present warning of interest to different individuals or groups ? How do we drill down to  say " Show me all log messages around this event" How can that be made sensible (some logs may be very large and have many entries per second).  What is the best electronic log book for operators to use and how will it tie in with procedure execution ?  Would want Chuck Claver around for this
  • (Kian-Tat Lim) Hack day: make Header Service configurable
  • (John Parejko) Technical debt cleanup
    • Finishing obs_base tests for decam and subaru (and porting them to the remaining obs packages): DM-8444
    • Implement RFC-322 (flux → instFlux)
    • Implement RFC-333 (*Sigma → *Err)
  • TAGGED! (Simon Krughoff) A (hack) session to add comment strings describing all the entries in obs_base.
    • To which Jim Bosch adds: "Could we do that in a separate file (e.g. YAML mapping name -> description), maybe, so it'd be easier to machine-read and load into whatever form those definitions take in Gen3?"
  • TAGGED! (John Parejko) Implement a deprecation system, and document it, following RFC-213.
  • TAGGED! (Jim Bosch) (hack) Convert Task documentation to Sphinx/reStructuredText (would address a lot of DM-13161 - Getting issue details... STATUS ).
  • TAGGED! (Jonathan Sick) Science Pipelines Documentation Hacks. Let's get the Stack into Participants will pick their favorite package and modernize its documentation:
    • Add Sphinx documentation boilerplate based on our new project template.
    • Learn how to build the docs locally and on Jenkins.
    • Convert and write docstrings in the Numpydoc format.
  • (Simon Krughoff) Document repositories in /datasets.  See  DM-13508 - Getting issue details... STATUS  and RFC-447 - Getting issue details... STATUS .
  • TAGGED! (John Parejko) Add Readme.rst files for all packages!
  • TAGGED! (Tim Jenness) Clean up afw:
    • flake8 clean
    • clang-tidy c++
    • ndarray API change
    • enable flake8 in travis, master protection, and up-to-date checks for afw pull requests.


March 512:30pm - 2pmlunchall

Caltech has class in the conference room

March 712:30pm - 2pmlunchall

Caltech has class in the conference room

Nearby restaurants

For lunch within walking distance: (of course Yelp provides more choices)

238 S Lake Ave
Pasadena, CA 91101
Phone number (626) 796-1613
273 Shoppers Ln
Pasadena, CA 91101
Phone number (626) 793-8844
913 E California Blvd
Pasadena, CA 91106
Phone number (626) 795-1123
140 Shoppers Ln
Pasadena, CA 91101
Phone number (626) 440-1008
146 S Lake Ave
Pasadena, CA 91101
Phone number (626) 304-7700
546 S Lake Ave
Pasadena, CA 91101
Phone number (626) 793-2600
200 S Lake Ave
Pasadena, CA 91101
Phone number (626) 535-9561
264 S Lake Ave
Pasadena, CA 91101
Phone number (626) 449-2005
246 S Lake Ave
Pasadena, CA 91106
Phone number (626) 229-9173