Data Management
Space shortcuts
Page tree
Skip to end of metadata
Go to start of metadata

Full res on flickr

Logistics

Date

  - 

Location

  • Petersen Room, Allen Library, University of Washington.
  • Room 485 (fourth floor).
  • Room is available 09:00 on 22 May to 13:00 on 24 May.

Accommodation

Travel

  • From Seattle-Tacoma International Airport, the Link light rail will take you to the University of Washington Station.
  • The Allen Library (see location, above) is a ten minute walk from the station.
  • Ride-share services (Lyft, Uber, etc) will likely cost around $30-40 from the airport to the University District.

Per Diem

TotalBreakfastLunchDinnerIncidentalsFirst & Last Day of Travel
$74.00$17.00$18.00$34.00$5.00$55.50

Attendees

Traveling

  • ...


Local

Remote

  • ...

Regrets

Agenda

Day 0: 2018-05-21

TimeTopicChairNotes
09:00SST Meeting

Zeljko Ivezic / Leanne Guy

  • Agenda provided by DM System Science Team.
17:00Close

Day 1: 2018-05-22

09:00WelcomeWil O'Mullane
09:30Review outstanding DMLT action items and ticketsWil O'Mullane
10:00Compression WGWil O'Mullane / Unknown User (gruendl)


  • The proposal is that we can store compressed versions of PVIs for easy access for (e.g.) producing postage stamps.
    • Note that we are currently sized to store 30 days worth of (uncompressed) PVIs.
    • Anything older than that needs to be recomputed (by imgserv).
  • We, as DM, meed to develop a “menu” of ~3 options for science collaborations, including:
    • Time to produce calibrated image from raw;
    • Compression level;
    • Time range of compressed images stored.
  • This menu can likely be appended to the end of RFC-325 - Getting issue details... STATUS .
  • John Swinbank: Ensure pipelines code for regenerating images on demand is included in LDM-151, that an equivalent service is included in the appropriate DAX design documentation, and that both of them are included in the DM construction plan.  
  • Wil O'Mullane, Kian-Tat Lim: Provide “menu” of compression of options as an addition to RFC-325.  
  • Fritz Mueller: Develop a complete (to the best of our current understanding) list of data products that have to be accessible through DAX services, and a plan for extending that list in future.   DM-15186 - Getting issue details... STATUS
10:30Break (Refreshments Provided)
11:00QA WGJohn Swinbank
  • Report on current status.
  • Slides.
11:30Summit–Base ITC & networks status update

Jeff Kantor

12:00Status of HEALPix/MOC

Unknown User (xiuqin) / Gregory Dubois-Felsmann

  • Per RFC-441 HEALPix and MOC should be adopted as standard LSST data products.
  • What is the plan for and status of implementation of this?
  • Should we draft some requirements ?
  • Slides
12:30Lunch (Provided)
13:30Next-to-database processing

Leanne Guy / Fritz Mueller

  • Colin Slater: Publish technical note on next-to-database processing use cases.  
  • Wil O'Mullane: Following the publication of Colin Slater's technical note on next-to-database processing, decide what further meetings or discussions are required.  
14:00Pipelines output metadata

Kian-Tat Lim

  • Plans for having LSST Science Pipelines provide the metadata that is required by DAX & SUIT as part of standard processing.
  • Slides
  • Gregory Dubois-Felsmann: Refresh Confluence page describing the mapping of the LSST data model to CAOM, then propose it for RFC.  
  • Kian-Tat Lim: Map Butler v3 schema to CAOM.  Delegated in  DM-14555 - Getting issue details... STATUS  
  • John Swinbank / Yusra AlSayyad: Schedule work during F18 to construct a task which translates pipeline outputs to a format consistent with the DPDD.  
  • Gregory Dubois-Felsmann: Provide “meta-schema” of information which must be provided in pipelines output tables.  
14:30Nightly reportingWil O'Mullane
  • Status of LCR-1203
  • Its impact on DM requirements
  • Other reporting tools needed (? e.g. real time alert stream diagnostics)
15:00Break
15:30Modernizing the Science Pipelines codebase

Jim Bosch

  • Jim Bosch writes:
    • There is currently a lot of ad-hoc, grass-roots effort going into cleaning up the stack and its documentation that has at best inconsistent support from the DMLT, and this is both distracting developers from scheduled work and leaving them frustrated because the effort they're donating via free/science time isn't making a big dent.  DMLT needs to agree to and present a united and coherent policy and plan for modernization so we aren't fighting our own devs (and each other) on this. I will come with at least one rough proposal, but it may be too ambitious to be tenable, and I'd really like other members of DMLT to come armed with additional proposals
  • Please come to this discussion prepared as Jim suggests.
17:00Close
18:00

Everybody is invited to join the DM Project Manager & the local UW LSST team for drinks at Ravenna Brewing, 5408 26th Ave NE (25 minutes walk from the meeting venue, 15 minutes by bus).

19:30

Dinner at BaBar, 2685 NE 46th St (10 minutes walk from Ravenna Brewing).

If you will be coming for dinner, please provide Wil O'Mullane with $34 (your dinner per diem).

If you will not be coming for dinner, please let John Swinbank know in advance so we don't wait for you.

The restaurant will accommodate all the dietary needs that we know of; please make your needs known to the server.

Day 2: 2018-05-23

09:00EFD interfaces & requirementsFrossie Economou

Discuss LCR-1344 and the path forward with starting to exercise DM interfaces to the EFD.

We agreed to appoint Simon Krughoff as product owner of the DM-EFD.

  • Margaret Gelman: Organize a meeting with T&S to decide whether we extract from the EFD databases or from DDL.  
  • Simon Krughoff: Develop written use cases/requirements for DM-EFD. This is being handled in DM-15012 - Getting issue details... STATUS  
09:30Generation 3 Middleware next steps

Fritz Mueller / Jim Bosch

  • What's the current status of Gen 3 Butler & SuperTask as at the end of S18?
    • When will they be ready for widespread use?
    • What's our migration plan?
  • What resources are needed to continue middleware development into F18 and beyond?

Fritz slides (F18 Supertask status)

  • Fritz Mueller: Organize “mini review” of RFC-484 - Getting issue details... STATUS .  
  • Fritz Mueller / Jim Bosch / Yusra AlSayyad / John Swinbank: Discuss staffing plans for the F18 BG3 group.  
  • Fritz Mueller: Provide example QuantumGraph to the Data Facility.  Initial example (illustrative of QG API) was sent. Andy S. awaiting feedback/questions about the the API from NCSA.
10:30Break (Refreshments Provided)
11:00Test specifications in Jira

Unknown User (gcomoretto)

11:30Status of documentation for this summer's reviewsWil O'Mullane
  • Leanne Guy: Coordinate production of test cases covering all LSE-61 requirements.  
12:30Lunch (Provided)
13:30Codebase consistency

Kian-Tat Lim

  • Portal, Firefly, Qserv, Header Service, "L1" Image Ingest, Workload/Workflow, Data Backbone, Jellybean, SQuaSH: these use different languages, different dependencies, different skills than Science Pipelines, Task Framework, Butler.  Their code is generally not reviewed, understood, or maintained outside of their own groups.  What standards and processes are we and should we be applying to these to ensure that they function well and are maintainable for the lifetime of the project?
  • Slides
  • Kian-Tat Lim: Figure out which (project and/or DM level) documents need to be updated to better capture standards & processes for development within DM and elsewhere. LIT-337 - Getting issue details... STATUS and DM-14570 - Getting issue details... STATUS  
14:15Operations rehearsals

Margaret Gelman

  • We have multiple “operations rehearsal” milestones scheduled through construction:
    • LDM-503-09 (2019-01-04)
    • LDM-503-11 (2019-10-31)
    • LDM-503-12 (2020-01-31)
    • LDM-503-13 (2020-11-30)
    • LDM-503-15 (2021-10-29)
    • LDM-503-16 (2022-05-31)
  • We should start planning for what's involved in these activities.
  • Slides
  • Margaret Gelman: Circulate a skeleton plan for an operations rehearsal based on AuxTel.  
  • Margaret Gelman: Arrange for Felipe to construct realistic (in terms of file structure, headers, etc) images based on TS8 data.  
15:00Break
15:07Photo at the Fountain   - out the door turn right and its on the left ..
15:30Internal on-disk data representations.Simon Krughoff

Currently, the standard processing results in FITS files containing the afwTable catalogs from the processing. These are then transformed to Apache parquet format for use in QA endeavors. This means we are storing the same information in two redundant formats. I would like to explore how we treat the various file formats.

  • Do we treat some as transient and others as persistent?
  • Can we settle on a single format that will serve multiple needs with the understanding that some transformation may be necessary to serve the community?
  • Depending on the above, are there impacts on the sizing model that need to be taken into account (realizing that the sizing model is specifically for operations)?
16:00Product treeUnknown User (gcomoretto)Based on the different documents available (and in preparation), I would like to share some thoughts, maybe a proposal, for a categorization of the different products that can be identified (at the moment) inside DMS. (GCM_ProductTree.pptx)
16:30Future software releasesUnknown User (gcomoretto)

What's the plan for upcoming releases?

  • Will the next release be 15.1 or 16.0? What's the timescale? What documentation and/or characterization needs to be prepared to accompany it?
  • Are future releases time based, or linked to specific events in commissioning/I&T/etc?
    • Note that we now have LDM-503 milestones described as “Science Pipelines Release” preceding major rehearsals.
  • Do these releases include just Pipelines or also Firefly, Qserv, Platform, Data Facility, ...?
    • If those are not included in the stack release, what are their release plans?
  • GCM_Releases.pptx


  • Next release will be 16.0.0, coming in June. No changes to standard release process (modulo addition of extra “0”).
17:00Close

Day 3: 2018-05-24

09:00Plans for F18

John Swinbank

Milestone “owners”:

  • LDM-503-05: Eric/John
  • LDM-503-06: Margaret (to move to more appropriate date?)
  • LDM-503-07: John (should be changed to say “making test stand data available through a Butler”.)
  • LDM-503-08: Michelle
  • LDM-503-08b: Margaret
  • LDM-503-09a: John
  • LDM-503-09: Margaret (to move to more appropriate date)


10:30Break (Refreshments Provided)
11:00Plans for F18 (continued)John Swinbank
  • Resolve outstanding cross-team issues.
  • Agree on final plan.
12:30Review action items & plans for next meetingWil O'Mullane
  • Expected to be 2018-11-05/07 in Princeton.
  • Also lets open diaries for the next meeting or two after that - Tucson and Illinois perhaps.


  • Provisional dates for next-but-one meeting 26/27/28 February likely in Tucson.
  • Provisional dates for next-but-two meeting 21/22/23 May likely at NCSA.
13:00Close


Pre-Meeting Planning

Suggested topics for discussion


TopicRequested byTime required (estimate)Notes
Update on status of Summit - Base ITC and Networks

Jeff Kantor

20 minutes plus questionsProvide a status of the Summit and Base IT infrastructure installation and networks.
Modernizing the Science Pipelines codebase, or agreeing how not to.Jim Bosch60-90 minutes

There is currently a lot of ad-hoc, grass-roots effort going into cleaning up the stack and its documentation that has at best inconsistent support from the DMLT, and this is both distracting developers from scheduled work and leaving them frustrated because the effort they're donating via free/science time isn't making a big dent.  DMLT needs to agree to and present a united and coherent policy and plan for modernization so we aren't fighting our own devs (and each other) on this.  I will come with at least one rough proposal, but it may be too ambitious to be tenable, and I'd really like other members of DMLT to come armed with additional proposals.

EFD requirements and interfacesFrossie Economou30 minsDiscuss https://project.lsst.org/groups/ccb/node/2470 and the path forward with starting to exercise DM interfaces to the EFD,
Test Specs in jira/MD with Unknown User (gcomoretto)

Wil O'Mullane

30 minsLook at JIRA test director and how SE are using it. Also how might we use this to make our test specs, test reports  and traceability.    There is also some terminology difference to be covered (test spec = test run etc. ) . 
Final Close out of Compression WG with Unknown User (gruendl)@womullan20 minsAssuming those interested will have looked at the report this would be highlights and potential work to be done in the future.
The DMS Product TreeUnknown User (gcomoretto)20 minsBased on the different documents available (and in preparation), I would like to share some thoughts, maybe a proposal, for a categorization of the different products that can be identified (at the moment) inside DMS.


Attached Documents

  File Modified
PDF File KTL Metadata DMLT 2018-05.pdf May 21, 2018 by Kian-Tat Lim
PDF File KTL Processes DMLT 2018-05.pdf May 22, 2018 by Kian-Tat Lim
PDF File Kantor Networks and Base.pdf May 22, 2018 by Jeff Kantor
PDF File LossyCompress_WG_20180522.pdf May 22, 2018 by gruendl
PDF File 2018-05-22 — DMLT QA WG.pdf May 22, 2018 by John Swinbank
Microsoft Powerpoint Presentation OpsRehearsalsv2.pptx May 23, 2018 by Margaret Gelman
JPEG File DMLT-UW-2018.jpg May 23, 2018 by Wil O'Mullane
PDF File SST Plans F18 .pdf May 24, 2018 by Leanne Guy
PDF File 2018-05-24 — AP — F18.pdf May 24, 2018 by John Swinbank
PDF File KTL Arch Epics DMLT 2018-05.pdf May 24, 2018 by Kian-Tat Lim
PDF File DMLT-RFC-441_201805-v3.pdf May 24, 2018 by Gregory Dubois-Felsmann
Microsoft Powerpoint Presentation SUIT_F18plan201805.pptx May 24, 2018 by xiuqin
PDF File DAX_DMLT_05_2018.pdf May 24, 2018 by Vaikunth Thukral
Microsoft Powerpoint Presentation DMLT_F2FMay2018_LDF.pptx May 24, 2018 by Margaret Gelman
Microsoft Powerpoint Presentation F18 Supertask Update.pptx May 28, 2018 by Fritz Mueller
Microsoft Powerpoint Presentation GCM_Product-Tree.pptx May 29, 2018 by gcomoretto
Microsoft Powerpoint Presentation GCM_Releases.pptx May 29, 2018 by gcomoretto
Microsoft Powerpoint Presentation GCM_TestApproach.pptx May 29, 2018 by gcomoretto