You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 47 Next »

Final DM Agenda

BlueJeans links for plenaries and breakouts are listed here: https://project.lsst.org/meetings/jtm2016/node/82, and copied below:

SANTA CRUZ
185348096 (link is external)

NATURAL BRIDGES
151507533 (link is external)

NEW BRIGHTON
619043751 (link is external)

RIO DEL MAR
919820913 (link is external)

Notes From Discussions

DM JTM 2016-02 Meeting Notes

Agenda

SessionTopic(s)

Tuesday 1

9:00 AM - 10:30 AM

9:00-9:30 Introduction and DM transition (New Brighton)

Non-Middleware Status:

  • 9:30-9:45 DRP
  • 9:45-10:00 AP
  • 10:00-10:15 SUI/T
  • 10:15-10:30 SQuaRE

(Note: DonP, MargaretG, PaulW excused to go to Chilean IT Infrastructure session; remote participation from NCSA expected.)

Tuesday 2

11:00 AM - 12:30 PM

11:00-11:45 Level 1 Process Control Design walkthrough: (New Brighton)
  • Interfaces, components, functions, interactions
  • What we get from the observatory, when, and how
  • Where the science pipelines execute, in what context, and with what services

11:45-12:30 Data Access status and walkthrough:

  • Butler
  • Qserv
  • DAX Web services
  • Global metadata service
  • Provenance

Tuesday 3

1:30 PM - 3:00 PM

Three pairwise discussions in separate rooms during each session, switching partners in the middle.  Some suggested topics are listed, but each pair of teams should develop their own.  The goals are to 1) make decisions on issues if possible or at least document what data is needed to make a decision, 2) agree on who is doing what, and 3) get to know team members.  These are not meant to be the only opportunity to have such discussions; they are meant to facilitate existing conversations and kick off new conversations that should happen on an ongoing basis.
1:30-2:15 Round-Robin 1:
  • Science Pipelines and SQuaRE (Natural Bridges)
    • Immediate and longer-term QA needs
  • Process Control and Data Access (New Brighton)
    • Provenance capture
  • SUI/T and Architecture (Rio Del Mar)
    • Workspace, VO, cross-DAC, other issues

2:15-3:00 Round-Robin 2:

  • Science Pipelines and Architecture (Natural Bridges)
  • Process Control and SQuaRE (Rio Del Mar)
    • Large-scale integration testing
  • SUI/T and Data Access (New Brighton)
    • DAX interfaces, schemas and metadata, etc.

Tuesday 4

3:30 PM - 5:00 PM

3:00-3:45 Round-Robin 3:

  • Science Pipelines and Process Control (Natural Bridges)
    • Workflow, inter-process communication, threading needs
  • Architecture and Data Access (New Brighton)
  • SUI/T and SQuaRE (Rio Del Mar)
    • Visualization

3:45-4:30 Round-Robin 4:

  • Science Pipelines and SUi/T (Natural Bridges)
    • Visualization
  • SQuaRE and Data Access (New Brighton)
    • QA schema, DAX/Butler for QA tool access, global metadata
  • Architecture and Process Control (Rio Del Mar)

Note: There is an LSST breakout on Authentication/Authorization for use in Data Access Policy implementation in parallel with this session. Purpose is to convey what is feasible technically, what policy implications that has, and state of current prototype. Cowell Room Jeff Kantor, Donald Petravick

Wednesday 1

9:00 AM - 10:30 AM

9:00-9:45 Round-Robin 5:

  • Science Pipelines and Data Access (Santa Cruz)
    • Uses of database, archive, Butler
  • SQuaRE and Architecture (Rio Del Mar)
  • SUI/T and Process Control (New Brighton)
    • Workspace and L3 processing invocation

9:45-10:30 Round-Robin Conclusion (6 x 7 min): conclusions and future work (New Brighton)

  • This is a DM-wide plenary to hear briefly from each team what was accomplished and where things will go.
  • Note that ideally we shouldn't hear about specific items from both sides – part of the agreements made should be who will report on a given item in this session.

Wednesday 2

11:00 AM - 12:30 PM

DM-wide decision-making: (New Brighton)
  • Data and file formats, Unicode
  • Naming
  • Documentation
  • VO

In parallel with the above, Networks Session. Note, the Networks Session will continue into lunch. RIO DEL MAR 919820913 (link is external) (Jeff Kantor)

Wednesday 3

1:30 PM - 3:00 PM

Closing plenary

Wednesday afternoon

3:30 PM - 5:00 PM

After the formal sessions, meeting with Camera DAQ, NCSA, Gregory about interface (New Brighton)

 

Original Proposed Sessions

Session Description and GoalPlenary or Required ParticipantsLengthRequester
Authentication and Authorization: convey what is feasible technically, what policy implications that has, and state of current prototypePlenary??James Basney
OCS Middleware: further refine ICDs"Principal System Software Interactions" Group4 hr?Gregory Dubois-Felsmann
Engineering and Facilities Database: define responsibilities and technical details of replication of "verbatim" copies of EFDDave Mills1 hr?Donald Petravick
Visualization: work with Camera team on improving current visualization systemUnknown User (jjt) Gregory Dubois-Felsmann Tony Johnson Stuart Marshall Tatiana Goldina Trey Roby2 hr?Unknown User (xiuqin)
Wavefront estimation: The T&S group have developed a wavefront estimation code which we plan to incorporate into the DM stack as part of 02C.04.03 (PSF estimation). Discuss its current state, what work remains to be done, and how best to proceed with integration.Sandrine Thomas, Unknown User (schandra)2 hr?John Swinbank
End to End Computing Support for Observing: walk though of capabilities/expectations on 1) "quick look" capabilities provides by the T&S project, 2) uses of the commissioning cluster, and 3) L1 processing system.TBD S&T people, someone to represent observing staff2hr?Donald Petravick
Networks: Review current contracts and schedules, recent LSE-78 updates, Network Operations and Management Plan, Network End-to-End Test plan.

Paul Wefel

Unknown User (rlambert)

Chip Cox (only available after 11am on 2/24)

Other FIU people remotely (need videocon support)  Julio Ibarra, Jeronimo Bezerra , and Heidi Morgan

Optionally Camera people who are in town for JTM (Mike Huffer, Tony Johnson),

2.5 hrJeff Kantor
Virtual Observatory: Talk about current strategies and whether we should take a more focused approach on standards involvement for Summer16.John Swinbank, Unknown User (ciardi), Mario Juric, Jacek Becla plus anyone else interested2 hrTim Jenness
Unicode: I would like to re-assess the lack of requirement for using Unicode in LSST. In the 2020s (and now) unicode is an accepted part of life and it seems strange that we will be unable to include names and math symbols in observing logs and processing logs.OCS representative, SQuaRE representative, John Swinbank, Jacek Becla2 hrTim Jenness

Python3 Migration: I would like to start planning the python3 migration of the DM software so it can be integrated into the Summer16 development plan.

John Swinbank, Simon Krughoff, Jonathan Sick2 hrTim Jenness

Documentation & Technical Communication Tools Spearheaded by DM. This session will review the new doc and web publishing work by DM, with the goals of: seeking adoption by other groups, coordinating with the project communications, and setting goals for Summer16 development. Topics will be:

  1. community.lsst.org - review status of the forum and the Mailbot service to integrate the forum with our existing mailing lists. Can more groups use the forum?
  2. Technical Notes - We've launched a new Technical Note platform to distribute web-native reports that capture information not applicable to change-controlled docs in Docushare or academic papers, while also being citeable in the astronomical literature (unlike Confluence pages or Docushare docs). I'll demonstrate the platform (Sphinx, GitHub, readthedocs.org, Zenodo), and how records are being listed in NASA/SAO ADS.
  3. docs.lsst.io - I'll talk about the new documentation landing page we're crafting to make all of DM's artifacts discoverable (software documentation, developer guides, technical notes, design documents, science papers, conference posters/talks/videos). The site will be automatically built from data in repositories such as Zenodo and ADS, and will integrate with the LSST DM bibliography being curated by Tim Jenness.
  4. Sphinx-based software documentation. I'll review the status of DM's new Sphinx-based documentation tooling to publish a new Developer Guide and docs for Science Pipelines, Qserv and Firefly. We're building a clone of ReadTheDocs.org to continuously deploy docs from our Eups/Scons-based software projects. I'd like to have an open discussion of what our priorities for software documentation should be in Summer16.
Anyone in other groups working on technical communications and software documentation. Members of DM are especially invited to give feedback on S16 planning. Of potential interest to: Beth Willman, Unknown User (suzanne), Zeljko Ivezic, Unknown User (schandra), Peter Joachim1+ hrJonathan Sick
Naming: This may sound like a tongue in cheek request, but it would be good if we could spend a bit of time trying to come up with a name for the software that DM is writing. We have Qserv, Firefly, SUIT but we don't have anything for the science pipelines. This makes it hard to develop branding.Anybody who would have an opinion.0.5 hrTim Jenness
Working session with SQauRE team on visualizationTatiana Goldina Angelo Fausti2 hrsUnknown User (xiuqin)
Alert Production interactions with DatabaseSimon Krughoff, others from UW?, Andy Salnikov, Brian Van Klaveren, Jacek Becla2 hrsJacek Becla
Alert Production interactions with Image and File ArchiveSimon Krughoff, others from UW?, Andy Salnikov, Brian Van Klaveren, Jacek Becla, John Gates. Serge Monkewitz2 hrsJacek Becla
L2 interactions with Image and File ArchiveJohn Swinbank, Jim Bosch, others from UW?, Andy Salnikov, Brian Van Klaveren, Jacek Becla, John Gates2 hrsJacek Becla
SUI interactions with image and File ArchiveUnknown User (xiuqin), Trey Roby, others from IPAC?, Andy Salnikov, Brian Van Klaveren, Jacek Becla, John Gates2 hrsJacek Becla
Integrate HSC pipeline with ProvenancePaul Price (not attending), others from Princeton?, Jacek Becla1 hrsJacek Becla
SuperTask + ProvenanceGregory Dubois-Felsmann, Jacek Becla, Unknown User (npease)1 hrsJacek Becla
Docker + QservJoshua Hoblitt, Fabrice Jammes, Fritz Mueller, Vaikunth Thukral2 hrsJacek Becla
Getting Jeff T from Oxford up to speed with QservFabrice Jammes, Vaikunth Thukral4 hrsJacek Becla
L2 PipelinesUnknown User (kelsey), John Swinbank, Jim Bosch, Robert Lupton1 hrsJacek Becla
CI for QservFrossie Economou, Joshua Hoblitt, Fritz Mueller, Fabrice Jammes2 hrsJacek Becla
Butler: review API and execution sequence plans for DM-4181 "Butler: add support for skymap based dataIds" and DM-3472 "Implement Basic spatial lookups for the Butler"Unknown User (npease), Serge Monkewitz, few people from apps team?1 hrUnknown User (npease)

Butler: review multiple-repository design and WIP

Gregory Dubois-Felsmann, others from IPAC?, Unknown User (npease)1 hrUnknown User (npease)
Butler: requirements & design spike for DM-4168 Data Repository selection based on versionSimon Krughoff, Russell Owen, Unknown User (npease)1 hrUnknown User (npease)
Butler: requirements & design spike for DM-4180 provide API so that a task can define the output dataset typeSimon Krughoff, others from UW?, Unknown User (npease)1 hrUnknown User (npease)
Butler: review design OR requirements & design spike for DM-4555
Add initial butler support for remote GET		 Frossie Economou, others from Tucson?,  Brian Van Klaveren,  Unknown User (npease) 1 hrUnknown User (npease)
Butler: review current thinking and and/or design spike for storage, persistence, and python type selectionFabio Hernandez, Unknown User (npease)1 hrUnknown User (npease)
DCR: discuss the status of tests involving Nate Lust's suggested algorithm. Figure out at what level Nate will be involved in further work here.Nate Lust, Unknown User (vpk24), Simon Krughoff, Ian Sullivan(question) David Reiss(question)1 hrNate Lust
Integration of the Calibration Products Pipeline plan with the overall calibration effort.John Swinbank, Robert Lupton, Patrick Ingraham, Merlin Fisher-Levine, other calibration team members(?)2 hrJohn Swinbank

Data and File Formats: discussion of the data and file formats and laying out a plan/timeline for a decision process of which formats are required, which formats we want to support, which formats are defaults. This discussion centers around both images and tables and includes output formats as well as 'internal' formats (like python objects).


Jacek Becla, Tim Jenness, Gregory Dubois-Felsmann, Mario Juric, John Swinbank, Kian-Tat Lim, Unknown User (xiuqin), Simon Krughoff - others ...1 hrUnknown User (ciardi)
  • No labels