Topics for the LSP design and engineering workshop, 4-8 December 2017, IPAC


1. Reviewing the basic documentation

   * Requirements
     - Review the draft top level & Portal Aspect requirements from the summer
     - Incorporate Notebook Aspect requirements
     - Incorporate API Aspect requirements
     (drafts of Notebook and API Aspect requirements should have been prepared in advance)
     - Discussion of what performance requirements may still be missing / needed to drive testing

   * Verification plans
     - Review all LSP-related L2 and L3 milestones
     - Review any available test definitions associated with milestones or with specific
       LSP requirements

   * Design
     - Comments on the existing design document
     - Agreeing on a framework for detailed design documentation


2. Reports on the Aspects

   The following reports are meant to be "as of start of workshop" and it is definitely OK for them to be modified by what is discussed / decided at the workshop.
   * Development, test, and deployment plan for JupyterHub/Lab / Notebook Aspect
   * D.T.D. plan for Firefly / SUIT / Portal Aspect
   * D.T.D. plan for Qserv / DAX services / API Aspect

   All reporters are encouraged to comment on areas where they think there is a lack of clarity on scope or on assignment of scope to a group.

   All reporters are encouraged to comment on any substantial technology choices or design decisions that remain to be made.


3. General planning

   * Review the "five deployments" (integration/PDAC, science validation, comm'g cluster, US DAC, Chilean DAC) and associated high-level schedule
     - It is a legitimate and expected output of the workshop to refine that schedule
   * Generally review datasets to be handled (HSC, Gaia, LSST-CatSim, LSST-PhoSim)
   * High-level PDAC LSP deployment planning: datasets, incorporation of Notebook Aspect, incorporation of additional high-level features to each Aspect (e.g., user database support)
     - Hardware plans for integration/PDAC environment
     - Migration to Kubernetes
     - Release and user testing cycle; distinction between verification and user testing
   * Science Validation LSP deployment planning: schedule, expectations, datasets, etc.
   * Commissioning Cluster LSP deployment planning: primarily confirming the high level schedule
     - When will the computer room be ready?  What are the plans for installation of hardware?
     - What are the first things its users will expect to be able to do?
     - Database and DAX planning for the CC deployment
   * (briefly) Final DAC LSP deployment planning: basic dates
   * Role of LSP in operations rehearsals
     - Use of simulated LSST data in operations rehearsals


4. High-level LSP architecture and design issues

   * Butler interface to released data in the DAC LSP deployments
     - Level 1 - how is the inflow of new data represented to the Notebook Aspect user?
     - Level 2 - how does the user get a Butler for a specific data release?
   * Operations concept for transition of a data release from production through science validation to public release and how this plays out in the LSP instances
     - How does the SV environment for DR(N+1) get access to the released data for DR(N)?
   * DPDD data model definition
     - Understanding the life cycle of data products from creation (e.g., as afw.table) to user retrieval from LSP Aspects (e.g., tables via TAP from database)
     - Includes semantic data modeling
     - When do we start getting deliberate prototypes of the actual DPDD data products?
   * Access to EFD data in the Commissioning Cluster, relationship with EFD-reformatter service
     - How is the "raw" EFD accessed?
   * What is the full envelope of VO interfaces we'll provide?  What interfaces do we need for which no satisfactory VO interface exists?


5. Engineering

5a. Cross-aspect

   * Authentication and authorization (session requires additional NCSA attendance - presumably remote)
     - Recommended ways for Aspects to check for rights
   * Full LSP deployment architecture - define all sub-components and identify internal interfaces
     - Identify specific architectural issues for follow-up
   * Cross-aspect interfaces from the user perspective
     - Transfer of queries by user
     - Transfer of queries by query-ID and associated cut-paste, drag, other UI actions supporting cross-aspect workflows
   * Details of the use of Kubernetes to create LSP deployments
     - Understand this for each Aspect and for lower-level services on which they depend
   * Support of "old releases"


5b. Aspect-specific issues (selected ones that are not opaque)

   API / DB
   * What is the ADQL subset we'll support?  Why / what are the limiting constraints?
   * Handling of large result sets from queries
   * Workspace
     - (file workspace) VOSpace: confirm the choice and understand development responsibilities
     - (table workspace) User databases: define the near-term and final versions
   * Next-to-DB processing interfaces

   Notebook
   * What is the batch / parallel computing model that will be exposed to Notebook users?

   Portal
   * All-sky viewing / data browsing