Note: there are many questions called out below as needing answers. We will try to pre-discuss a number of them and present a proposed answer at the workshop for confirmation. We don't expect to discuss everything "live" at length.
We are currently planning for relatively long breaks, to encourage spontaneous followup discussions, and we are not planning "working lunches" in a meeting of this many days - we hope stretching our legs will help us think better!
770 S Wilson Ave., Pasadena
Review basic documentation (not fully approved after the DM review)
Mostly done but not folded in yet? Pre-meeting preparation
Identify work required on performance requirements - expected to be needed to drive testing
Identify any missing items and associated groups
Continued discussion on requirements
Summary reports from the Aspect groups
(1) Current development, test, and deployment plan ("d.t.d.") for SUIT / Portal Aspect
(2) Current d.t.d. plan for JupyterHub/Lab / Jellybean / Notebook Aspect
(3) Current d.t.d. plan for Data Access / API Aspect
(These are meant to be "as of start of workshop" and it is definitely OK for them to be modified by what is discussed / decided at the workshop. All reporters are encouraged to comment on areas where they think there is a lack of clarity on scope or on assignment of scope to a group. All reporters are encouraged to comment on any substantial technology choices or design decisions that remain to be made.)
LSP design discussion: connection among three aspects
workspace interfaces in each Aspect
"file workspace", i.e., VOSpace/WebDAV (confirm development responsibilities)
"database workspace" / "MyDB"
How are these accessed in each Aspect? E.g., is the VOSpace workspace mounted as a filesystem visible in the Notebook Aspect?
cross-aspect connections for queries:
by user (i.e., by being able to ask the API aspect for "my previously run queries")
by query ID returned from an asynchronous query request
UI actions for transferring queries between Portal and Notebook
use of the workspace for query results
Lunch on our own (campus cafeteria or Lake Ave. restaurants)
Authentication and authorization
login, token passing across Aspects
granularity of access rights
This is about the "toolkit" we give to the operations team, not about our making policy decisions in this workshop.
Should we implement separate rights for each Aspect (e.g., "may use the Notebook Aspect")? This may be useful for managing abusive users.
Should we plan for the possibility of data rights separated by Data Release? (E.g., if you had rights to DR3 because you were a grad student of someone with data rights, and then you move back to your home country, can you retain DR3 access for some time, perhaps while you finish a paper, without being granted DR4 rights?)
What is the granularity of control we give to users to manage access to user-created ("Level 3") data?
what is the programming model for testing whether a user has a specific right? Just by attempting the operation desired, or is a "pre-verification" possible?
What is the API and/or UI for self-service group management? (E.g., for users to create collaborations that can have rights?)
Does NCSA provide a flexible profile service (e.g. a key-value store for each user)?
LSP architecture and deployment
Complete discussion of the role of each of the five (six?) LSP instances
rough feature-deployment schedule and cycle for all the LSP instances
when / where is the first integration of the Notebook Aspect with the others (on PDAC?)
Plan for completion of documentation of full LSP deployment architecture
Define all sub-components and identify internal interfaces
Identify specific architectural issues for follow-up
API Design and AAIM discussion
Authentication and security for all exposed APIs
API for workspace(VOSpace?) access, including security token pass
API for "next to DB" data processing
How does each Aspect run code as the logged-in user? (I.e., how do the Aspect do "setuid-like" things when needed?) A/k/a "userid mapping". Do all users actually have separate NCSA Unix/NIS-type identities?
Programming model for 3rd-party tool developers (e.g., TOPCAT) (time permitting)
When will the computer room be ready? What are the plans for installation of hardware?
What are the first things its users will expect to be able to do?
Database and DAX planning for the CC deployment
Access to EFD data in the Commissioning Cluster, relationship with EFD-reformatter service
Is access to the "raw" EFD required? If so, how?
What is the DAX interface to the EFD? Where does the table and column metadata come from?
Cross-aspect access to computing resources (e.g., for Portal extensions that need to run Python code as the user)
Lunch on our own (campus cafeteria or Lake Ave. restaurants)
LSP Data Model
How do Science Platform outputs (e.g., afw.table FITS files) become data visible in the LSP?
Is this round-trippable? What is the Python API for access to the DPDD data products in the Notebook Aspect? Is round-tripping only possible in the Notebook Aspect when run inside a DAC?
What is the life cycle of data model metadata (e.g., column metadata like units, UCDs, VO-DML descriptions)? Is it created "at birth" in the Science Pipelines code? Or at ingest? How do database columns get their released names? Are these all in the table-creation code in afw?
Getting issue details...STATUS
- when will we start getting deliberate prototypes of the actual DPDD data products?
Butlers in the Notebook Aspect and in Portal extensions
How do users get Butlers that provide access to the released data products?
How does a Butler user follow the release of new Level 1 / nightly data?
Support for "older releases" (see LCR-908) (probably will not get to this this time)
Detailed DAX API discussion
What is the full envelope of VO interfaces we'll provide? What interfaces do we need for which no satisfactory VO interface exists?
ADQL support level. Any limitations?
VOSpace. Any limitations? Also WebDAV? Third-party transfers?
extra API or special DB access needed by portal (besides metaServ, DBServ, and ImgServ), for example certain data exploration flow sequence,
K8s and notebook deployment
Further discussion of deployment
K8s, docker, more details
What is the batch / parallel computing model that will be exposed to Notebook users?
Next-to-DB processing architecture
Meta data for data holdings (continued from Data Model session)
Detailed schedule and planning
Role of LSP in operations rehearsals
Need for simulations of real LSST datasets
Successively approach real data model
Generally review datasets to be handled (HSC, ZTF(?), Gaia, LSST-CatSim, LSST-PhoSim)
for PDAC, SV
Operations concept for transition of a data release from production through science validation to public release
How does the SV environment for DR(N+1) get access to the released data for DR(N)?
Table data formats:
Support for very large tabular results
VOTable usage - do we want the header but send the body a different way?