Implemented under and
milestones PR #38
These are all intended as Level 2 milestones for Data Management - i.e., to be included in LDM-503.
I've left room for adding lists of requirements that will be satisfied by Milestone-2 and Milestone-3, once we've agreed to this framework (or some revision of it).
The milestone descriptions are not meant to be comprehensive lists of every single feature remaining to be developed, just the highlights.
Implemented under |
(we are ready - May 2021 - to run a formal test and claim)
The Rubin Science Platform is ready to support Data Preview 0.1 as deployed on the IDF. The RSP capabilities are largely those demonstrated under previous milestones, but ported to the IDF and with deployment and authentication/authorization infrastructure improvements that support the scale of DP0.1 usage. The Notebook Aspect provides access to the Rubin Science Pipelines stack and to data exposed in a cloud-deployed Gen3 Butler repository; the API Aspect provides TAP queries against a Qserv deployment of catalog tables, but not image metadata; and the Portal Aspect provides a UI for visual and ADQL construction of queries against those tables. Image and image metadata access services in the API and Portal Aspects are not included.
Predecessors: none that are not already completed
Test Plan: LVV-P79 (prepared under ); report published as DMTR-301
Completion Ticket:
(set to some date in late 2021 / early 2022 in time for DP0.2)
The Rubin Science Platform is ready to support Data Preview 0.2 as deployed on the IDF. The RSP serves an end-to-end Rubin-Gen3-processed dataset with a substantial set of image and SDM-ified catalog data products. Beyond the capabilities provided under Milestone-1 (LDM-503-14a), this deployment provides: initial IVOA-oriented image metadata and image services in the API Aspect, specifically an ObsTAP service and a basic SODA-compatible image cutout service; Portal Aspect support for accessing these services and browsing images from the released dataset; and access to a User File Workspace from all three Aspects.
Compare to existing text of LDM-503-14a:
“Take the output of a DRP-like, BG3-based, SDM-ified data production and, without a great deal of manual labor, make the resulting image and catalog data available to users through the aspects of the LSP. This includes the generation of image metadata table content for the LSP.”
Predecessors: LDM-503-14a, DM-SQRE-4 (Notebook service ready for V&V), DM-SQRE-6 (SODA), DM-SQRE-7 (ObsTAP/SIAv2), DM-SUIT-5 (search and display processed HSC data)
Test Plan: (to be prepared under )
Completion Ticket:
Additional LSP-related requirements that will be satisfied:
(set to some appropriate date near the start of full-LSSTCam operations)
The Rubin Science Platform is ready to support Science Verification activities in the final phase of Rubin Observatory commissioning. In additional to the capabilities demonstrated in Milestone-2 (LDM-503-RSPa), the RSP will: be scalable to the user load required to support Science Verification; provide access to a regularly updating stream of newly-acquired LSSTCam data{dep-1}; provide access to next-to-data computing and user batch computing{dep-2} to support intensive data analysis; provide access to a User Database Workspace{dep-3}; provide a means for integrating with a TBD focal-plane-scale visualization tool{dep-4}; provide capabilities in all three Aspects for following links between related datasets{dep-5} and exploiting provenance information{dep-6}; provide access to the EFD for recently acquired data, extending the capability demonstrated for LDM-503-EFDc; and provide substantial user-facing documentation, with links between the data and documentation.
This milestone encompasses approximately the set of features expected to be established in time for the Rubin Observatory Operational Readiness Review. One difference is that for ORR the ability to serve all the planned image and catalog data types from production survey operations must be demonstrated.
Predecessors: LDM-503-RSPa, LDM-503-EFDc, DM-DAX-9 (Provenance system), DM-SQRE-5 (Notebook service ready for general science users), plus {dep}s from above:
{dep-1} - Existence of internal access to a regularly-updating collection of newly-acquired LSSTCam data, including as a Gen3 repository and as a source of ObsCore-compatible image metadata (Data Facility task)
{dep-2} - Availability of a batch computing service open to Science Platform users (Data Facility task)
{dep-3} - Availability of user database services via both a Notebook-accessible interface and an API Aspect interface (i.e., TAP).
{dep-4} - Availability of a full-focal-plane visualization tool, whether inside the LSP Portal Aspect or not, including regular ingest of newly acquired data (e.g., if preprocessing is needed) (possible in-kind contribution)
{dep-5} - Availability of DataLink services in the LSP, including decoration of TAP query results, and corresponding micro-services (intra-RSP task)
{dep-6} - Availability of provenance-access services in the LSP (if this is not already covered by the definition of DM-DAX-9)
Test Plan: (to be prepared under )
Completion Ticket:
Additional LSP-related requirements to be satisfied: