Defining and exercising the full set of steps required to go from raw data to end user products and their service to users is important for understanding where the gaps are in the operational and software systems and how to estimate key performance metrics.
End-to-end testing in 2017 will extend to include both additional Science Platform functionality and a new processing of precursor data (the first HSC public data release) with the LSST stack.
- DRP group plans (as requested at 2017-04-03 DMLT meeting).
- SUIT's main goal is to access a data set as close as possible to the final LSST data set. We want to learn all the possible connections among all the tables and images so we can design and implement the system to provide a UI that aids users more effectively to explore and analyze the data. This is closely related to - RFC-243Getting issue details... STATUS , but on a larger scale.
- Bi-weekly Reprocessing of the HSC RC dataset
- AP “minimum viable system”
- S17B HSC PDR1 reprocessing
- SUIT continues PDAC development, adding the WISE data, further exercising the DAX
imgservAPIs, and taking advantage of
metaservonce it becomes available. Please see LSP Integration and Prototype Deployment for more details.
End-to-end testing in Fall 2016 and into early 2017 is based around the integration of database services (Qserv and others), DAX services above them (
imgserv, and later
metaserv), an initial version of the SUIT Portal components (services and pages), and the NCSA-provided integration-cluster infrastructure. The data used for this testing are previously-processed datasets (the LSST D&D-project's Summer 2013 processing of SDSS Stripe 82, and the IPAC processing of the WISE mission data) - this round of testing does not include current LSST-stack data processing.
Bremerton (August 2015)
Initial prototyping will be done at the Bremerton meeting in August. The aim is to do as complete an end to end test as we can manage although it does not have to be completely automated; stages can have manual intervention. The aim it to be able to determine the areas where we are most lacking in terms of a full end-to-end test (a full test from simulated OCS/Camera to Alert Production is defined in - DLP-584Getting issue details... STATUS ) and to make initial estimates for our Key Performance Metrics.
Key Performance Metrics