Deploy the WISE primary mission Object-like and ForcedSource-like catalogs in PDAC, along with the associated image metadata tables, integrated with the existing Stripe 82 data to permit spatial joins in Qserv.
WISE primary mission (and AllWISE) catalog data have been loaded into Qserv, but are not yet supported in the Portal. Work remains to be done on loading the image metadata, supporting spatial joins between WISE and SDSS/S82, and a number of other details. The necessary storage space is available already in the PDAC ("integration cluster") hardware at NCSA.
Attempt to maintain backward compatibility with the 2013 Stripe 82 processed image data for as long as this does not produce significant costs for Science Pipelines.
At the moment there are no specific functional issues in PDAC arising from backward-compatibility breakage, but not all Science Pipelines functionality is available now when reading in calibrated images from the 2013-era stack.
Status
colour
Grey
title
Stretch
Science Pipelines (if needed), DB-DAX/SLAC (as it affects components like the cutout service)
PDAC Data, Portal
Deploy the WISE and NEOWISE Source-like catalogs in PDAC, when they become available for download from IPAC-IRSA.
As of all the primary-mission single-epoch Source-like tables were available for bulk download from IRSA.
Bulk download of the NEOWISE (reactivation mission) Source-like tables from IRSA is not yet available, but work on this is in progress (outside the IPAC LSST group). We show this as a stretch goal only because of this dependency - once the data become available this is actually an important thing to add to PDAC because of the ability to exercise Source-like semantics in the Portal and the database. The necessary storage space is available already in the PDAC ("integration cluster") hardware at NCSA.
Status
colour
Grey
title
Stretch
DB-DAX/SLAC (load data), SUIT/IPAC (support in Portal)
As of - IRSA has now done a significant fraction of the work required. Still working on proper packaging (e.g., including column metadata in the export files, computing checksums). No firm schedule, but completion during Summer 2017 seems likely.
Support access to the HSC public release input data as well as outputs of 2017 Science Pipelines processing of that data.
This may well include an additional “release” step to make outputs available in PDAC after they have been processed. The details of this remain to be worked out.
For SUIT, the main goal is to access a data set as close as possible to the final LSST data set: "We want to learn all the possible connections among all the tables and images so we can design and implement the system to provide a UI that aids users more effectively to explore and analyze the data."
Status
colour
Green
title
Normal
DB-DAX/SLAC and Science Pipelines for creating/ingesting data; SUIT/IPAC for developing a Portal more specific to LSST's expected data
Integrate the SUIT Portal application(s) with the NCSA Authentication & Authorization system prototype. Support user logins and begin to provide persistent Portal state across logins.
Deploy DAX-metaserv for all loaded datasets in PDAC.
Column metadata for the Summer 2013 SDSS Stripe 82 catalog outputs appears to have been lost sometime in the intervening years, It is in the process of being manually regenerated by the DB-DAX team.
WISE catalog metadata is available from IRSA (not yet including UCDs) and will be ingested by the DB-DAX group.
Priority "critical" because a lot of the Portal development depends on the availability of a metadata service.
Deploy on the integration cluster (i.e., PDAC) hardware a JupyterHub service, integrated with the NCSA A&A system, supporting the launch of JupyterLab-beta (and possibly traditional Jupyter) notebooks with their IPython kernel server processes running with the user’s NCSA identity and with read/write access to the NCSA LSST GPFS service, and with access to the PDAC DAX services.
Status
colour
Green
title
Normal
SQuaRE/Tucson and MW&Inf/NCSA, boundaries to be determined
SQuaRE work:
Jira
server
JIRA
serverId
9da94fb6-5771-303d-a785-1b6c5ab0f2d2
key
DM-10744
NCSA work:
Notebook
Deploy the ability for the IPython kernel process to come up in an environment with a pre-installed LSST stack.
Status
colour
Green
title
Normal
SQuaRE/Tucson
Jira
server
JIRA
serverId
9da94fb6-5771-303d-a785-1b6c5ab0f2d2
key
DM-9984
Notebook, Infrastructure
Ensure that DAX query services are available and readily usable in the PDAC Jupyter service.
This is primarily a matter for the deployment architecture, to ensure that the Jupyter back end IPython kernel processes have network access to the DAX services.
It may also include work on supporting users in exploiting those services from Python. What this means is TBD.
Status
colour
Green
title
Normal
Infrastructure/NCSA, DB-DAX/SLAC
APIs (SUIT)
Develop extensions to the afw.display interfaces to expose existing table-image overlay and related SUIT functionality through an LSST-aware interface.
Develop usable early versions of JupyterLab-flavored table and image visualization widgets.
The intent here is primarily to ensure that we understand and can work with the JupyterLab widget interface, which is different from the original Jupyter 3.0 widget model, and to do this in a way that delivers some basic functionality useful for developers.
Add the ability to create user tables through the DAX services.
If this goal is not realistic in 2017, a fallback would be to allow the creation of user databases in a more bare-bones, less integrated way. (E.g., as previously supported on the lsst-dev cluster.)
Status
colour
Grey
title
Stretch
for doing this the DAX way,
Status
colour
Green
title
Normal
for doing it the bare-bones way
DB-DAX/SLAC
Portal (first)
Begin cyber-security vetting of the PDAC / Science Platform components.
Note that the existing PDAC components have not yet been subjected to security review, and are therefore being kept behind an NCSA VPN. The goal is to bring several of these components out onto the public Internet. This requires both security review and the incorporation of authentication and authorization into the components that are exposed to the Internet.
Status
colour
Green
title
Normal
Infrastructure/NCSA for strategy/screening, development teams for responding to issues.
DM Documentation
Prepare a documentation package defining the Science Platform's expected capabilities, requirements, functional design, and deployment architecture.