LDM-482
The Prototype Data Access Center (PDAC) is an LSST Data Management (DM) construction-phase engineering exercise to stand up an early prototype LSST Data Access Center service, with the ability to keep and serve one or more LSST-like datasets (e.g., simulated LSST data, LSST-reprocessed SDSS Stripe 82, single-epoch WISE catalog, etc.).
Its purposes are:
- Enable an end-to-end integration of (at least) the Data Access, SUIT, and Middleware/Infrastructure components of DM into a cooperating system.
- Enable operational and scaling tests of the above.
- Expose the Data Access (e.g., database query) and SUIT components to a limited science user community for early feedback on the design and implementation.
In order to maximize the value of this exercise, and the opportunity for independent, external feedback, LSST Data Management wishes to make the PDAC available to a small number of non-project-team users.
PDAC access may be granted to:
- Project team members who specifically request it and receive approval from DM management, after consultation with the teams building and operating the components.
- Astronomers with LSST data rights not employed by the project ("external users"), who submit an application substantively describing investigations they wish to perform, including a description of how they plan to provide technical and usability feedback to the project, and expressing their understanding that the PDAC is not a production system. The PDAC will initially only be able to accommodate a small number of external users.
DM management will be responsible for the approval of all PDAC access requests. Requests by external users will be approved based on an assessment of the match between the proposed investigation and the available PDAC capabilities and resources, the likelihood that the proposed work will result in useful and actionable feedback to DM, and the requestor's preparedness for an efficient and timely execution of the proposed work. External users will submit requests for access to the PDAC semi-annually; user access to the PDAC will not automatically be continued past 6 months unless users submit a new or continuation request. The LSST DM expects to make use of the existing public LSST mailing lists as well as communication tools such as community.lsst.org in order to inform the community of this opportunity.
The PDAC service is intended to support the development of LSST DM and is not intended to offer the service levels of a full production astronomical archive. As such, in order to avoid occupying project personnel with a large burden of low-level user support issues, the number of PDAC users needs to remain small and limited to people with suitable levels of skill and interest in contributing to the successful development of LSST DM. All PDAC users will need to understand that the service is not intended to be, nor can the DM team support, a true 24x7 service. As an engineering test facility, users should expect that the PDAC team will bring the system up and down periodically, deploy bug fixes and new features, remove unsuccessful features, and temporarily restrict access to the system for internal testing (e.g., scaling exercises).
The PDAC is a prototype service designed to aid in the design and construction of the final LSST operational system. Therefore, as DM construction proceeds, the datasets and capabilities of the PDAC may change without warning. It is expected that additional datasets may be added, while other datasets may be dropped or superseded, depending on the needs of the LSST Project; however, as the chosen datasets are of substantial value, the LSST Project will attempt to preserve access to these datasets as long as resources permit.
The PDAC in its 2016-2017 form will have access limited to people with logins on an identity management system provided by NCSA. Approved external users will be required to formally agree to an Acceptable Use Policy (AUP) covering the usual IT security issues as well as their commitment to the project's goals. As the hosting organization for the PDAC, NCSA has a particular role in determining the limits on the number of users supported and the resources provided. The AUP will be consistent with applicable NCSA security policies. The LSST project reserves the right to withdraw access at any time for a user who is not actively participating in the evaluation of the PDAC, who is not behaving in a constructive manner, or who has violated the AUP.
Any publications resulting from the use of the PDAC will be subject to the LSST Publications Policy.
Nothing in this Policy applies to access or use of data taken as part of Commissioning; the policy for those will be determined at a later date.
14 Comments
Mario Juric
Gregory Dubois-Felsmann, Unknown User (ciardi), thanks this looks excellent.
I have three modifications I'd propose:
Unknown User (ciardi)
Mario Juric (Gregory Dubois-Felsmann)
Thanks for the comments ...
I like your suggested changes - it helps alleviate the concerns we (all) have about too many users and not being able to support them.
Here are my brief comments on your comments (but I don't think they will change your changes)
Mario Juric
Thanks! Gregory Dubois-Felsmann or Unknown User (ciardi) – could you generate a next version, based on the feedback?
Gregory Dubois-Felsmann
We have edited the document with Mario Juric's comments in mind and adopting a more formal tone.
In response to concerns about getting too much interest, we have deleted all the material relating to the possibility of making the data available outside the Science Collaborations. It should be noted that there was discussion at the May 2016 DMLT about whether the outside community might be concerned about the fairness of the process in the event that the existence of the PDAC becomes known, e.g., through someone making an interesting discovery using the data and services we provide.
Mario Juric
I made all publications resulting from PDAC use subject to LSST pub policy.
I think it's important for us to a) give an opportunity to our developers to take credit in enabling any science coming out of their work, and b) have some control (through the pub board review process) that the users are correctly using the services provided (and that the results are sound).
Donald Petravick
We need a "no warranty" kind of statement if the functionality includes retaining user data.
Make sure that there is a clear warning – the system is a prototype, and the structure of supporting data may change without warning – for example,
queries can break due to underlying changes in database.
Unknown User (ciardi)
Donald Petravick (Gregory Dubois-Felsmann) ... Thanks Don, I made a change to the third to last paragraph - that paragraph was intended to be the no warranty paragraph, but I made it stronger.
Gregory Dubois-Felsmann
Donald Petravick, I agree with that specific point; once we provide the workspace capabilities in 2017, the issue of user-created data will become particularly important. I think we want to walk a careful line and be able to say something like "we understand you are trying to do real science with this system and we will treat your work with respect, but we cannot make firm commitments about availability and data integrity".
What Unknown User (ciardi) and I wrote above was in response to Mario's request for a policy focused on how to grant access. It's not meant to be the full policy for the management of the system, or the document that users actually sign. That will require another round of careful crafting.
Mario Juric
I made some modification in response to e-mail feedback from Beth Willman; she'll take a look at this draft, and possibly also run it by Steve K.
Beth Willman
So glad to see this plan developed, and policy written. Looks great. Just one text suggestion and two small questions:
Discussed the contents with Steve; he saw no issues.
Mario Juric
Thanks Beth Willman!!
Regarding your comments:
Thanks! I'll take this through the rest of the DM approval process & making sure all of our internal concerns are addressed, in preparation for standing up the PDAC later this year.
Gregory Dubois-Felsmann
This is now under review as RFC-208 - Getting issue details... STATUS .
Gregory Dubois-Felsmann
Inline comments addressed as requested by Kian-Tat Lim.
Gregory Dubois-Felsmann
Exported to PDF and saved as LDM-482.