LDM-482

The Prototype Data Access Center (PDAC) is an LSST Data Management (DM) construction-phase engineering exercise to stand up an early prototype LSST Data Access Center service, with the ability to keep and serve one or more LSST-like datasets (e.g., simulated LSST data, LSST-reprocessed SDSS Stripe 82, single-epoch WISE catalog, etc.).

Its purposes are:

  1. Enable an end-to-end integration of (at least) the Data Access, SUIT, and Middleware/Infrastructure components of DM into a cooperating system.
  2. Enable operational and scaling tests of the above.
  3. Expose the Data Access (e.g., database query) and SUIT components to a limited science user community for early feedback on the design and implementation.

In order to maximize the value of this exercise, and the opportunity for independent, external feedback, LSST Data Management wishes to make the PDAC available to a small number of non-project-team users.

PDAC access may be granted to:

  1. Project team members who specifically request it and receive approval from DM management, after consultation with the teams building and operating the components.
  2. Astronomers with LSST data rights not employed by the project ("external users"), who submit an application substantively describing investigations they wish to perform, including a description of how they plan to provide technical and usability feedback to the project, and expressing their understanding that the PDAC is not a production system. The PDAC will initially only be able to accommodate a small number of external users.

DM management will be responsible for the approval of all PDAC access requests. Requests by external users will be approved based on an assessment of the match between the proposed investigation and the available PDAC capabilities and resources, the likelihood that the proposed work will result in useful and actionable feedback to DM, and the requestor's preparedness for an efficient and timely execution of the proposed work. External users will submit requests for access to the PDAC semi-annually; user access to the PDAC will not automatically be continued past 6 months unless users submit a new or continuation request. The LSST DM expects to make use of the existing public LSST mailing lists as well as communication tools such as community.lsst.org in order to inform the community of this opportunity.

The PDAC service is intended to support the development of LSST DM and is not intended to offer the service levels of a full production astronomical archive. As such, in order to avoid occupying project personnel with a large burden of low-level user support issues, the number of PDAC users needs to remain small and limited to people with suitable levels of skill and interest in contributing to the successful development of LSST DM.  All PDAC users will need to understand that the service is not intended to be, nor can the DM team support, a true 24x7 service. As an engineering test facility, users should expect that the PDAC team will bring the system up and down periodically, deploy bug fixes and new features, remove unsuccessful features, and temporarily restrict access to the system for internal testing (e.g., scaling exercises).

The PDAC is a prototype service designed to aid in the design and construction of the final LSST operational system.  Therefore, as DM construction proceeds, the datasets and capabilities of the PDAC may change without warning. It is expected that additional datasets may be added, while other datasets may be dropped or superseded, depending on the needs of the LSST Project; however, as the chosen datasets are of substantial value, the LSST Project will attempt to preserve access to these datasets as long as resources permit.

The PDAC in its 2016-2017 form will have access limited to people with logins on an identity management system provided by NCSA. Approved external users will be required to formally agree to an Acceptable Use Policy (AUP) covering the usual IT security issues as well as their commitment to the project's goals. As the hosting organization for the PDAC, NCSA has a particular role in determining the limits on the number of users supported and the resources provided.  The AUP will be consistent with applicable NCSA security policies. The LSST project reserves the right to withdraw access at any time for a user who is not actively participating in the evaluation of the PDAC, who is not behaving in a constructive manner, or who has violated the AUP.

Any publications resulting from the use of the PDAC will be subject to the LSST Publications Policy.

Nothing in this Policy applies to access or use of data taken as part of Commissioning; the policy for those will be determined at a later date.

14 Comments

  1. Mario Juric

    Gregory Dubois-Felsmann, Unknown User (ciardi), thanks this looks excellent.

    I have three modifications I'd propose:

    • Instead of reviewing external users semi-anually, change that to "re-apply semi-annualy". I think that's better than having to decide to kick someone out (which never occurs).
    • I'd propose to drop the goal to broadly announce this (e.g., through AAS newsletter). I'm worry we're not ready for that level of exposure/involvement. A slow start may be better (e.g., maybe even going as far as only advertising on community.lsst.org, for a start – that is already a self-selected community of users)?
    • I'd propose we don't send this call out until we have the PDAC service established and running to our satisfaction, at least for a couple of weeks.

  2. Unknown User (ciardi)

    Mario Juric (Gregory Dubois-Felsmann)

    Thanks for the comments ...

    I like your suggested changes - it helps alleviate the concerns we (all) have about too many users and not being able to support them.  

    Here are my brief comments on your comments (but I don't think they will change your changes) 

    • I like the idea of re-applying in a semester-like fashion.  This is akin to observatories and we can (maybe) use this as a starting model for operations.  We should keep in mind that the applications should be short (we can define this later but 1 page science, 1 page tech) and re-application should be 'shorter' - just so this is not a giant pain for the users and us.
    • @gpf and I went back and forth on this.  I agree and this leads to next point 
    • I agree - though at the AHM meeting I would like us to announce our plans to the community - that this is coming ...

     

  3. Mario Juric

    Thanks! Gregory Dubois-Felsmann or Unknown User (ciardi) – could you generate a next version, based on the feedback?

  4. Gregory Dubois-Felsmann

    We have edited the document with Mario Juric's comments in mind and adopting a more formal tone. 

    In response to concerns about getting too much interest, we have deleted all the material relating to the possibility of making the data available outside the Science Collaborations.  It should be noted that there was discussion at the May 2016 DMLT about whether the outside community might be concerned about the fairness of the process in the event that the existence of the PDAC becomes known, e.g., through someone making an interesting discovery using the data and services we provide.

  5. Mario Juric

    I made all publications resulting from PDAC use subject to LSST pub policy.

    I think it's important for us to a) give an opportunity to our developers to take credit in enabling any science coming out of their work, and b) have some control (through the pub board review process) that the users are correctly using the services provided (and that the results are sound).

  6. Donald Petravick

    We need a "no warranty"  kind of statement if the functionality includes retaining user data.  

    Make sure that there is a clear warning – the system is a prototype, and the structure of  supporting data may change without warning – for example,

    queries can break due to underlying changes in database.

  7. Unknown User (ciardi)

    Donald Petravick (Gregory Dubois-Felsmann) ... Thanks Don, I made a change to the third to last paragraph - that paragraph was intended to be the no warranty paragraph, but I made it stronger.

  8. Gregory Dubois-Felsmann

    Donald Petravick, I agree with that specific point; once we provide the workspace capabilities in 2017, the issue of user-created data will become particularly important.  I think we want to walk a careful line and be able to say something like "we understand you are trying to do real science with this system and we will treat your work with respect, but we cannot make firm commitments about availability and data integrity".

    What Unknown User (ciardi) and I wrote above was in response to Mario's request for a policy focused on how to grant access.  It's not meant to be the full policy for the management of the system, or the document that users actually sign.  That will require another round of careful crafting.

  9. Mario Juric

    I made some modification in response to e-mail feedback from Beth Willman; she'll take a look at this draft, and possibly also run it by Steve K.

  10. Beth Willman

    So glad to see this plan developed, and policy written.  Looks great.  Just one text suggestion and two small questions:

    1. I suggest the change propose -> request, proposals -> requests. 
    2. "Any publications resulting from the use of the PDAC will be subject to the LSST Publications Policy." - Given that Tim Jenness is both in DM and Pub Board manager, I assume he's been consulted here?  The statement sounds right to me; Just want to be sure the t's are crossed.
    3. "The PDAC in its 2016-2017 form will have access limited to people with logins on an AIM system provided by NCSA." - Does this mean that access is limited to those that have NCSA logins prior to requesting access to the PDAC?  Or does it mean that approved external users will need to get an NCSA login?

    Discussed the contents with Steve; he saw no issues.

  11. Mario Juric

    Thanks Beth Willman!!

    Regarding your comments:

    1. Should be implemented now
    2. I'll ping Tim Jenness, but I suspect this won't generate much additional workload on the PubBoard (if it does, we'll all go and celebrate (smile)).
    3. It's the latter – approved external users will need to get an NCSA login.

    Thanks! I'll take this through the rest of the DM approval process & making sure all of our internal concerns are addressed, in preparation for standing up the PDAC later this year.

    1. Gregory Dubois-Felsmann

      Thanks! I'll take this through the rest of the DM approval process & making sure all of our internal concerns are addressed, [...]

      This is now under review as  RFC-208 - Getting issue details... STATUS .

  12. Gregory Dubois-Felsmann

    Inline comments addressed as requested by Kian-Tat Lim.

  13. Gregory Dubois-Felsmann

    Exported to PDF and saved as LDM-482.