Types of Services

The following services will be provided:

• Metadata Service
  
  • Provides efficient, flexible querying capabilities to all stored metadata information by querying Metadata Store, or by extracting information from the data
• Query Service (Qserv)
  • Provides access to Database repositories
• Image Service (ImgServ)
  • Cutout Service: provides access to individual images / cutouts of images
  • Mosaic Service: handles complex operations that involve multiple images
• Web Data Access Service
  • Main point of entry for SUI. Parses incoming commands (RESTful format), channels to the appropriate service.

Querying Through Services

Allow users to browse and search repositories (these queries are against Metadata Service). Query by:

• intention
• user
• tag(s)
• date
• task (and options)
• config value
• area on the sky

Allow users to retrieve image datasets (this goes to Image Cutout Service or Mosaic Service):

• Images
• Image cutouts

• Example queries:

  • return a list of images (note these would be unique image ids, not physical locations), or actual images that:
    • include a given point
    • contain a given area (entire area falls inside image)
    • are within a given area (entire image fall inside the area)
    • overlap with a given area (at least one pixel). For pixels that fall outside, fill them with NaN or user-provided value
  • return the best (most centered) image (or image id) for the usecases listed above
  • return cutout of a given size centered around a given point 
  • for the above further filtering might be involves, e.g., images owned by certain user, from certain release, inside certain time window, images of certain type, etc

Allow users to retrieve catalog information (this goes to Qserv):

• SELECT and WHERE clauses for database table datasets
• Optionally full SQL query against databases

Allow users to browse history of queries they run in the past, rerun queries from that list.

Allow users to generate output (images, tables) and treat it as input for next query.

Allow users to upload a list of "things" to workspace, then request "repeat a given query for each "thing" from the uploaded list". Example of "things": ra/dec points, ra/dec points + distances, object ids, object names, bounding boxes, etc. In IPAC terminology, this is called "table upload queries". Note, we will need to assign some sort of unique id to such list. Results must contain information which result correspond to which "thing". E.g, if user asks for neighbors near (1,1), (4,4), it is not enough to just return list of neighbors, we need to tell which neighbor is for which point. (related epics: DM-1556 for Qserv, DM-1557 for ImgServ)

Submitting query from SUI – typical scenario:

• run "explain <query>"

• "explain <query>" returns cost estimate (based on #chunks accessed, # joins, math complexity), and query type (interactive / scan, if scan: scan type, scan speed)

• submit query (interactive or in background depending on what "explain" suggested)

• if interactive and query takes too long, kill and resubmit as background query.

Note that SUI will facilitate rerunning past queries easily. Metadata Store will contain information about availability of results from queries recently executed, which can be used by SUI to return result without rerunning the query, if results are available. Note, it can get tricky with L1, as "latest" is a moving target.

Returning Results from Services

• Streaming, not stored anywhere

• Persisting in a user workspace (to database table, or a file or a set of files), return "location").

• All background queries use "persisting" mode. Interactive can do either one.

Output should be delivered in a variety of data formats (even in the short term):

• FITS images
• FITS tables
• Text files (e.g. pex_config or serialized metadata)
• JSON
• CSV

• In the long term, we'll also need to support VOTables (TAP), but that may be solely via the DB.

Authentication / Authorization

Access to all services will be controlled. Envisioned modes:

• who: owner only / group only / open to everyone
• access type: read / write

Authentication will be handled through Data Center-provided mechanisms (e.g., provided by NCSA for the main Data Archive Center).

Single-sign-on system would be ideal, especially for users coming from SUI: upon successful authentication (through a cookie or whatever) appropriate mysql-credentials associated with given user will be fetched and used. Note, it can be hard to implement for direct database API connections, it is more feasible if access occurs through wrapped, LSST-provided APIs.

Every science user should have MySQL credentials.

Relevant Off-the-shelf Systems