Things we know:

  • We are going to have an ObsTAP service
    • The ObsTAP service will use DataLink to help clients find related data
  • We are going to have a SODA service
    • The SODA service will talk directly to the production Butler repository
    • The SODA service's actual cutout code will be implemented as a Task/PipelineTask

Things we don't know:

  • What is access_url in the ObsTAP service's ivoa.ObsCore  table going to point to?
    1. Directly to a GET-able image file on an HTTPS service?
      • Is there a way to point directly to the backing store for the production Butler repository?
      • Or do we need a proxy service?  If so, who is writing that proxy service?
    2. An IVOA "links service" (CADC-style)?  This would serve up pointers to at least the original image file and to an associated SODA service.  (This is my preference.) 
      • If so, who is writing the links service?
      • Even if we have a links service, the "#this" entry in the returned table should point to the original image file, so just as in 1. above - what does that URL point to?  We would still need something like the above proxy service.
  • How is the ivoa.ObsCore table going to be loaded from the production outputs?
    • Via something like Kenny's undeployed code, executed in bulk over the production Butler?
    • Do we have to completely rewrite it?
    • The previous design relied on a mapping from Butler dataset types to a long list of ObsCore attributes.  Where does that mapping live and how will we maintain it?
    • Do we need / do we have a solution for incremental updates?
    • Is it possible to replace the ETL process previously envisioned, for which Kenny's code was written, with stored procedures in the Butler registry Postgres database that can generate an ObsCore table on the fly as a view?
  • Is ivoa.ObsCore going to be served from a separate TAP server talking to a separate Postgres database, as it is in the small hand-loaded prototype we now have?  Or will it be a replicated table on Qserv?
  • Will there be image metadata tables (e.g., "Exposure", "Visit") in the Qserv database, joinable to the catalog tables?
    • Will those tables contain columns that let them be used equivalently to the ivoa.ObsCore table?  (I.e., with DataLink pointers to image services)
      Hint: They really should. 
  • If (as seems obvious by now given the schedule) we don't deploy CAOM2 tables at the start of DP0.2, will we supply them later?
  • Are we going to have a HiPS coverage image for DP0.2, made from the coadds?
    • If so, it needs a web service endpoint in the RSP.
    • Where do the actual tile files live?  In the production Butler repo?  Or will we export them to a static web server (just nginx in front of a POSIX directory tree)?

Things we might know already:

  • What is the exact format of obs_creator_did and/or obs_publisher_did in the ObsCore table? (Russ Allbery  might know this.)
  • Does the SDM-ification process already produce Parquet files for tables like Visit, Exposure, etc.? (Yusra AlSayyad  might know this.)

Architecture

Rough sketch that illustrates some of the above

  • No labels