• Overview of data access/databases/file-storage-systems in operations.
  • Specify/split up and group/consolidate use cases.


  • Michelle Gower presented an overview of the systems used in operations. These are primarily:
    • Operation production (archive, raw files, metadata in databases, etc.)
    • Operation L1
    • Release (qserv, read-only that external users query, provenance, metadata, tables, etc.).
  • Michelle GowerJim BoschBrian Van Klaveren contributed additional data access systems:
    • Developer systems (subsets of `Operation production` used for developing and debugging, etc.)
    • Personal databases (accessible from Jupiterlab, live in qserv (probably the same instance as `Release` to enable cross-joins))
      • Brian Van Klaveren mentioned that creating and ingesting data into these user databases is not done by the user itself)
    • External local pre-cache
    • External remote on-the-fly
    • External large local (e.g. DESC@NERSC, HSC@Japan)
  • Jim Bosch then suggested to go through the use cases and decide for each of them for which of the above systems it was relevant. A case from Simon Krughoff was selected as an example: "A science user sees a bad coadd and wants to find out what went wrong by inspecting the full chain of things that went into this coadd." This was further specified into:
    • 1.a Find inputs into coadd (warps, calexp, etc)
    • 1.b Get files into my sandbox (exists on disk in DB)
    • 1.c Get intermediate files that were used to generate this product (Brian Van Klaveren, Michelle Gower does this involve regenerating data on-the-fly?)
    • 1.d Inspect files by eye.

Additional discussion

Conclusion and task for next meeting

  • Go through each use case and:
    • split it up into specific steps,
    • then mark each step based on relevance to each data access system (using template matrix).

  • No labels