Attending
Agenda
- Overview of data access/databases/file-storage-systems in operations.
- Specify/split up and group/consolidate use cases.
Overview
- Michelle Gower presented an overview of the systems used in operations. These are primarily:
- Operation production (archive, raw files, metadata in databases, etc.)
- Operation L1
- Release (qserv, read-only that external users query, provenance, metadata, tables, etc.).
- Michelle Gower, Jim Bosch, Brian Van Klaveren contributed additional data access systems:
- Developer systems (subsets of `Operation production` used for developing and debugging, etc.)
- Personal databases (accessible from Jupiterlab, live in qserv (probably the same instance as `Release` to enable cross-joins))
- Brian Van Klaveren mentioned that creating and ingesting data into these user databases is not done by the user itself)
- External local pre-cache
- External remote on-the-fly
- External large local (e.g. DESC@NERSC, HSC@Japan)
- Jim Bosch then suggested to go through the use cases and decide for each of them for which of the above systems it was relevant. A case from Simon Krughoff was selected as an example: "A science user sees a bad coadd and wants to find out what went wrong by inspecting the full chain of things that went into this coadd." This was further specified into:
- 1.a Find inputs into coadd (warps, calexp, etc)
- 1.b Get files into my sandbox (exists on disk in DB)
- 1.c Get intermediate files that were used to generate this product (Brian Van Klaveren, Michelle Gower does this involve regenerating data on-the-fly?)
- 1.d Inspect files by eye.
Additional discussion
Conclusion and task for next meeting
- Go through each use case and:
- split it up into specific steps,
- then mark each step based on relevance to each data access system (using template matrix).
{"serverDuration": 130, "requestCorrelationId": "77d6f1e82f28eef6"}