Database Meeting 2022-04-13

Fritz Mueller 

Supporting documentation within Qserv package:

• Where are we standing with this? We need to begin with setting up the documentation tree.
• Fritz Mueller: the draft version is available at https://qserv.lsst.io/v/DM-32709/user/index.html
• Igor Gaponenko: where do we put info on the Replication/Ingest syste,?
• Fritz Mueller: this was in  (LINK NEEDED)

Status of the "grand unification":

• Fritz Mueller: the next step is qserv-operator. Waiting for Fabrice Jammes here.

Using clang-format to unify code format:

• Unknown User (npease): recent version of VSCode broke the plugin
• Fritz Mueller: will do all-out formatting next week 

 Fritz Mueller, team

Context:

• what's the role of this setup? Migrating from NCSA? Setting up an advanced development platform? Serving LSST needs for DP02, ComCam and beyond?
• there is an ongoing discussion on the first batch of hardware for installing Qserv at USDF
• a couple of technical questions still exist

Fritz Mueller:

• originally IDF was meant to be expanded to support the development Qserv instances
• now we are switching to the on-prem paradigm at SLAC (USDF) since we need to see how it all will work here (GKE and USDF are different)
• Igor Gaponenko: what are the storage capacity requirements for this instance of Qserv?
• Fritz Mueller: no demand to support Qserv for commissioning. There are two aspects here: 1) USDF folks are going to use it for practicing how to set up and wire (Kubernetes, etc.) the infrastructure for running Qserv in (the future) production (the infrastructure perspective), and 2) for the DB team to continue developing and improving Qserv as an application (the application  perspective)

Fabrice Jammes 

Context:

• the new version of the R-I system adds the worker registry that needs to be integrated into the operator
• the latest bugfix release of Qserv should be used by the operator: qserv/lite-qserv:2022.4.1-rc1
• support for the liveliness probe is about to be merged  DM-34404 - Getting issue details... STATUS

team

DP02 is supposed to be ready in June 2022

Support for ingesting the image metadata tables Visit  and CcdVisit into Qserv has been added. Both tables have been test-ingested into the 

Questions/problems that are still remaining:

• technical aspects of using Felis as a "source of truth" for preparing table schemas as needed by the Qserv ingest protocol (JSON is expected, while Felis  s still producing the "traditional" MySQL CREATE TABLE  statements.
• publishing/unpublishing/extending catalogs to add more tables and adding rows to existing tables. What protocol/API do we need here?

Fritz Mueller:

• manually ingest tables Visit  and CcdVisit into Qserv into the small first and into IDF after that
• work on improving the Ingest system to add support for adding tables in the long-run
• then work with Colin Slater on ingesting the final version of the Object table into IDF (dev or int?). No need to ingest it into the small cluster at NCSA. Then work on improving the Ingest system system
• get in touch with Hsin-Fang Chiang on how to use Felis for generating table schema in the JSON  representation, and using the ingest workflow
• concerned about 8,000 MySQL partitions limit per table in the long run

Colin Slater:

• no need in adding rows to existing tables
• the final version of the Object  table (as Parquet files) is available in IDF in Google buckets
• the table size is 1 TB (in the Parquet format)
• the schema was upgraded compared with the first attempt to ingest a slice of the DP02 catalog

Colin Slater and Fritz Mueller discussed the need in translating magnitudes into fluxes:

• the first option would be to let users do it using SciSql (the current implementation is not suitable for that). A possible solution would be to have the right UDF function in SciSql for this transformation.
• another option would be to set up views on the object tables
• Igor Gaponenko: the Ingest system could be improved to support creating views
• Fritz Mueller: at the meantime, we could create views manually in IDF

Discussion on using/supporting views in Qserv:

• Igor Gaponenko: we need to see if Qserv correctly supports views. Views need to be described in Qserv's CSS.
• Andy Salnikov: indexing won't work on views. Perhaps it's better to implement views at czar.

Fabrice Jammes 

Unknown User (npease) Igor Gaponenko 

Unknown User (npease): Query support has been implemented for unconstrained COUNT(*).

Igor Gaponenko: we need to discuss how to construct the row counter tables in Qserv, if this needs to be done automatically at each ingest or consider this as an optional optimization to be done after the fact (publishing catalogs) as we do it for the table local indexes at workers.