Skip to end of metadata
Go to start of metadata




Discussion items


Project news

Summary notes from 3rd Data Facilities Planning Workshop - 2022-04-05/07

Project All Hands on Thu April 14 at 9am PDT

LSST Camera Party at SLA on Thu April 14 (detail in e-mail)

Farewell party for Steven Khan on Wed Apr 20 at 5pm

Progress on topics discussed at the previous meeting Database Meeting 2022-04-06team

Supporting documentation within Qserv package:

Status of the "grand unification":

Using clang-format to unify code format:

Hardware needs for setting up Qserv at USDF


  • what's the role of this setup? Migrating from NCSA? Setting up an advanced development platform? Serving LSST needs for DP02, ComCam and beyond?
  • there is an ongoing discussion on the first batch of hardware for installing Qserv at USDF
  • a couple of technical questions still exist

Fritz Mueller:

  • originally IDF was meant to be expanded to support the development Qserv instances
  • now we are switching to the on-prem paradigm at SLAC (USDF) since we need to see how it all will work here (GKE and USDF are different)
  • Igor Gaponenko: what are the storage capacity requirements for this instance of Qserv?
  • Fritz Mueller: no demand to support Qserv for commissioning. There are two aspects here: 1) USDF folks are going to use it for practicing how to set up and wire (Kubernetes, etc.) the infrastructure for running Qserv in (the future) production (the infrastructure perspective), and 2) for the DB team to continue developing and improving Qserv as an application (the application  perspective)

Progress on qserv-operator 


  • the new version of the R-I system adds the worker registry that needs to be integrated into the operator
  • the latest bugfix release of Qserv should be used by the operator: qserv/lite-qserv:2022.4.1-rc1
  • support for the liveliness probe is about to be merged  DM-34404 - Getting issue details... STATUS

Status of DP02


DP02 is supposed to be ready in June 2022

Support for ingesting the image metadata tables Visit  and CcdVisit into Qserv has been added. Both tables have been test-ingested into the 

Questions/problems that are still remaining:

  • technical aspects of using Felis as a "source of truth" for preparing table schemas as needed by the Qserv ingest protocol (JSON is expected, while Felis  s still producing the "traditional" MySQL CREATE TABLE  statements.
  • publishing/unpublishing/extending catalogs to add more tables and adding rows to existing tables. What protocol/API do we need here?

Fritz Mueller:

  • manually ingest tables Visit  and CcdVisit into Qserv into the small first and into IDF after that
  • work on improving the Ingest system to add support for adding tables in the long-run
  • then work with Colin Slater on ingesting the final version of the Object table into IDF (dev or int?). No need to ingest it into the small cluster at NCSA. Then work on improving the Ingest system system
  • get in touch with Hsin-Fang Chiang on how to use Felis for generating table schema in the JSON  representation, and using the ingest workflow
  • concerned about 8,000 MySQL partitions limit per table in the long run

Colin Slater:

  • no need in adding rows to existing tables
  • the final version of the Object  table (as Parquet files) is available in IDF in Google buckets
  • the table size is 1 TB (in the Parquet format)
  • (warning) the schema was upgraded compared with the first attempt to ingest a slice of the DP02 catalog

Colin Slater and Fritz Mueller discussed the need in translating magnitudes into fluxes:

  • the first option would be to let users do it using SciSql (the current implementation is not suitable for that). A possible solution would be to have the right UDF function in SciSql for this transformation.
  • another option would be to set up views on the object tables
  • Igor Gaponenko: the Ingest system could be improved to support creating views
  • Fritz Mueller: at the meantime, we could create views manually in IDF

Discussion on using/supporting views in Qserv:

  • Igor Gaponenko: we need to see if Qserv correctly supports views. Views need to be described in Qserv's CSS.
  • Andy Salnikov: indexing won't work on views. Perhaps it's better to implement views at czar.

Debugging in Kubernetes

Query optimization based on the row counters populated by the Qserv Replication system

Unknown User (npease): Query support has been implemented for unconstrained COUNT(*).

Igor Gaponenko: we need to discuss how to construct the row counter tables in Qserv, if this needs to be done automatically at each ingest or consider this as an optional optimization to be done after the fact (publishing catalogs) as we do it for the table local indexes at workers.

Action items