Discussion items


Project news, DLMT, etc.

PCW 2022 will be in person:

  • a question was regarding using STAP fund money to (partially) pay for some participants who are no longer paid from the construction fund

FY23 contract:

  • 100% ops money for most of us (but John) in FY23
  • Officially, Qserv is expected to be delivered. After that, any work is supposed to be done in the context of "improving", "fixing bugs", etc.


  • discussed further development plans in this context, including bypassing scalability limitations by decreasing the number of chunks. That will result in large chunks. We may need to turn off preloading chunk table sin memory to address that. Experiments with more RAM or NVMe-based storage are needed to see the effect of this change.

Colin Slater

  • by the end of FY23 "the user-facing functionality" is fully expected. The inner improvements of Qserv will be still permitted after that.
  • user-generated data in Qserv?

Progress on topics from the previous meeting Database Meeting 2022-04-27

(warning) The link is mentioned here as a reference.

Development cluster at USDF


  • losing all development resources (interactive machines, batch nodes, storage, and Qserv( at NCSA on August 15, 2022

Igor Gaponenko on the status of the test Qserv cluster at SLAC:

  • a snapshot of the large6 Qserv's data is staged on the Luster filesystem at SLAC
  • the loaner hardware (by Amedeo P.) is not ready yet. The last status update (2 days ago from Yee): "...awaiting out data center team to actually mount servers and pull some disks out...".

Fritz Mueller: any update on the status of the interactive development machines (similar to lsst-qserv-master02)? 

  • the idea to send 4 (or at least 1) Qserv lsst-qserv-master[1-4] nodes to SLAC is still on the table
  • an alternative option is a new machine rubin-devl01 pulled from the batch system. It's meant to be used primarily for Pipeline (Hsin-Fang Chiang ). The machine still requires additional configuration (Docker and sudo  privileges for the DAX team members). It could be a temporary platform for the new few months.

Status of DP02team

Igor Gaponenko: ingesting the full-blown table Object into various Qserv instances

  • ingested into the lasrge6  cluster at NCSA
  • poor performance of the unconstrained COUNT(*) queries: 6 minutes or so.
    • this is the high-density catalog, the average chunk size is 1 GB 
    • Unknown User (npease): are you still blocked by anything on this PR that's needed ASAP:
      • DM-34624 - Getting issue details... STATUS  
  • issues with ingesting products directly from the S3 buckets at GCS/GKE, data access restrictions, authentication options, signed URLs, etc.
  • it's not clear what's the long-term plan (if any) here, considering the known limitations

Fabrice Jammes: on plans for ingesting the catalog into Qserv at IN2P2:

  • not clear what's the requirement here
  • Colin Slater: IN2P3 is doing its own processing of the DP02 data
  • it would be nice to have a way to compare two versions of the catalog

Fritz Mueller: status of the table ObsCore, geometry support in Qserv, further plans?

  • ingested into qserv-int  at IDF
  • has a new schema ("data link URLs containeing UUID")
  • the special (spatial) index was built 
  • Unknown User (npease) 's fix (see below) is needed to correctly use the table
  • the last step needed to unblock this development is to publish TAP schema (30 columns in Felis schema definition). This is still waiting for GPDF. 

Unknown User (npease): what's the status of the following development which is needed for OsCore?

What's next for DP02?

  • Colin Slater: more tables will get ingested as we'll have them (Source, ForcedSource)
    • ForcedSource will be ready by the end of this week
    • Source will be ready by  the end of the next week
    • After that, we will have Ccd and CcdVisit tables
    • Right now the Pipeline is at state 6 (of 7)
  • Fritz Mueller:
    • ObsCore metadata will still evolve in a course of the following weeks before getting stable
  • Fritz Mueller or Colin Slater is going to verify the status of the Object  table schema loaded into the small  cluster at NCSA versus what's expected by the official schema

Colin Slater: reminded of a need to provide the magnitude translation SciSQL UDF.

  • Fritz Mueller will resume working on this after finishing work on the ObsCore table

Improved error reporting in Qserv


Igor Gaponenko: Main goals here are:

  • retaining errors in the persistent state of Qserv for further analysis
  • retaining quantitative info on the result sets (number of chunks, rows, bytes)
  • improving error reporting for users
  • facilitating problem and performance analysis by the data administrators


  • extend the table schema with performance counter columns in the current scheme to avoid further schema migrations
  • clean the message table from "success stories" to keep the table size concise
  • fix timestamps on the table entries
  • fix a problem with the "large result" message that was lost in the PR

Progress on qserv-operator  and qserv-ingest Are you still blocked by the code reviews on the PR?

Action items

  • Fritz Mueller will work with the GKE support team to make the pre-ingest products of DP02 visible via HTTP. This will unblock Igor Gaponenko's work on ingesting the catalog into qserv-int  at IDF>