Skip to end of metadata
Go to start of metadata

Date

Attendees

Notes from the previous meeting:

Discussion items

DiscussedItemNotes
(tick)Project news

Fritz Mueller:

The upcoming event:

Fritz Mueller on the office move:

  • there is a lot of pressure to move out of B50
  • a decision has to be made ASAP
  • most likely it's going to be the first-floor office (yet to be settled within the team)
  • the equipment will be moved by SLAC (there is a special team of people who do it)

(tick)

(continued from the previous week)

A bug in Qserv czar when handling failed chunk queries

 Context:

  • the topic was discussed at the previous meeting Database Meeting 2022-11-02
  • creating table indexes for the sub-chunk column didn't seem to help to address the issue
    • a "byproduct" of the investigation was an interesting discovery on the ordering of rows in sub-chunk and chunk tables in which rows belonging to sub-chunks are not clustered together. This would dramatically reduce the performance of the subchunk materialization algorithm, even if the sub-chunk index is present in the tables. Note that the order of the rows is defined by the partitioning tools ( sph-partition and sph-partition-matches ). A few ideas on how to improve this were discussed betweenFritz Mueller and Igor Gaponenko, including improving the partitioner, postprocessing (sorting rows) partitioned contributions by the ingest workflows before ingesting them into Qserv or doing this inside the Ingest system.
      • Action items for Igor Gaponenko:
        • Experiment with reordering rows of the fully ingested chunk and overlap tables to group the rows based on their sub-chunk numbers to see if such optimization would speed up the sub-chunk materialization. This experiment assumes the table indexes are already present.

        • make a JIRA ticket if there will be any benefits in the reordering
        • John Gates commented on this by saying that the reordering may not help since sub-chunks are normally materialized from tables that are already locked in the memory of the worker host.
        • Igor Gaponenko replied: we still have cases (deployments, configurations) where the memory locking mechanism may not work. The experiment will tell.
  • the problem was the root cause of the OOM crash at the workers 
  • JIRA ticket was opened by John Gates 
  • Igor Gaponenko has started testing the effect of this improvement using the slac6 Qserv

Igor Gaponenko:

  • should simplify the management scripts for slac6 to help John Gates manage Qserv
  • may run the low-level experiments with manually materializing sub-chunk tables and running MySQL queries against those to see if there are some scalability issues within MariaDB itself for slowing down these queries when there are a few hundred in MEMORY tables in use


(minus)

(no news, postponed till the next meeting)

User-generated data productsTBC...
(tick)qserv-operator and qserv-ingest

One topic from the previous meeting:

  • the FrDF team is looking at developing a better Parquet-to-CSV  translator (to be written in C++) with a possible option of integrating the one into the partitioning tools
  • any news here?

Fabrice Jammes and Igor Gaponenko discussed an option for formal schema validation:

  • the Ingest API's services will report the expected schema in some formally recognized schema definition format
  • the ingest workflows would have to employ the corresponding formal schema validation tools to make sure the JSON files to be used by the workflow are compatible with the expected schema
  • the very same should be done for the qserv-ingest 's own file metadata.json since the schema of the file may also evolve over time, and different copies of the files might be laying around (GitHub, file systems, etc.). 
  • altogether, this early checking should reinforce the ingest campaigns and prevent the "midflight" failures

Fabrice Jammes Where to put the documentation on the versioning of metadata.json?

  • Fritz Mueller: unlike the Ingest API, qserv-ingest is facing operations. So, it needs to be documented in the relevant (for operations) area of the documentation.

Igor Gaponenko: there is a new feature of the Ingest system that's about to be merged into the main branch of Qserv:

(tick)Status of ObsCore  tables

Andy Salnikov: the most recent status of the project can be found at: Live ObsTAP service deployment

Action items

  •