Database Meeting 2022-11-09

Fritz Mueller:

• gave a presentation on Qserv at ADASS
• there is a 3-page document to be DMTN-243 at Pub Board for review: https://dmtn-243.lsst.io/
• action item for Igor Gaponenko: send Fritz Mueller a few sentences on the parallel ingest system to be included in the publication

The upcoming event:

• next Mon & Tue: 4th Data Facilities Workshop - 2022-11-14/15

Fritz Mueller on the office move:

• there is a lot of pressure to move out of B50
• a decision has to be made ASAP
• most likely it's going to be the first-floor office (yet to be settled within the team)
• the equipment will be moved by SLAC (there is a special team of people who do it)

 Context:

• the topic was discussed at the previous meeting Database Meeting 2022-11-02
• creating table indexes for the sub-chunk column didn't seem to help to address the issue
  • a "byproduct" of the investigation was an interesting discovery on the ordering of rows in sub-chunk and chunk tables in which rows belonging to sub-chunks are not clustered together. This would dramatically reduce the performance of the subchunk materialization algorithm, even if the sub-chunk index is present in the tables. Note that the order of the rows is defined by the partitioning tools ( sph-partition and sph-partition-matches ). A few ideas on how to improve this were discussed betweenFritz Mueller and Igor Gaponenko, including improving the partitioner, postprocessing (sorting rows) partitioned contributions by the ingest workflows before ingesting them into Qserv or doing this inside the Ingest system.
    • Action items for Igor Gaponenko:

      • Experiment with reordering rows of the fully ingested chunk and overlap tables to group the rows based on their sub-chunk numbers to see if such optimization would speed up the sub-chunk materialization. This experiment assumes the table indexes are already present.

      • make a JIRA ticket if there will be any benefits in the reordering
      • John Gates commented on this by saying that the reordering may not help since sub-chunks are normally materialized from tables that are already locked in the memory of the worker host.
      • Igor Gaponenko replied: we still have cases (deployments, configurations) where the memory locking mechanism may not work. The experiment will tell.
• the problem was the root cause of the OOM crash at the workers 
• JIRA ticket was opened by John Gates 
  • DM-36935 - Getting issue details... STATUS
• Igor Gaponenko has started testing the effect of this improvement using the slac6 Qserv

One topic from the previous meeting:

• the FrDF team is looking at developing a better Parquet-to-CSV  translator (to be written in C++) with a possible option of integrating the one into the partitioning tools
• any news here?

Fabrice Jammes and Igor Gaponenko discussed an option for formal schema validation:

• the Ingest API's services will report the expected schema in some formally recognized schema definition format
• the ingest workflows would have to employ the corresponding formal schema validation tools to make sure the JSON files to be used by the workflow are compatible with the expected schema
• the very same should be done for the qserv-ingest 's own file metadata.json since the schema of the file may also evolve over time, and different copies of the files might be laying around (GitHub, file systems, etc.). 
• altogether, this early checking should reinforce the ingest campaigns and prevent the "midflight" failures

Fabrice Jammes Where to put the documentation on the versioning of metadata.json?

• Fritz Mueller: unlike the Ingest API, qserv-ingest is facing operations. So, it needs to be documented in the relevant (for operations) area of the documentation.

Igor Gaponenko: there is a new feature of the Ingest system that's about to be merged into the main branch of Qserv:

• DM-36722 - Getting issue details... STATUS

Andy Salnikov: the most recent status of the project can be found at: Live ObsTAP service deployment