Database Meeting 2023-12-06

Fritz Mueller :

• no major news
• a lot of work on the Telescope assembly and control system
• the dome moved for the first time!

Travels/vacations:

• Fritz Mueller is back home on December 16th

Reminder: SLAC Holiday Party on Tue December 12th

Igor Gaponenko from the last Data Facilities meeting:

• no major news
• SLAC IT team is still working on bringing new hardware up
• current Cassandra loaners demanded to be returned to the k8s  pool
• reminded them that setting up Cassandra by January 2024 has a higher priority for the team than Qserv

No progress since the previous meeting

• Database Meeting 2023-11-15

The most recent release:

• 2023.11.1-rc3 :
  • still not deployed at -prod (IDF)

Fritz Mueller:

• suggested that Igor Gaponenko tried modifying Kubernetes
  • ACTION ITEM for Fritz Mueller: provide written instructions on how to modify the operator and get it tested

Preliminary plan for ad-hoc deployment of the release in -prod (IDF) and ACTON ITEMS for Igor Gaponenko :

• test this release on -int today
• deploy ad-hoc  this release on -prod tomorrow (during the Thursday outage window)
• one week after that enable HTTP-based result delivery protocol using rolling upgrades of workers
  • UPDATED 2023-12-08: There was the last minute change in the plan based on the success of deploying and testing the new release on -int. The release was configured for the HTTP protocol in -prod  as well.
  • See the status of releases in various Qserv instances at:  Qserv Deployments

A new case reported by Gregory Dubois-Felsmann :

• 3-way JOIN query seems to be unreasonably slow at IDF (40 minutes) and USDF (12 minutes)

Igor Gaponenko experimented with two optimizations:

• The MySQL engine-independent table statistics
  • DM-27695 - Getting issue details... STATUS
  • observed ~2 or ~3 speedup at USDF
  • collecting stats was a lengthy process (1 thread per worker) which took 3 days for the DP02 catalog alone
  • need to implement the parallel construction as indicated in the above-linked Jira ticket
• disabling the locking tables in memory
  • unexpectedly significant speedup at a level of ~50
  • to be discussed

John Gates :

• interactive queries do not 

Fritz Mueller :

• we can't afford to make indexes for ForceSorce  tables (narrow tables)
• we need to test it using a reasonable query mix using Kraken 

Igor Gaponenko:

• finalizing the PR of:
  • DM-41291 - Getting issue details... STATUS
• next projects in this direction:
  
  • DM-42005 - Getting issue details... STATUS
  • DM-42096 - Getting issue details... STATUS
• ongoing work on extended monitoring in the context of file-based result delivery:
  • DM-41535 - Getting issue details... STATUS

Igor Gaponenko :

• The problem was reported this morning
• More details at the meeting...