Database Meeting 2023-03-29

Fritz Mueller :

• no major news from DMLT
• 4 days IDAC meeting last week. Qserv was one of the hot topics. GAve a presentation on Qserv in there. There is a possibility that more parties will be interested in setting up Qserv beyond what we have now (USDF, FrDF, and UKDF).

Igor Gaponenko here is what we have so far:

• 6 out of 15 nodes are available
• the ETA for adding the remaining nodes to the cluster is 2 weeks
• slac6 is back to life. Though, it had to be run under my own account gapon since we still don't have the service account rubinqsv ( the name of the account follows the naming convention adopted by LSST at USDF). A few tricks had to be made to allow running the Docker containers under an account that is not qserv (UID: 1000).
• there was some (minor) issue with locating the core files on RHEL8 as it's configured at USDF. Qserv management scripts (for the Docker mode) have been updated to re-enable debugging sessions (using gdb to inspect the core files of the terminated services).
• Documentation still needs to be updated on the slac6 at: Managing Qserv instances at SLAC
• the first test of Qserv indicates that ZFS compression doesn't seem to affect the performance of Qserv. This conclusion is based on comparing the run time of queries launched before the service migration to the permanent cluster vs the post-migration tests of the same queries 

Next steps:

• migrate Qserv to the service account rubinqsv once it's available
  • Fritz Mueller:
    • the container build tools and Kubernetes operation will need to be extended to allow  injecting role accounts into the containers
• expand the Docker-based Qserv to fill all 15 nodes once we have the full cluster available
• Fritz Mueller Kubernetes-based Qserv(s) to be installed as well (still depends on the availability of all nodes of the cluster)
  • It's important for us to get first-hand experience with installing and configuring Kubernetes and setting up Qserv on top of that

Igor Gaponenko :

• started experiments with this version of Qserv at slac6.
  • this version supports both file-based result-delivery protocols:  XROOTD and HTTP (via qhttp)
    • the HTTP path presently has a sub-optimal implementation which needs to be improved at both ends. Still need to add connection & DNS resolve caching on the client side and enable HTTP/1.1 in qhttp. This protocol (introduced in 1999) was disabled due to stability issues observed a year ago or so when testing the service within the Docker environment.
  • it also does the automatic garbage collection for files pulled and merged by Czar 
• the code still needs to be improved. For instance, I may have tripped the query cancellation made as a result of the "LIMIT <N>" optimization.
• first observations are to be reported

Igor Gaponenko and John Gates discussed the possibility of reviving the idea of "Uber chunks" whose implementation may greatly benefit from file-based result delivery. Fritz Mueller suggested to have a dedicated discussion of "Uber chunks" at the next group meeting.