Skip to end of metadata
Go to start of metadata

Date

Attendees

Goals

  • Continue discussing the configuration and schema management aspects of the lite  containers
  • Version management and integration for qserv  and qserv-operator 

Discussion items

TimeItemWhoNotes
Project newsFritz Mueller 

Progress on topics discussed at the previous meeting Database Meeting 2022-01-12


John Gates: rebased against the main branch, the latest version is not showing any lockups in the N-N queries. The LIMIT <N>  optimization is back in the version. Igor Gaponenko needs to test this version in both clusters at NCSA and review the new code.


Configuration and schema management aspects of the lite  containers

On entry-points:

Unknown User (npease)Igor Gaponenko discussed database connection strings in the command-line API, and if passing complete database connection URI is a better idea than passing arguments of the URI. Fritz Mueller suggested passing URI with JINJA templates for the run-time substitution as an alternative. The template substitution could also be done (by Fritz Mueller's proposal posted in the Database Meeting 2022-01-05 notes) in the extended arguments (specified after "--").

Igor Gaponenko mentioned that we need to reduce (where possible) the complexity of the configurations by choosing the right defaults.

Igor Gaponenko proposed extending the entry point implementations to allow seeing application's --help. Unknown User (npease) will implement it.


Discussing qserv-operator: integration with qserv

Fabrice Jammes needs to build a new version of the operator based on the latest state of Qserv. It's blocked by:

  • Unknown User (npease) 's work on the current ticket DM-33277 - Getting issue details... STATUS (Fritz Mueller was wondering what new features of the lite  container's CLI are needed for the operator. Fabrice Jammes has no use for the CLI yet).
  • Igor Gaponenko's work on making changes to the operator in DM-33296 - Getting issue details... STATUS as required by the last improvements in the configuration service of the Replication/Ingest system. And this was blocking Fabrice Jammes due to problems reported by the GHA CI.

Andy Hanushevsky mentioned that a new (better scalability) version of XROOTD/SSI is available.

  • (warning) This may be good news for Qserv. Though, more details on this subject is needed!

Discussed a need for multiple XRootd redirectors. Apparently, this might be needed for two reasons:

  • from the fault tolerance point of view (automatic failover to an alternative redirector)
  • and to increase the scalability of the service (the resource metadata would be divided between redirectors), which would result in better load balancing during operations with metadata.

The current implementation of the operator has a built-in (easy to configure) support for this feature.

Igor Gaponenko mentioned that he experimented with multiple redirectors and saw no improvements in the Qserv performance under heavy load.


Cowork to test the operator locally and to fix the problems reported by the GHA CI due to the latest changes made by Igor Gaponenko to the operator in 

DM-33296 - Getting issue details... STATUS


Fritz Mueller was wondering oif it's possible to run CI locally and wanted to understand how the GHA actions for the operator are set up.

Fabrice Jammes has demonstrated how to run CI locally by following his own instructions posted in https://qserv-operator.lsst.io/devel/build.html

Problems in the implementation of DM-33296 - Getting issue details... STATUS (Igor Gaponenko's ticket) were identified and fixed.

Action items