Database meeting 2022-03-16

Fritz Mueller 

Grand Unified Repo:

• PR for merging qserv_testdata is still on the review for DM-33618 - Getting issue details... STATUS

Database init:

• TBC

Subchunk sizes and overlap:

• the problem is pertinent to the high-density catalogs
• a goal is to decrease the number of rows in each subchunk to improve the of cross-joins in the N-N queries
• do the query profiling first on the existing catalog (Fritz Mueller )
• Igor Gaponenko has proposed to use an existing Data Exportation service of the Replication/Ingest system to get a subset of chunks from the existing kpm50  catalog where the problem is seen and reingest these data into the same 
• Fritz Mueller will further investigate it and schedule it based on existing priorities.

John Gates, team

The context:

• DM-33346 - Getting issue details... STATUS

The context was set in the previous meeting (see the link Database meeting 2022-03-09):

• seems to be XRootD version dependant (Andy Hanushevsky 's help is needed here)
• Andy Hanushevsky still needs to see the redirector's logs from the redirector and from one of the workers to see what's going on. John Gates would do this.

Andy Hanushevsky:

• affinity works fine before an overload happens. After that XROOTD begins shifting chunk requests to further workers. This explains the linear behavior.

Resolution:

• Andy Hanushevsky and John Gates will work on the further investigation based on the log files

Context:

• This is the second time we're seeing this in IDF. The previous time was reported in Sep/Oct 2021. These are links to the Slack channel:
  • https://lsstc.slack.com/archives/C8EEUGDSA/p1633024306132700
  • https://lsstc.slack.com/archives/C8EEUGDSA/p1634233691204600
• The problem was caused by the rolling update of GKE. Allegedly, it's caused by stale IP addressed cached by XRootD services.
• It's due to optimization in the service that won't re-resolve the address at each request.

How do we investigate this?

• Andy Hanushevsky inspect the log files to see what service has the wrong address
• Andy Hanushevsky 's theory is that we may have some "rogue" service in Qserv using the wrong IP address

Possible short-term solutions:

• coordinate GKE upgrades with complete restarts of Qserv 

Fabrice Jammes 

Fabrice Jammes 

Context:

• The new version of the Replication/Ingest system adds the required worker registry service that needs to be properly configured and used by the dependent services of the system.
• Igor Gaponenko published instructions for that at Configuring worker registry in the Replication system of Qserv

Fabrice Jammes is still working on the Operator to integrate the change.