Page History

Fritz Mueller Colin Slater 

Igor Gaponenko Colin Slater Andy Hanushevsky 

Igor Gaponenko: to report on the status and obstacles

• it's still not clear to me how to treat correctly the empty (NULL) values in the Parquet files. There was a discussion on this subject last night on the team's Slack channel: https://lsstc.slack.com/archives/G2JPZ3GC8/p1652852985598069

Andy Hanushevsky : commented on pq2csv:

• The Parquet files have the schema, including the NULL  requirements for select columns
• Apparently, there is a bug in the tool that needs to be investigated
• Error reporting for incorrect input schema needs to be improved to report the incorrectly configured columns

Igor Gaponenko will provide Andy Hanushevsky with a sample Parquet file (DP02 table Source) and the schema file for further investigation

Andy Hanushevsky: the NULL option for the columns needs to be documented in the LSST documentation for schemas

Fritz Mueller Igor Gaponenko 

Igor Gaponenko: to report on the last round of negotiation with Yee

• preliminarily agreement on the following configuration of workers: 2U server, single-socket CPU 32C/64HT, 256 GB, 24 disk slots, 12 slots will be populated with NVMe drives of 3.5 GB each. So the total amount of the raw storage per worker would be 42 TB. The usable amount after setting up RAID would be 2 drives less.

Fritz Mueller: any news on the development machines?

Unknown User (npease) Fabrice Jammes Fritz Mueller 

Why two versions of the integration tests?

• The qserv-ingest is meant for testing the Kubernetes-based ingest framework. At the end of the ingest workflow the result validation test based on DB Bench is being run. It's used for various purposes, including performance.
• The qserv itest is meant for testing Qserv in the Docker compose environment.

Are we going to merge both packages into qserv? What implications of this move we should expect?

Igor Gaponenko John Gates 

Context:

• Jira
  server JIRA serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-34464

• There are still uncertainties about what needs to be retained in the persistent state (new table qservmeta.QMessages. How many details we should retain?
• What needs to be done in the long term to improve error reporting to users and to make it usable for Qserv data administrators?

Fritz Mueller is going to look for the use of the column code by mysql-proxy. If it's not used then we should eliminate it.

Context:

• there is a little bit of confusion on the overlap  tables of the child  tables

Igor Gaponenko will run extensive tests and report results on the Confluence page (TBC)