Database Meeting 2022-03-23

Unknown User (npease) Fritz Mueller 

Unknown User (npease) reported from the last DMLT meeting

• Camera cooling system problems. Switching to the liquid cooling system.
• Some discussion on the data provenance in Buttler
• Decommissioning of Qserv is shifting from September to August 2020

Grand Unified Repo:

• the outstanding PR on merging qserv_testdata is still waiting to be reviewed. Fritz Mueller will do it by Friday.

Optimizations in processing results of the N-N queries. Is the PR ready to be merged into the main branch?

• Fritz Mueller OK do the merge this Friday
• This is a blocking issue for the grand code formatting effort using clang.

Investigating worker load imbalance problem:

• A configuration problem with LSST logger of Qserv setups at NCSA has been fixed, and we can see now the desired logging info from the XrootD redirector service.
• John Gates is going to repeat the previous tests to trigger the problem, capture the content of the redirector's log and hand it over to Andy Hanushevsky for further investigation.

Database init schema evolution:

• The following page summarizes the state of this project: Database schema initialization
• Fritz Mueller's proposal has been discussed GKE experts
• Fabrice Jammes and Fritz Mueller will continue discussing the proposals with GKE experts

Felis schema status:

• the TAP service depends on this
• it has a few bugs that need to be fixed

There is a discussion on the case sensitivity mismatch between TAP and Qserv (MySQL):

• TAP service is claimed to be case-insensitive
• MySQL can be configured to be case-insensitive for database and table names (and it's not right now)
• the Replication/Ingest system IS case-sensitive
• Qserv MAY  be case-sensitive

Temporary solution:

• Fritz Mueller will apply a temporary fix to the Felis schema and deploy the one in TAP
• no need to re-ingest the small slice of the catalog into Qserv. Soon it will be replaced with a better dataset

Date-time representation in Qserv schema:

• continued discussing options mentioned yesterday at the team's Slack channel: TIMESTAMP, DATETIME or any numeric time to store nano (micro, or milli) seconds
• decided on using MySQL type DATETIME 
• Igor Gaponenko is going to experiment with the data conversion tool pq2csv to see what this tool produces for a sample input file provided by Colin Slater 
• Igor Gaponenko asked a question on the required environment for running the tool. Fritz Mueller will help Igor Gaponenko 

Colin Slater on status of the "real" data:

• Object table is ready. Available in IDF. ForcedSource table will be available later.
• Fixes are still needed on the Source table (delayed till the very end of the reprocessing)

This opens an option for ingesting Object table first and then doing other tables as they will be getting ready. Igor Gaponenko confirmed that this "table-by-table" mode is supported by the Replication/Inget system.