Database Meeting 2024-03-06

Fritz Mueller :

• Jira migration?
  • The link to the new Cloud-based Jira (a filet for all known issues was applied):
    • https://rubinobs.atlassian.net/jira/software/c/projects/DM/issues/?filter=allissues
  • added sprint filter first
  • other filters will added later
• news from DMLT
  • lots of planning and slides preparation for the operations review in Tucson (April 16-18)
  • Will make a short trip to Tucson to work on a few projects. Will coincide with the trip with the review
• lots of activities at Sumit, including:
  • TMA testing 
  • ComCam
  • refrigeration (coping with the last power outage)
  • the mirror is moving ... slowly
• Reminder: March 10 is the US daylight savings switch
• Cristin Bannek left the project. A new hire is going to take over. The hire will be a contact point for TAP.

Notes from this week's meeting:

• Joint Data Facilities Meeting - 2024-03-04

Andy Salnikov

• Butler schema migration status
• Cassandra
  • still testing using 1/2 of the cluster
  • AP people are still not ready to take over the other 1/2 of the cluster
    • Fritz Mueller: is there a separate instance for them?
    • Andy Salnikov: yes. However, it has not been tested by AP people.
  • hopefully, no PPDB schema changes will be needed if the good performance will be demonstrated by the tests
  • running Cassandra via Docker since it's simple to make it visible to clients (it would be hard to do in Kubwernetes)
    • Fritz Mueller agreed with that as a temporary solution 

Igor Gaponenko:

• no news for the Qserv cluster expansion
  • existing batch of 28 nodes which is still being installed, no specific ETA so far
  • the PO for the next batch of 50 nodes is still on hold(?)
  • Fritz Mueller from a response  got from Richard: no news for both
• the USDF-wide monitoring effort for application services seems to be gaining traction
• there is progress on building an infrastructure to support large-scale backups:
  • tape-based RSE integration in progress
  • ETA: ~3 weeks
  • Fritz Mueller:
    • Qserv backups need to be coordinated with catalog ingests
    • we still have uncertainties in the cadency of the ingest inputs delivered to Qserv by pipelines
    • we have no plan on how the ingest will happen after DRP
    • LSST catalog is ~100 compared with the largest catalog that has been ingested into Qserv so far
  • Colin Slater 
    • Some tables may appear earlier than others. It's Object table. Table Source could delivered earlier as well.
    • Expected scale - weeks, not months.
    • Data products would be delivered as tracts and patches (a fraction of a %)

IDF is still on 2023.11.1-rc3.

Fritz Mueller is about a make a new Qserv release based on the latest state of the main branch of Qserv:

• ETA: this afternoon 

A fix for the "table leak" in the temporary message tables created by MySQL proxy has been addressed:

• DM-42638 - Getting issue details... STATUS
• it will address (prevent) a known issue with "garbage collection" of the old result and message tables
• the fix will be included in the new release (see above)

Igor Gaponenko on intermittent issues with GHA for PRs have been observed:

• are seen during test catalog ingests as timeouts to contact the Replication Controller
  • is the Controller dead for some reason?
  • is it possible to see what containers are running at that time?
  • is it possible to get the log of the containers from GHA?
  • if the same test runs successfully locally then perhaps there are some differences in the GHA setup which might cause the effect.
  • will inspect the history of GHA runs

Context:

• DM-11526 - Getting issue details... STATUS

John Gates on progress with "booting" the worker tasks:

• managed to keep them from being a problem for normal queries.
• still having trouble with booting queries
• still having a "Fast" scheduler hanging on a mutex which gets held up when processing long queries
• It doesn't happen very often
• It happens occasionally
• There seems to be a race condition

Igor Gaponenko :

• (presently) working on extending the Docker-compose-based integration test to include the new frontend
  • DM-42810 - Getting issue details... STATUS
  • it looks like the test needs to be migrated to compose V2 since V1 is no longer supported as of July 2023: https://docs.docker.com/compose/migrate/
  • Fritz Mueller noticed the version change in other contexts. And the changes in the API weren't that big (if any).
• (next) add the frontend as a service to qserv-operator 
  • TODO: Jira ticket
• (next) extending the front for TAP uploads of user tables
  • Fritz Mueller: what should we plan on receiving from TAP as the input for ingest? We need to know that to make a suitable REST API for that.
  • Colin Slater:
    • No answer yes. However, TAP presently supports PostgreSQL. So it could be a text file with INSERT .... ?
  • Igor Gaponenko here is what we need as input:
    • the name of a database
    • the name of a table
    • the table schema
    • the table payload in CSV, or JSON (preferably), or XML (if needed)