Database Meeting 2023-05-17

Fritz Mueller:

• M1/M3 is under tests, M2 will start soon (initially w/o the mirror itself)
• project/budget planning is being discussed by the project management

Colin Slater:

• started HSC processing

Fritz Mueller:

• the effort is documented at:
  • PREOPS-3413 - Getting issue details... STATUS
• the initial version of the catalog was ingested before going to Chile
• using PgSphere for spatial searches
• a few minor issues had to be resolved
• tested by Colin Slater from the Portal
  • 3 hours to make indexes
  • 0.5 to ingest
  • realistically should be planning to spend ~1 day 

 Igor Gaponenko:

• still, no progress with finding the root cause of the occasional timeouts for POST requests sent to the worker Ingets services
• the catalog (1 TB of data, 1.5k chunks, 3k contributions, single director table) was successfully ingested at USDF (low-tech Docker-based deployment). No timeouts were seen during the ingest. The aggregate network I/O performance for pulling contributions from UK was ~120 MB/s.
• the catalog was also ingested at IDF (qserv-dev) using two different methods:
  • qserv-ingest
  • low-tech tools (curl and a simple Python script)
• Long (hundreds of seconds) timeouts were observed at the beginning of each ingest effort. However, both efforts succeeded.
• An attempt to increase the timeout limit to 30 min at UkDF still resulted (eventually) in the same failures
• My preliminary conclusion: the timeouts may be caused by the communication delays within the Kubernetes network
• next steps in the investigation:
  • run a Web server within the worker pod to test the POST requests
  • configure LSST Logger of the worker ingest service to see the internal diagnostics
• This investigation triggered a few improvements/bug fixes on the Ingest system:
  • DM-39201 - Getting issue details... STATUS
  • DM-39241 - Getting issue details... STATUS
  • DM-39200 - Getting issue details... STATUS
  • More improvements may be made based on the results of the investigation

Igor Gaponenko:

• DM-39080 - Getting issue details... STATUS
• didn't actually appear to be the "leak". Apparently, the effect is caused by how the Linux memory allocator works for multiple threads and by the memory fragmentation
• (naively, w/o any tuning) tried jemalloc as well w/o any significant differences
• it looks like we're not using it in any Qserv deployment (we used to at NCSA)
• found an old ticket where we observed a similar memory footprint growth
  • DM-26207 - Getting issue details... STATUS

Andy Hanushevsky 

• try optimization options for jemalloc 
• standard memory allocation tries to keep the memory closer to the CPU and sometimes it misses to rebalance the arenas
• it's actually the CPU pool, not the thread one!

John Gates:

• send increments instead of snapshots

Fritz Mueller :

• put jmalloc back as it allows tuneup kabobs for the memory allocator
• it looks like the problem was triggered by switching to all-sky (150k chunks) catalogs and large core count machines
• the long-running REST services of the R-I system need to be re-engineered to be async 

The lockups seem to have been resolved and merged into the main branch by John Gates in:

• DM-38909 - Getting issue details... STATUS
• DM-39052 - Getting issue details... STATUS

This clears the path for Igor Gaponenko 's work on the file-based result delivery protocol.

Also, there is an interesting idea to be investigated for implementing a more efficient query cancellation logic. We need a similar mechanism for notifying workers on the completion of the queries to allow workers to release resources associated with the queries. Apparently, XROOTD has a mechanism allowing to broadcast requests to all workers (so-called "subscribers" for some resources). I have made no progress on that.

Fritz Mueller:

• talked to Yee and Richard
• going to schedule a meeting (including myself, Andy S, Yee and Richard) to figure out how to move this forward to unblock the scalability testing of Cassandra