Database Meeting 2023-10-18

Fritz Mueller:

• on updated project schedule:
  • ComCam Spring 2024 (no guarantee)
  • LSSTCam shipment scheduled for March 2024
  • system "first light" ~2025
• Other news:
  • Joined operation review in April 2024 at Tucson
  • DMLT virtual face-to-face next week (Tue, Wed)
• Fritz Mueller will be on vacation next week (Thir, Fri)

Release  qserv/lite-qserv:2023.10.1-rc1 was built last week

Igor Gaponenko tested at USDF. No new issues were seen.

Fritz Mueller on the Kubernetes deployment status

• the CI is failing on qserv-operator 
• waiting for a response from Fabrice Jammes
  • if not hear from Fabrice by the end of the day today then may have a look at this by myself

Context:

• DM-40891 - Getting issue details... STATUS

Status:

• completed
• an issue with a new version of libcurl addressed and worked around (still need to investigate it)
• waiting for Fritz Mueller to do the final review

A new release will be built after merging this PR into the main branch

John Gates Fritz Mueller and Igor Gaponenko:

• had a meeting to recap what was being discussed at SLAC in September (Frossie's and Colin's visit), including 4 major tracks:
  • ingesting user-generated data products (Frossie)
  • Qserv API for users (Colin)
    • user experience with Qserv to deal with long-running queries
    • query progress availability to users
    • query control
  • Qserv operational support (deployment and maintenance) in the long run (Frossie)
    • the same question applies to other databases
  • Qserv architecture & implementation
• agreed upon the next steps in modernizing Qserv. Considering a number of development (parallel) threads:
  • deploy and validate the file-base result delivery version at IDF (Kubernetes)
    • this will unblock a number of other developments that depend on this improvement
  • (from the CADC/TAP meeting) the developers started looking at adding support for the aync API
    • Presently it's sync JDBC
    • the developers are looking at various ideas on how to extend the TAP API
    • Fritz Mueller's proposal is to design an asyc API for Qserv that would be isomorphic to the UWS data model
      • more details are needed on this idea
  • discussed options for the new API to Qserv to replace mysql-proxy :
    • REST API
    • returning results as Parquet files (Frossie's favorite)
    • has to be designed to suit the needs of the TAP service
    • one option is to write a result set into a file at an Object Store and return back a URL to the result file residing at the store
      • advantages:
        • it will be up to the TAP to take care of storage resources, authorization, result lifecycle, and garbage collection of the results
        • reusing results (caching)
    • Fritz Mueller we may begin building the new infrastructure by setting up another container at Czar pod to be run in parallel with mysql-proxy. This would let us experiment with the new protocol.
• on the user-generated data products, there are two basic use cases
  
  • for ingesting small temporary tables before sending queries to Qserv. The tables would be fully replicated. And they could be JOIN-ed against the large catalog. The temporary tables would be deleted after finishing the corresponding queries.
  • for ingesting large long-lasting tables
    • these are most likely to be partitioned
  • what would be the ingest API?

Fritz Mueller 

• looking at an option to offload Andy Salnikov from this
• Started a discussion with Birwood