Fritz Mueller:

• At the last Joint Data Facilities Meeting - 2022-11-28 there was a discussion on the commissioning replan (ComCam)
• The main telescope gets delayed till the end of 2024
• ComCam commissioning is getting cut from the schedule
• everything is now planned Fall 2024
• data preview (DP1) was based on ComCam
• DP1 is now in an undetermined state. The project is looking at other options.
• Challanges for Qserv:
  • a lack of a sufficiently large dataset that would be closer to DR1 in size and complexity
  • Colin Slater there is a possibility of generating a (significantly) larger (~1000 sq degrees) version of DP0.2 that won't be published for general use. It would be good for testing the pipeline and Qserv. Note that DP0.2 has ~300 sq degrees. LSST will have 18,000 sq degrees.
• On the USDF hardware:
  • delayed from November till January
  • considering staged delivery

On moving offices from B50 to ROB:

• agreed on the first-floor office
• realistically speaking, the move won't happen before mid-January 
• a separate conference room 

Fritz Mueller:

• We need to make a sketch for the point-and-shoot TAP upload via some REST
• For the input data format, it's not clear if the data are in the eCSV or in the VO table format. This has to be clarified.
  • https://docs.astropy.org/en/stable/io/ascii/ecsv.html
  • https://www.ivoa.net/documents/VOTable/20130920/REC-VOTable-1.3-20130920.html

Architecture: independent Python service standing in between TAP and Qserv (Ingest system)

Constraints: the SRE team doesn't have spare resources to modify the TAP service. So, we would have to deal with whatever data format to deal.

We may need to have a local (at SLAC) TAP expert to complement SRE's effort in the context of USDF.

 Context:

• the topic was initially discussed at Database Meeting 2022-11-09
• focus on improving the worker and czar  monitoring to investigate the origin of the problem
• on-going efforts:
  • 
  • 

Fritz Mueller made an interesting observation:

• Certain queries that seem problematic might have 5,000 tasks still queued at workers.
• However, the query history was completed czar for those. No error conditions were reported. Yet, the queries are still hanging. Is it possible that we might have "zombie" tasks at Qserv workers? Or, is there some missing/incorrect bookkeeping for those?
• Is there a problem with thread allocations?

Fritz Mueller is planning to build a new Qserv release based on the latest improvements.

Any news? Any progress for the proposed Parquet-to-CSV translator?

Andy Salnikov :

• PostrgeSQL + pgSpehere has been installed at SLAC (USDF). It's been tested. It seems to work. This is the only production instance of PostgreSQL that we have. Need to agree on the configuration of live ObsCore table. It's going to up updated as new data will be received from the Summit. The ball is now in the GPDF's department. Need feedback from him before moving on.