Skip to end of metadata
Go to start of metadata

Date

Attendees

Notes from the previous meeting:

Discussion items

DiscussedTopicNotes
(tick)Project news

Fritz Mueller:

  • At the last Joint Data Facilities Meeting - 2022-11-28 there was a discussion on the commissioning replan (ComCam)
  • The main telescope gets delayed till the end of 2024
  • ComCam commissioning is getting cut from the schedule
  • everything is now planned Fall 2024
  • data preview (DP1) was based on ComCam
  • DP1 is now in an undetermined state. The project is looking at other options.
  • Challanges for Qserv:
    • a lack of a sufficiently large dataset that would be closer to DR1 in size and complexity
    • Colin Slater there is a possibility of generating a (significantly) larger (~1000 sq degrees) version of DP0.2 that won't be published for general use. It would be good for testing the pipeline and Qserv. Note that DP0.2 has ~300 sq degrees. LSST will have 18,000 sq degrees.
  • On the USDF hardware:
    • delayed from November till January
    • considering staged delivery

On moving offices from B50 to ROB:

  • agreed on the first-floor office
  • realistically speaking, the move won't happen before mid-January 
  • a separate conference room 
(tick)User-generated data products

Fritz Mueller:

Architecture: independent Python service standing in between TAP and Qserv (Ingest system)

Constraints: the SRE team doesn't have spare resources to modify the TAP service. So, we would have to deal with whatever data format to deal.

We may need to have a local (at SLAC) TAP expert to complement SRE's effort in the context of USDF.

(tick)A bug in Qserv czar when handling failed chunk queries

 Context:

  • the topic was initially discussed at Database Meeting 2022-11-09
  • focus on improving the worker and czar  monitoring to investigate the origin of the problem
  • on-going efforts:

Fritz Mueller made an interesting observation:

  • Certain queries that seem problematic might have 5,000 tasks still queued at workers.
  • However, the query history was completed czar for those. No error conditions were reported. Yet, the queries are still hanging. Is it possible that we might have "zombie" tasks at Qserv workers? Or, is there some missing/incorrect bookkeeping for those?
  • Is there a problem with thread allocations?

Fritz Mueller is planning to build a new Qserv release based on the latest improvements.

Fritz Mueller would like to see extended query info (including templates) on the Web Dashboard.

Igor Gaponenko will work on it ASAP. Perhaps, this feature could be included in the new Qserv release?

(minus)

(postponed)

qserv-operator and qserv-ingest

Any news? Any progress for the proposed Parquet-to-CSV translator?

(tick)Status of the ObsCore tables.

Andy Salnikov :

  • PostrgeSQL + pgSpehere has been installed at SLAC (USDF). It's been tested. It seems to work. This is the only production instance of PostgreSQL that we have. Need to agree on the configuration of live ObsCore table. It's going to up updated as new data will be received from the Summit. The ball is now in the GPDF's department. Need feedback from him before moving on.

Action items

  •