Versions Compared

Old Version 14

changes.mady.by.user Igor Gaponenko

Saved on

compared with

New Version Current

changes.mady.by.user Igor Gaponenko

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

DiscussedItemWhoNotes
(tick)Project news

Fritz Mueller 

  • mostly preparing for the upcoming reviews
  • reminder: DM JTM will be in Chile on March 13-17, 2023
(tick)RefMatch tables support in Qserv team

Fritz Mueller:

  • Qserv may have a bug in the query rewrite for the RefMatch tables when querying the table directly in the SELECT queries
  • The investigation will require attaching gdb  to test the RelationGraph 
  • status of MatchesTruth  in IDF?

Fritz Mueller and Colin Slater think the table should not be (at least temporarily, before the bug got fixed) directly exposed in the TAP service. In the meantime, we would be looking at fixing the bug. Another reason not to expose the table is that some of its content may be confusing to users. Some columns make sense in Parquet but not in Qserv.

Fritz Mueller suggested that we might drop some confusing columns. Colin Slater says this could be done at the TAP schema level.

Colin Slater the ETA for getting the TAP schema description for the table this table in IDF is roughly next week.

Igor Gaponenko updates on the R-I system support and the integration tests

  • Jira
    serverJIRA
    serverId9da94fb6-5771-303d-a785-1b6c5ab0f2d2
    keyDM-35578

Fritz Mueller: in order to improve the test coverage of Qserv we need:

  • self-diagnostic on the startup
  • self-contained diagnostic dataset testing for other services of the upper stack (TAP, RSP, etc.). Gregory and Frossie would like to have the one
  • Colin Slater might be interested in handling this

Igor Gaponenko mentioned an idea to make the test catalog synthesizer that would specifically target Qserv functionality

Fritz Mueller: IN2P3 is still waiting before TruthMatches  gets validated by loading this table and the new version of ForceObjectOnDiaObject.

Fritz Mueller suggested building the new Qserv release after Igor Gaponenko merges the RefMatch  improvement PR into the main branch (this is expected to be done by the end of the day today). Then Fabrice Jammes would be able to build another release of qserv-ingest  to extend the integration test to cover the RefMatch support.

(tick)Status USDF/S3DF

Fritz Mueller:

  • NCSA is now officially shut down for LSST
  • all data have been migrated to SLAC
  • as of the last Monday we have joined USDF + UKDF + FrDF meetings  meetings 
  • NCSA negotiating with SLAC terms for moving 4 master nodes of the former Qserv clusters to SLAC. The machines are planned to be used as the development platform by the DAX team. A problem is that the hardware warranty for machines has expired. The ETA for making the machines available at SLAC is unknown.

Igor Gaponenko on status of Qserv at SLAC:

  • No significant progress as I was busy with the code development and working on overhauling the integration tests
  • Unpacked and deployed the data at all workers and the master node
  • started making adjustments to the Qserv configs (the "igor" mode)
(tick)ObsCore  Andy Salnikov 

Still working on the stand-alone tool (the task) which needs to be incorporated into the Registry.

No views due to various performance and synchronization complexity issues. Ended up creating the real table in PostgreSQL. The table will get periodically updated. The table will be served via the TAP service. GIS is not quite suitable for implementing spatial operation on the catalog's data since it's not. For now, the intent is to use: https://pgsphere.github.io/

(tick)qserv-ingest , qserv-operator 

Fabrice Jammes 

Just made the PR to be reviewed by Andy Salnikov since it has a lot of Python code.

Made improvements to the configurations using ConfigMaps and the lite  container's entry points.

The operator now generates the configs and injects those into ConfigMaps.

Fritz Mueller:

  • it would be better to bring the operator into the same Git package where we have Qserv
  • then we will have the default configs used by the operator in the same source tree 
(tick)The OR  bug in the Qserv parser/analyzer

Colin Slater Fritz Mueller 

Context:

Fritz Mueller is going to investigate this and fix the one if possible

...