Andy Salnikov has proposed adding the integration test for the RelationGraph
Igor Gaponenko should add support for the RefMatch tables to the Replication/Ingest system
Fritz Mueller will extend the integration test to support RefMatch tables
Colin Slater: status of the input data of theForcedSourceOnDiaObjecttable?
is about to run the final fixup stage
the data will be ready within a timeframe of 1 day
the number of the Parquet files will be greatly reduced (no splitting tracts to patches) which would help a lot with speeding up the PArquet-to-CSV translation
Igor Gaponenko: further improvements to the Replication/Ingest system based on the DP02 experience
Colin Slater: need to improve the monitoring of the Qserv usage in IDF
Colin Slater proposed to pull metrics from Qserv and ship them to Google for monitoring, aggregation, and visualization
Fritz Mueller right now we are not logging anything from Qserv since it's too expensive. So, we don't even have the searchable log
Development infrastructure for Qserv
Igor Gaponenko: need a replacement for Qserv development platform that we're losing at NCSA in 1 month (the August 15th deadline)
still, no progress on the temporary instance based on the loaner hardware at SLAC
the IT is quite busy working on the USDF infrastructure
Fritz Mueller will work with SLAC IT and Richard to accelerate this. Also, in September we are supposed to have 4 former master[1-4]machines as the temporary development platform. Aa the meantime, the IDF development cluster (qserv-dev?) is the only option.
NOTE: the topic will continue to be discussed next week after Fabrice Jammes will get back from vacation.
Igor Gaponenko: extending the versioning mechanism in REST API of the Replication/Ingest and the implementation. This requires making changes to the JSON config files (to store the version numbers) and the ingest workflows (including qserv-ingest). There is the PR on the Git package qserv-ingest in the JHIRA ticket mentioned below.
The CI is failing on the PR due to Repl service version incompatibility. Two PRs exist in the scope of the same JIRA ticket. Both migrated to version 9 simultaneously.
Do we need the common Python API for the Replication/Ingest system? We have presently 2 separate implementations: one in the Qserv container's CLI (source path src/admin/python), and the other one in the GIt package qserv-ingest (source path rootfs/python). This seems like a duplicate effort.