Date

Attendees

Notes from the previous meeting

Discussion items

DiscussedItemWhoNotes
(tick)Project news

Fritz Mueller Colin Slater

No significant project news due to the Labour Day holiday
(tick)RefMatch tables support in Qserv

team


Status:

Fritz Mueller?

  • Anything else here?
  • What are the next steps?
  • Should we give the green light to our colleagues at FrDF to ingest this table into their Qserv instances?

Fabrice Jammes?

  • Would it make sense to try ingesting the final version of DP02 (including MatchesTruth) into qserv-dev at IDF using the latest version of qserv-ingest?

Colin Slater needs to update the Felis schema to allow the new version of the table in TAP

Fritz Mueller has investigated the problem of the extra condition clause generated by RelationGraph for queries that directly query the RefMatch  tables. It seems to do the right thing. No bugs have been found. We shouldn't publish the special column flags not confuse users.

Colin Slater is still concerned about the case sensitivity of Qserv for queries that involve the object identifiers of the director tables.

Fritz Mueller to Colin Slater how urgent is deploying MatchesTruth in qserv-prod (IDF)?

  • Colin Slater is more interested in the conservative approach (the way it was done in -int)
  • Fritz Mueller says the new version of Qserv in -prod won't be available before Thursday
  • Igor Gaponenko it will take 1 or 2 days to ingest this table into Qserv

Fritz Mueller: the bottom line:

  • the RelationGraph is working
  • waiting for the TAP schema published
  • need to build a new Qserv release that fixes a few bugs and adds support for ingesting RefMatch tables
  • after that, we need to deploy this version in IDF

Fritz Mueller the final decision on ingesting MatchesTruth  into -prod is

(tick)Status of qserv-ingest and qserv-operator 

Fabrice Jammes Fritz Mueller 

Fabrice Jammes:

  • The new version of the Ingest system has been successful tests. The integration test passes. We are ready to release this version.
  • Thought, ingesting the new capability of the Replication/Ingest system version 9 that adds support for ingesting RefMatch tables still need to be tested

Fritz Mueller:

  • agreed that it would make sense to test the new version of qserv-ingest & qserv-operator  in qserv-dev  (IDF) after the integration test passes

Igor Gaponenko Please, note:

  • integration test case05  has been dropped from Qserv
  • integration test case03  has been extended to have 2 director tables and 1 RefMatch table

Fritz Mueller:

  • the new Qserv tag will be published
  • the new version of qserv-operator should be built on top of that to be ingested into IDF, elsewhere.
(tick)Qserv instance at USDF

team

Igor Gaponenko on the recent status

  • back to this development after finishing working on RefMatch  tables and another delayed development
  • ran into a few obstacles with the configuration of the cluster:
    • the cluster was intentionally cut off from the Internet (no outbound connections were allowed)
    • couldn't connect to GitHub 
    • couldn't pull anything from (or push into) the DockerHub 
  • reasonable workarounds were found with Yee's help
  • working on starting up Qserv

Fritz Mueller once the cluster is up:

  • ingest DP02 into the cluster
  • point the TAP service to the cluster

Igor Gaponenkoon the potential performance issues with ingesting large-scale catalogs into this instance:

  • we presently have data in the Google Cloud
  • we may see the poor performance since all communications with those data sources would have to go via the single proxy host a SLAC
  • I will experiment with the current setup to see the effect in the quantitative terms

Fritz Muelleron alternative options:

  • we don't have yet the Object Store at SLAC that would be the right answer
  • there is an option to pull the input data from Google Cloud into the shared filesystem at SLAC
  • though, the performance may be the problem here since we would need to pull over 30 TB of data and 1.7 million files.
(tick)ObsCore

Andy Salnikov 

Working on the implementation that is not touching spatial indexes yet. That would be done next.

Action items

  •