Skip to end of metadata
Go to start of metadata

Date

Attendees

Goals

  • Please, add topics if you have any!

Discussion items

TimeItemWhoNotes

Project newsFritz Mueller 

Takeaways from the last DMLT meeting:

  • Operations meeting undergoing this week (Fritz Mueller will post a summary after it will be done)
  • No specific concerns regarding Qserv
  • More reviews of the project are coming this year
  • PCW in August, possibly in-person (still not decided if)

Fritz Mueller any comments "from the top" on the last SLAC memorandum regarding the modified hybrid work arrangements?


Progress on topics discussed at the previous meeting Database Meeting 2022-02-02

The "Grand unification" of the Git packages:

  • qserv_web has been merged
  • removed numpy dependency
  • Any further steps? 
    • Fritz Mueller we shall move qserv_operator & qserv_ingest into qserv in coordination with Fabrice Jammes. This could be done by the end of this week or earlier next week.
    • CI integration needs attention.
    • Configuration of the operator could be improved using new mechanisms implemented in the lite containers. Could be based on the Git SHA.
    • Fabrice Jammes need to commit to the operator code w/o triggering an automatic rebuild of the Qserv container.
    • Unknown User (npease) will look for a solution to restrict the scope of the Qserv container build to specific subfolders. To trigger a complete build, we shall make the PR on the branch.
    • Fritz Mueller a plan to integrate qserv_operator into qserv has been adopted!
    • Fabrice Jammes we're going to retain the original Git package of  qserv_operator for the history.
    • Fritz Mueller integrating qserv_testdata into qserv has one problem: 184 MB versus 15 MB in qserv alone. Another option would be to make qserv_testdata the Git submodule.
    • Fritz Mueller a plan to move qserv_testdata into qserv has been adopted.

Preparing the infrastructure for integrating Qserv documentation into the lite containers:

  • Fritz Mueller on the status of the build container?
    • the container is working and the documentation is building.
    • still, some work is still needed. Fritz Mueller will make a PR for that this week.
    • this will be of benefit for the qserv_operator after it would be merged into qserv.
    • a nice thing about the new documentation building is that it's created as a collection of the HTML pages in the build directory which could be explored live if needed.
  • Fritz Mueller has prepared a proposal for the structure of the documentation in Straw-man skeleton for the Qserv documentation.
  • Igor Gaponenko asked if it would be possible to generate Doxygen-based documentation.

Getting ready to vacate NCSA:

  • Igor Gaponenko made a request to archive files to the cold storage at NCSA IHS-5897 - Getting issue details... STATUS
    • no progress on this has been made
    • need to get ping NCSA on the latest status of the request
  • No specific plans for migrating these data to SLAC yet

DP0.2 preview ingests

  • Igor Gaponenko have you had a chance to look into this yet?
    • No progress yet. This will be the highest priority project for me.

Fabrice Jammes will work on integrating the ASYNC-based ingest into qserv-ingest.


Status and development of the lite  container and the entrypoint-based configuration and management of Qserv

Fabrice Jammes reported good progress in migrating qserv-operator that uses entrypoints.

Fritz Mueller: we may retain the startup scripts (start.sh), however, the scripts should be trivial. No complex logic.

A discussion on the Kubernetes init containers:

Fritz Mueller: we need to investigate the possibility of replacing docker-compose with kind  and minikube.


Shall we divert automated Git messages from the Slack channel #dm-qserv to some other channel?

The context:

  • #dm-qserv is a public channel meant to interact with Qserv developers and the rest of the DM team on various aspects of the software (development, deployment, bugs, etc.). The channel is presently filled with automated messages sent by GitHub (and the CI) on new PRs created and merged. NOTE  that these are presently only for the Git package qserv. The lifecycles of other packages (qserv-operator, qserv-ingest  are still "invisible".
  • There is a proposal initiated by Kian-Tat Lim to move the automated messages into a separate channel. A link to a discussion can be found at https://lsstc.slack.com/archives/C2B70NXFD/p1643924617966479

Questions:

  • Shall we go for the separation, and if so then what should still be left in #dm-qserv?
  • What channel would be good for the GitHub messages? Shall we use an existing channel #dm-dax-builds?
  • What about other relevant Git packages that are still in the "stealth mode"?

Proposals after discussing the topic:

  • Leave the channel as it is now.
  • Unknown User (npease) wants to see an automated message when a build of the main branch is falling. Send it to the team's room #dm-db-team? 

Action items

  •