Database Meeting 2022-02-09

Takeaways from the last DMLT meeting:

• Operations meeting undergoing this week (Fritz Mueller will post a summary after it will be done)
• No specific concerns regarding Qserv
• More reviews of the project are coming this year
• PCW in August, possibly in-person (still not decided if)

Fritz Mueller any comments "from the top" on the last SLAC memorandum regarding the modified hybrid work arrangements?

The "Grand unification" of the Git packages:

• qserv_web has been merged
• removed numpy dependency
• Any further steps? 
  • Fritz Mueller we shall move qserv_operator & qserv_ingest into qserv in coordination with Fabrice Jammes. This could be done by the end of this week or earlier next week.
  • CI integration needs attention.
  • Configuration of the operator could be improved using new mechanisms implemented in the lite containers. Could be based on the Git SHA.
  • Fabrice Jammes need to commit to the operator code w/o triggering an automatic rebuild of the Qserv container.
  • Unknown User (npease) will look for a solution to restrict the scope of the Qserv container build to specific subfolders. To trigger a complete build, we shall make the PR on the branch.
  • Fritz Mueller a plan to integrate qserv_operator into qserv has been adopted!
  • Fabrice Jammes we're going to retain the original Git package of  qserv_operator for the history.
  • Fritz Mueller integrating qserv_testdata into qserv has one problem: 184 MB versus 15 MB in qserv alone. Another option would be to make qserv_testdata the Git submodule.
  • Fritz Mueller a plan to move qserv_testdata into qserv has been adopted.

Preparing the infrastructure for integrating Qserv documentation into the lite containers:

• Fritz Mueller on the status of the build container?
  • the container is working and the documentation is building.
  • still, some work is still needed. Fritz Mueller will make a PR for that this week.
  • this will be of benefit for the qserv_operator after it would be merged into qserv.
  • a nice thing about the new documentation building is that it's created as a collection of the HTML pages in the build directory which could be explored live if needed.
• Fritz Mueller has prepared a proposal for the structure of the documentation in Straw-man skeleton for the Qserv documentation.
  • Igor Gaponenko: automated injection of the Release Notes is required
  • Fritz Mueller proposed to do this by following special messages in the Git commit notes.
  • Igor Gaponenko would like to get a list of the relevant JIRA tickets
  • Andy Salnikov has provided a great example of how this kind of documentation looks like on the "Land of the Buttler": https://github.com/lsst/daf_butler/blob/main/doc/lsst.daf.butler/CHANGES.rst
  • Everyone should look at the documentation to see if the proposed documentation structure and make comments
• Igor Gaponenko asked if it would be possible to generate Doxygen-based documentation.
  • Fritz Mueller will make a separate PR for that.

Getting ready to vacate NCSA:

• Igor Gaponenko made a request to archive files to the cold storage at NCSA IHS-5897 - Getting issue details... STATUS
  • no progress on this has been made
  • need to get ping NCSA on the latest status of the request
• No specific plans for migrating these data to SLAC yet

DP0.2 preview ingests

• Igor Gaponenko have you had a chance to look into this yet?
  • No progress yet. This will be the highest priority project for me.

Fabrice Jammes will work on integrating the ASYNC-based ingest into qserv-ingest.

Fabrice Jammes reported good progress in migrating qserv-operator that uses entrypoints.

Fritz Mueller: we may retain the startup scripts (start.sh), however, the scripts should be trivial. No complex logic.

The context:

• #dm-qserv is a public channel meant to interact with Qserv developers and the rest of the DM team on various aspects of the software (development, deployment, bugs, etc.). The channel is presently filled with automated messages sent by GitHub (and the CI) on new PRs created and merged. NOTE  that these are presently only for the Git package qserv. The lifecycles of other packages (qserv-operator, qserv-ingest  are still "invisible".
• There is a proposal initiated by Kian-Tat Lim to move the automated messages into a separate channel. A link to a discussion can be found at https://lsstc.slack.com/archives/C2B70NXFD/p1643924617966479