Date
Nov 14, 2014
Official meeting notes
Official meeting notes are available here: 2014-11-14 Qserv Integration testbed @ CC-IN2P3
Attendees
- Yvan Calas
- Fabrice Jammes
Goals
- Exchange ideas in order to define a good deployment procedure of Qserv on CC-in2p3 DELL cluster
Discussion Items
Use RPM packaging in order to prepare a production-oriented way of installing Qserv on a cluster
- CC-IN2P3 wants to have a standard way of distributing Qserv, RPM is a mastered solution and is integrated with puppet
- Qserv team is interested in RPM packaging. Nevertheless LSST DM management standard is to use current Qserv eups-packaging for development purpose (Please note that current Qserv packaging is similar to LSST DM-stack packaging)
- Yvan agree to package Qserv in RPMs, and may use ccqserv003 in order to test current Qserv install procedure
- Fabrice Jammes can provide support on current install procedure to Yvan: http://lsst-web.ncsa.illinois.edu/~fjammes/qserv-doc/
- Yvan can also request support on LSST developper list: https://lists.lsst.org/mailman/listinfo/dm-devel
- Mario Juric may plan to add RPM-support to eups, so Yvan may contact him in order to implement RPM-support in an LSST standard way. So that it can be re-usable for the entire LSST DM-stack.
- Qserv team would like to have a flexible way to deploy (at least) new Qserv versions: Qserv RPM package must contains only Qserv binaries, and not the full Qserv distribution (ie. Qserv re-compiled dependencies: qserv-source-dependencies.txt should be in separate RPM(s)). Qserv source dependencies doesn't change very often.
Access to https://sw.lsstcorp.org/eupspkg/
- Qserv team uses ccqserv001 as a build machine in order to produce Qserv distribution binaries
- CC-IN2P3 will provide an access from ccqserv001 to LSST official repositories
Qserv developers requirement
- ssh access to Qserv cluster nodes
- switch easily Qserv versions on all the cluster, or on a subset of nodes
- apply a software patch quickly on one or several nodes (example: patched code with additional log messages while debugging on a given worker node, test a bugfix on a given node before deploying it to the whole cluster)
- Easily update a Qserv source dependency on one or several nodes (example: new version of xrootd or MySQL, MariaDB, ...)
- re-configure Qserv software manually on one or several nodes
- install debugging tools (editors, pdb, gdb, valgrind, ...) on one or several nodes
- debug code on a node
- access to a shared-file system which contains the input data (i.e. csv files) in order to load it in Qserv.
- CC-IN2P3 understand these requirements and will do its best to support them
Remarks about alternative install procedure proposed by the Qserv team
- this procedure may be used if RPM packaging is too complex or require too much work
- please note that this procedure is mastered by Qserv team as it is used on their development machines, and is compliant with Qserv developers requirements
- this procedure is quite similar to what Fabio has done with LSST DM stack on CernVM-FS, but using nodes local filesystems, instead of CernVM-FS to run the software stack
- Qserv team would be interested to use this procedure on the cluster with CernVM-FS, whereas this shouldn't be compliant with applying patch/debugging on a given node
- this procedure allows to run a LSST developer-friendly environment on each node. This is very convenient for development and testing purpose, but not compliant with sysadmin standards