Date
Attendees
Fritz Mueller, Andy Hanushevsky, Vaikunth Thukral, Unknown User (npease), Andy Salnikov, Fabrice Jammes, John Gates, Brian Van Klaveren, Igor Gaponenko
Discussion items
Welcome Igor Gaponenko!
L1/AP Database
- Code to instantiate an L1 database based on the proposed schema has been completed.
- Working with this now to create a simulated data stream and populate a test database.
- Need to talk to Serge Monkewitz to find out status of HTM indexing extension to sphgeom.
Docker
- Swarm/OpenStack bug found: need to shutdown VMs before snapshotting, not just wait for cloud-init completion
- Fabrice waiting for NCSA OpenStack to come back up to finish testing of his Swarm rebase now that bug addressed
- Fabrice working on Docker usage tech note
- Fabrice curious about benefits of microservices. Fritz and Brian to investigate
PDAC
- Brian is working on containerizing DAX services. Needs full LSST stack base container, because DAX services need Butler.
- Found some issues with scripts along the way (non-existent jre package on debian, build fails on VM <2GB RAM, etc).
- CI is building stack images, but they are really large (2-3GB). What is going on? Needs to be investigated.
- Probably we will end up with separate base containers for Qserv and DAX because Qserv has a lot fewer stack package dependencies
- Working out how configuration for DAX service containers should best be done
- Igor has talked with K-T and Greg to track down the SDSS Stripe82 catalog load data. Some is at NCSA, but some also at IN2P3?
- Igor planning to move a part of the data to IN2P3 to start experimenting with the loader script.
Shared-Scans
- "Snail-scan" added, which will end up executing queries expected to exceed the scan period of all other scans. Code still needed to feed this scan.
- John is testing/validating memman changes from this week, concurrent with ongoing review.
- John will be adding some plumbing which tracks statistics on tasks.
Butler
- Nate has two tickets wrapping up review. Needs to disentangle some commits, and move some work to follow-up stories.
X-SWAP
- Vaikunth showed some baseline graphs of memory, IO, CPU usage on an idle cluster. Memory stats look anomalous. Investigating
- Tried to gather 24hr graphs under load from ruqueries.py, but services crashed after 14hrs. Also needs to be investigated.
- John has added the requested query begin/end messages to qserv logging
Documentation
- Brian working on DAX API docs, has run into many tooling issues re. Sphinx and CI. Next stage is to get it working with Travis CI.
- Nate will be updating LDM-463 draft branch as part of his upcoming Butler commits.
Misc
- John investigating czar-config crasher. Looks to be related to log4cxx; need to build out-of-stack.
- Andy H reports 4.4 release of XRootD looking likely in the next few weeks. After this, xrdssi branch can be merged to master.
- Andy H working on XRootD logging enhancements as well.