Date
Attendees
John Gates, Fritz Mueller, Vaikunth Thukral, Brian Van Klaveren, Andy Salnikov, Serge Monkewitz, Fabrice Jammes, Andy Hanushevsky, Jacek Becla
Kian-Tat Lim for part of the meeting
Discussion items
panstarrs data
- available ~Dec
- ~150 TB db size (ideally need ~2x storage)
- ideally want that data distributed, on local disks for loading
- fast inter-network for shuffling would be best
- would be good to switch to MariaDB mysql (DM-224, maybe next month?) prior to loading data
data compression in db
- note, mysql decompresses per query, we'd need to write much more complex shared scans code to do one decompression for all queries that access data
- might be useful to see if that can be fixed inside mysql (at the storage engine level)
- best not to compress unless data compresses really well (3x or so), and/or data is very rarely used, because otherwise decompression cost will likely outweigh value of compression
- revisit trade-offs of compression (DM-4064)
new xrootd build
- best to wait few more days and then update our release, Andy is releasing all the latest bits
mock library
- deal with python and c++ mocks separately
- in python, there is "mock" library in python 3. Can't switch dax_* because dax_imgserv depends on afw (all the rest easy to migrate to python 3)
- only need the dependency on the mock library for testing, but that is hard to do in scons
- use anaconda for now (mock is part of anaconda)
clean xrootd shutdown
- two driving forces: proper clean up of mysql in-memory tables, and valgrind false-positives
- we have to deal with unexpected crashes, so need to have robust solution for the former anyway
- in C++11 impossible to wake up and stop a thread, lots of rework in xrootd to do clean shutdown
- not worth
- might be useful to have integration test to stress the unclean shut down of xrootd
docker
- Fabrice planning to experimentally release qserv release inside docker containers each month
parser
- need to revisit the code around parser (intermediate representation in particular)
Fabrice's visit at slac
- tickets purchased, Tue Nov 3 – Thu Nov 12
- perhaps Joshua Hoblitt could come to slac for Thu-Fri (Nov 5-6)? - check with Frossie Economou