Skip to end of metadata
Go to start of metadata

Date

Attendees

Discussion items

panstarrs data

  • available ~Dec
  • ~150 TB db size (ideally need ~2x storage)
  • ideally want that data distributed, on local disks for loading
  • fast inter-network for shuffling would be best
  • would be good to switch to MariaDB mysql (DM-224, maybe next month?) prior to loading data

data compression in db

  • note, mysql decompresses per query, we'd need to write much more complex shared scans code to do one decompression for all queries that access data
  • might be useful to see if that can be fixed inside mysql (at the storage engine level)
  • best not to compress unless data compresses really well (3x or so), and/or data is very rarely used, because otherwise decompression cost will likely outweigh value of compression
  • revisit trade-offs of compression (DM-4064)

new xrootd build

  • best to wait few more days and then update our release, Andy is releasing all the latest bits 

mock library

  • deal with python and c++ mocks separately
  • in python, there is "mock" library in python 3. Can't switch dax_* because dax_imgserv depends on afw (all the rest easy to migrate to python 3)
  • only need the dependency on the mock library for testing, but that is hard to do in scons
  • use anaconda for now (mock is part of anaconda)

clean xrootd shutdown

  • two driving forces: proper clean up of mysql in-memory tables, and valgrind false-positives
  • we have to deal with unexpected crashes, so need to have robust solution for the former anyway
  • in C++11 impossible to wake up and stop a thread, lots of rework in xrootd to do clean shutdown 
  • not worth
  • might be useful to have integration test to stress the unclean shut down of xrootd

docker

  • Fabrice planning to experimentally release qserv release inside docker containers each month

parser

  • need to revisit the code around parser (intermediate representation in particular)

Fabrice's visit at slac