Date

Attendees

Fritz Mueller, Vaikunth Thukral, Unknown User (npease), Andy Salnikov, Fabrice Jammes, John Gates, Igor Gaponenko

Discussion items

Qserv

  • Czar config bug is no longer reproducible on new stack.  Closed for now; will have to keep an eye out for this.
  • "Snail scan" scheduler added to shared-scan system.
  • John now working on adding code to track stats for user queries on workers; to be used to evict slow-running queries from scans.
  • John noticed an overnight cluster failure at IN2P3, similar to failures which Vaikunth has been experiencing.

L1/AP Database

  • Andy S. is working on input event simulator.  Can now generate DiaSources and DiaObjects and store them, then run an initial set of queries.
  • Using HTM indexing code in sphgeom off one of Serge's development branches.  Serge anticipates merging this branch by the end of the month.
  • Running slowly on VMs right now.  Andy is adding code for query performance analysis (SQL EXPLAIN, timestamps in logging) to get a grip on this.

PDAC

  • Igor got a monolithic stack set up for beginning loader experimentation.
  • Igor has been familiarizing himself with the Summer '15 SDSS Stripe 82 data set.  Schema seems somewhat different from current published LSST survey schema.  Which is to be authoritative for the PDAC?
  • Fabrice to set up a small multi-node cluster for Igor at NCSA using new tools/scripts, so Igor can run loader experiments without perturbing Vaikunth or John on the IN2P3 clusters.

Butler

  • Repository refactor and persistence improvements are in review, reviewed except unit tests, Jim Bosch to review by end of week.
  • Nate has begun to look at the research & design task for object storage and format epic.

X-SWAP

  • Vaikunth ran some low-volume query tests, but they did not seem to complete in reasonable time.  Additionally, some strange Elastic Search behaviors on these runs (<.5 number of samples collected).  Vaikunth has started a dialog with the IN2P3 admins to investigate these behaviors as well as the strange memory allocation numbers reported last week.
  • Vaikunth has also been prepping slides for the X-SWAP all-hands, which is Thursday, 14 Jul.  Vaikunth and Fritz to attend (Berkeley)

Docker

  • Docker tickets covering Swarm proof-of-concept and container infrastructure documentation are up for review.
  • Swarm proof-of-concept is now using outdated Swarm; new release looks like it will simplify things (fewer pre-reqs).  Not a lot of work to upgrade, but maybe wait for next release.
  • Investigation of large (1.5G) container size: does not seem unusual given what's included.  Mariadb, for example is 600Mb on its own.   Should be moved to its own container as a first step.
  • Documentation on building release containers was updated.

Misc

  • Brian and Fabrice are on vacation this week