Date

May 14, 2014

Attendees

Goals

  • Discuss blockers, short term plan and design

Discussion Items

DM-510

  • remove objIdIndex
  • rename drivingTable and keyColName to dirTable, dirColName (dir == director)

Unassociated sources

  • special case, complicates design
  • we need an estimate of the count
  • options: 
    • dummy object for each unassociated source (problem: Object table grows)
    • one dummy object per chunk for all unassociated sources in a chunk (problem: complicates design elsewhere)
    • maintain a separate table for all unassociated sources (preferred solution for now)
  • Open a new issue (Jacek)

Doing subchunking with overlap

  • for background, see Daniel's email sent to qserv-l 5/3/2014
  • Write short description why we need subchunking (Daniel)
  • Decide in ~Sept, maybe we can outsource this problem to mysql developers, after all, subchunking = optimization needed because optimizer is dumb 
  • and ask Monty how hard

DM-666

  • Serge wrote a patch
  • Fabrice tested, made one tiny fix, packaging now

DM-220

  • Daniel found memory leak
  • Add comments what we want AndyH to implement/fix (Daniel)

DM-195 (logging)

  • finished another iteration
  • better docs on trac
  • few more features suggested by K-T, biggest item is: want both boost and printf, question: do we want
  • to maintain both
  • printf is much faster than boost (based on tests)
  • playing with speeding up boost version
  • one super wrapper? No
  • For packaging:
    • need to change namespaces
    • look at the partition.git
    • push to contrib (Bill), then Serge will take a look
    • package name: "logging"? Double check with Mario Juric and Kian-Tat Lim
    • expecting to have it packaged by next meeting

DM-70

  • Trying to decide how much to rewrite
  • Daniel wants to hear what was ugly about existing code eg AsyncQueryMgr

DM-365

  • would like to rename names, but eups package must have the same name as git repo
  • wants: qservdata --> qserv_testdata
  • find out if it is ok to rename report, and how easy (Jacek)
  • qserv should not depend on qservdata

DM-648

  • It is related to DM-481
  • Discussed last wee and assigned to Andy Salnikov, Fabrice Jammes wants to have a look and will try to implement

DM-516

  • see comments, just submitted (no locking guarantee, consider using kazoo locking, removing KVInterface)

DM-198

  • progress, but not as far as planned

DM-626

  • "distinct" was never handled
  • "order by" - old parser passed everything

DM-207

  • divide into: 
    • migrate Qserv into the new logging system, remove old logging (2 weeks)
    • polishing what messages are printed, debug levels etc

New issues opened since last meeting:

  • DM-661 Parser has inverted order for "limit" and "order by"
  • DM-659 running individual query in the automated test suite
  • DM-658 automated tests should use their own dedicated database, not "LSST"
  • DM-657 qserv initialization broken when previously registered database does not exist
  • DM-656 selecting from non-partitioned table is broken
    • was table unpartitioned?Jacek will check
  • DM-655 unknown column derails Qserv
    • squashing will be rewritten in DM-70
  • DM-664 "out of range value" message when running qserv-testdata (loader.py)
  • DM-666 DM-366 partition package should use sconsUtils to detect eups-related boost

Action Items