Skip to end of metadata
Go to start of metadata

Date

Attendees

Goals

  • Get together, resolve bottlenecks, make progress faster, plan, design, exchange ideas

Discussion items

Problems with linking (pthread library) seen by Vaikunth on 14.04 (running in VM)

  • Fabrice will look into after he arrives at SLAC (tomorrow)

Project for Vaikunth (Vaikunth is joining our group through subcontract as a grad student, till ~ end of May)

  • improving integration tests, ultimate goal: running multiple queries on multi-node Qserv
  • worth for Vaikunth to check out DM-595 (multi-node)

  • worth for Vaikunth to implement DM-658 first
  • then try implementing running multiple queries on single node, 

  • then multi-node qserv. Note, this will need distributed loader

  • also, now output is hard to capture, optimize it, maybe generate some summary table what queries has run, what succeeded, what failed etc

    • Captured in the new story: DM-1390

DM-513

  • discussions through comments attached to this story in JIRA went well beyond the scope of this ticket
  • will capture in separate epic(s) and stories

CSS-related discussion

  • L3 use case: 
    • user creates table, then inserts into it using "insert into select from"
    • czar should detect this type of queries and act accordingly
    • most likely redirect to loader-like tool
  • deletes
    • dangerous, if we mess up, can wipe out entire production database

    • don't just automatically delete when something not found in metadata

    • consider smart "janitor" that is responsible for deleting
  • usefulness of watchers
    • watcher is unreliable, because of network glitches. Note, watcher receive alerts asynchronously, and they can get lost without anyone noticing

    • so to compensate for that, we would have to poll often to look for things watcher might have missed

    • so is watcher really useful?

    • perhaps just have a tool, like the loader, which would load (and delete?). A.la. qserv_admin that we have now, except the one we have now only deals with metadata operations (and watcher executes the actual actions)
  • uuids
    • have them in CSS already, need to start leveraging
    • need to find a way to compress uuids into shorter string that is compatible with mysql db/table naming convention. Plain uuid is 48 bytes, way too long, if we combine with user-defined table name, we will hit 64 bytes mysql limit

    • maybe use base64, but we want to avoid having to quote db/table names
    • opened DM-1391

Loader

  • In W15, don't deal with L3 use cases
  • Loader will do database/table creation "directly", not through any watcher
  • Use case to support:

    • Object table 3 chunks: A, B, C. Create Source, but it only has data in chunks A and C

    • Note, unless we create an empty chunk B for Source, we can't join Object with Source!

  • Another use case:
    • as above, and now we are loading more data to the Object table, the data ends up in chunk D

    • Note, now loader will need to create chunk D for Source table (and any other table that is partitioned the same way as Object table)

  • Thoughts about empty chunk-tables
    • idea: don't create empty chunk tables, and treat non-existing chunk table as a table that has no data in it

      • but we will need it to support joins between multiple distributed tables

      • another tricky part: we won't have schema for results

      • so that won't work
    • optimization idea: avoid dispatch for the joining chunks that have no data

      • but we still need to create empty tables, because of left joins

DM-1249

  • split into two tickets, one for this month, one for next month. Unknown User (danielw) will do it (before Friday)

DM-1210

  • unplanned (but very useful) work. In review now
  • ideally that is part of Refactor Qserv epic (DM-1030), which we haven't started yet

DM-631 (new geom code)

  • ideally, would like to reuse this code inside scisql
  • which means it should not be part of (proposed by Jim) "base" package