LSST Database Hangout 2014-10-15

Date

15 Oct 2014

Get together, resolve bottlenecks, make progress faster, plan, design, exchange ideas

Problems with linking (pthread library) seen by Vaikunth on 14.04 (running in VM)

Project for Vaikunth (Vaikunth is joining our group through subcontract as a grad student, till ~ end of May)

improving integration tests, ultimate goal: running multiple queries on multi-node Qserv
worth for Vaikunth to check out DM-595 (multi-node)
worth for Vaikunth to implement DM-658 first
then try implementing running multiple queries on single node,
then multi-node qserv. Note, this will need distributed loader
also, now output is hard to capture, optimize it, maybe generate some summary table what queries has run, what succeeded, what failed etc
- Captured in the new story: DM-1390

DM-513

discussions through comments attached to this story in JIRA went well beyond the scope of this ticket
will capture in separate epic(s) and stories

CSS-related discussion

L3 use case:
- user creates table, then inserts into it using "insert into select from"
- czar should detect this type of queries and act accordingly
- most likely redirect to loader-like tool
deletes
- dangerous, if we mess up, can wipe out entire production database
- don't just automatically delete when something not found in metadata
- consider smart "janitor" that is responsible for deleting
usefulness of watchers
- watcher is unreliable, because of network glitches. Note, watcher receive alerts asynchronously, and they can get lost without anyone noticing
- so to compensate for that, we would have to poll often to look for things watcher might have missed
- so is watcher really useful?
- perhaps just have a tool, like the loader, which would load (and delete?). A.la. qserv_admin that we have now, except the one we have now only deals with metadata operations (and watcher executes the actual actions)
uuids
- have them in CSS already, need to start leveraging
- need to find a way to compress uuids into shorter string that is compatible with mysql db/table naming convention. Plain uuid is 48 bytes, way too long, if we combine with user-defined table name, we will hit 64 bytes mysql limit
- maybe use base64, but we want to avoid having to quote db/table names
- opened DM-1391

Loader

In W15, don't deal with L3 use cases
Loader will do database/table creation "directly", not through any watcher
Use case to support:
- Object table 3 chunks: A, B, C. Create Source, but it only has data in chunks A and C
- Note, unless we create an empty chunk B for Source, we can't join Object with Source!
Another use case:
- as above, and now we are loading more data to the Object table, the data ends up in chunk D
- Note, now loader will need to create chunk D for Source table (and any other table that is partitioned the same way as Object table)
Thoughts about empty chunk-tables
- idea: don't create empty chunk tables, and treat non-existing chunk table as a table that has no data in it
  - but we will need it to support joins between multiple distributed tables
  - another tricky part: we won't have schema for results
  - so that won't work
- optimization idea: avoid dispatch for the joining chunks that have no data
  - but we still need to create empty tables, because of left joins

DM-1249

split into two tickets, one for this month, one for next month. Unknown User (danielw) will do it (before Friday)

DM-1210

unplanned (but very useful) work. In review now
ideally that is part of Refactor Qserv epic (DM-1030), which we haven't started yet

DM-631 (new geom code)