Database Meeting 2016-02-03

Date

03 Feb 2016

Secondary index

rocksDB team very responsive
performance (after tuning ~comparable to innodb)
plan
- go with innodb
- sort input
- divide into small number of buckets (<100)
- load each bucket to table (< 1 billion per table)
- maintain mapping min/max --> table
the above should work well for user queries, up to 3 disk I/O in worst case (btree related) per access
loader: ok to assume we will have all related data (for a given patch) when loading, so no lookups needed (40b lookups in 24h is problematic)
potentially difficult use case: L3 table with objectIds corresponding to production objectIds, user wants to break it into chunks
- we can insist data is sorted, and sort-merge

Documentation

LDM-135 as is (L1, Qserv), plus SDQA, calibration db, efd db
new LDM "Data Access Design' (provenance, global metadata, butler, dax)
- Jonathan Sick will create placeholder for the new doc in github
- once we fill it with data, need TCT approval
the two above for documenting the design
for users, keep documentation with the product (inside modules)
reserve ~2 weeks at the very beginning of each cycle for documentation sprint. 1st one in March
keep updating baseline docs when doing actual work as part of the stories we work on

Data Access status and walkthrough (45 min) at JTM

S16 plan updates

SDQA

mariadbclient