Skip to end of metadata
Go to start of metadata



Discussion items

Secondary index

  • rocksDB team very responsive
  • performance (after tuning ~comparable to innodb)
  • plan
    • go with innodb
    • sort input
    • divide into small number of buckets (<100)
    • load each bucket to table (< 1 billion per table)
    • maintain mapping min/max --> table
  • the above should work well for user queries, up to 3 disk I/O in worst case (btree related) per access
  • loader: ok to assume we will have all related data (for a given patch) when loading, so no lookups needed (40b lookups in 24h is problematic)
  • potentially difficult use case: L3 table with objectIds corresponding to production objectIds, user wants to break it into chunks
    • we can insist data is sorted, and sort-merge


  • LDM-135 as is (L1, Qserv), plus SDQA, calibration db, efd db
  • new LDM "Data Access Design' (provenance, global metadata, butler, dax)
    • Jonathan Sick will create placeholder for the new doc in github
    • once we fill it with data, need TCT approval
  • the two above for documenting the design
  • for users, keep documentation with the product (inside modules)
  • reserve ~2 weeks at the very beginning of each cycle for documentation sprint. 1st one in March
  • keep updating baseline docs when doing actual work as part of the stories we work on

Data Access status and walkthrough (45 min) at JTM

S16 plan updates

  • add ~60 SP documentation sprint
  • add ~5 SP for integrating imgserv with panstarrs data
  • DM-4884 and DM-4790 go to W17, shrink DM-2881


  • keep sdqa locally within each database
  • plus one global that has statistics about all local sdqa
  • implementation mostly by square, consult with Data Access as needed


  • in the process of switching all stack from mysqlclient to mariadbclient