Skip to end of metadata
Go to start of metadata



Discussion items

1 min round table status

  • AndyS waiting for review (DM-3574), Serge will do soon
  • Brian writing requirements about authorization/authentication. Create story to capture the work
  • Vaikunth playing with elk + our log files. Debugging problem with services failing after they run for long time
  • John working on thread poll / scheduling
  • Nate: getting metadata out of fits file. Need to talk to AndyH about css story
  • Fritz: dht prototyping, demo likely end of this month. Helping AndyH
  • AndyH getting up to speed with running qserv
  • Mike Kelsey (working with us @50% for 3 months, possibly longer): started looking at objectId/chunk index optimizations.
  • Jacek: provenance prototype

ObjectId / Chunk index discussion

  • two use cases: user queries, and data loader
  • for data loader, need to load 40B in 24h, so 1/2 million/sec
  • build index locally on each worker (one index for all chunks), preferably serve it from RAM, or at least SSD
  • serve it through xrootd, or memcached
  • Reddis might be an option, it has some extra smarts eg for compressing ranges
  • for memcached
    • put o(1000) elements per bucket to fill the pages we transfer through network
    • we could further rely on replication to speedup recovery
  • for loader use case, preload cache for area that loader works on
  • conclusions: try xrootd and memcached at in2p3 cluster and see what throughput we could get
  • related epic:  DM-2119 - Getting issue details... STATUS