...
- two use cases: user queries, and data loader
- for data loader, need to load 40B in 24h, so 1/2 million/sec
- build index locally on each worker (one index for all chunks), preferably serve it from RAM, or at least SSD
- serve it through xrootd, or memcached
- Reddis might be an option, it has some extra smarts eg for compressing ranges
- for memcached
- put o(1000) elements per bucket to fill the pages we transfer through network
- we could further rely on replication to speedup recovery
- for loader use case, preload cache for area that loader works on
- conclusions: try xrootd and memcached at in2p3 cluster and see what throughput we could get
- related epic:
Jira server JIRA serverId 9da94fb6-5771-303d-a785-1b6c5ab0f2d2 key DM-2119