Date: Thu, 28 Mar 2024 09:28:22 +0000 (UTC)
Message-ID: <1500529850.18892.1711618107886@confluence>
Subject: Exported From Confluence
MIME-Version: 1.0
Content-Type: multipart/related;
boundary="----=_Part_18891_463744036.1711618102876"
------=_Part_18891_463744036.1711618102876
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Content-Location: file:///C:/exported.html
Database Meeting 2015-10-21
Database Meeting 2015-10-21
Date
Attendees
Discussion items
1 min round table status
- AndyS waiting for review (DM-3574), Serge will do soon
- Brian writing requirements about authorization/authentication. Create s=
tory to capture the work
- Vaikunth playing with elk + our log files. Debugging problem with servi=
ces failing after they run for long time
- John working on thread poll / scheduling
- Nate: getting metadata out of fits file. Need to talk to AndyH about cs=
s story
- Fritz: dht prototyping, demo likely end of this month. Helping AndyH
- AndyH getting up to speed with running qserv
- Mike Kelsey (working with us @50% for 3 months, possibly longer): =
started looking at objectId/chunk index optimizations.
- Jacek: provenance prototype
ObjectId / Chunk index discussion
- two use cases: user queries, and data loader
- for data loader, need to load 40B in 24h, so 1/2 million/sec
- build index locally on each worker (one index for all chunks), preferab=
ly serve it from RAM, or at least SSD
- serve it through xrootd, or memcached
- Reddis might be an option, it has some extra smarts eg for compressing =
ranges
- for memcached
- put o(1000) elements per bucket to fill the pages we transfer through n=
etwork
- we could further rely on replication to speedup recovery
- for loader use case, preload cache for area that loader works on
- conclusions: try xrootd and memcached at in2p3 cluster and see what thr=
oughput we could get
- related epic: =
DM-21=
19 - Jira project doesn't exist or you don't have permission to view it=
.
------=_Part_18891_463744036.1711618102876--