Time, Date & Place
10AM Pacific on ls.st/hhb
Attendees
- Unknown User (mbutler)
- Ross Thomson
- Jim Ballentine (G)
- Robinson Maureira (G)
- Mariano Colin (G)
- Igor Gaponenko
- John Gates
- Fabrice Jammes
- Benjamin Rozier
- Jacob Rundall
- Leanne Guy
Goals
- Kickoff meeting for Google PoC – introductions, plan discussion and initial work assignment
Meeting Minutes & Discussion Items
- Introductions for new faces from DAX team, NCSA and Google
- Data transfer to the cloud
- Rather than transferring to gpfs/DTN access, use GCS
- WISE file size is ~1GB and KPM ~5GB and won't be a problem
- Use all 32 nodes in parallel for the transfer preferably
- What about non-GCS? Could use simple scp/rsync - looks like we are going ahead with GCS unless performance issues are observed
- Require documentation to get started - should be enough for now (Unknown User (mbutler))
- Vaikunth Thukral to test a transfer on a test setup
- Data loading
- Pre-sharded data means worker-chunk mapping might cause issues with metadata and replicas
- Might be useful to try original 30 nodes rather than 40 nodes
- Recommended cloud storage, then move to locally-attached nodes
- Cloud storage has built in "replication"
- Qserv-level replicas have their own benefits, will likely need more discussion
- Reserved loading/replication discussion for next time or after transfer is complete, will most likely require some testing (Igor Gaponenko)
- Qserv kubernetes deployment
- Have documentation to give to Google from Fabrice Jammes
- Should work with locally attached storage setup that is planned, might run into issues with other setups like network-attached nodes later
- Fabrice Jammes from IN2P3 and Unknown User (mtlong2) from NCSA should be in communication with Robinson regarding our kubernetes effort and development
- Node configuration might need some more discussion
- Training
- Vaikunth Thukral needs links for cloud account training, Jim is sending an email with intro
- For technically specific areas basic training can be provided via Coursera, more can be provided based on proficiency levels
- Other/Logistics
- Jim Ballentine needs an account in JIRA
- Google personnel need to be invited to #dm-google-poc channel in Slack as single-channel guests
- Account billing from LSST is still needed
Action items/TO-DOs
- Vaikunth Thukral Make JIRA stories/plan for data-loading (NCSA) and Qserv k8s (DAX + Google)
- Vaikunth Thukral Take relevant trainings for account/billing management
- Vaikunth Thukral Request Slack access for Google personnel