Time, Date & Place

10AM Pacific  on ls.st/hhb 

Attendees

Goals

  • Kickoff meeting for Google PoC – introductions, plan discussion and initial work assignment

Meeting Minutes & Discussion Items

  • Introductions for new faces from DAX team, NCSA and Google

  • Data transfer to the cloud
    • Rather than transferring to gpfs/DTN access, use GCS
    • WISE file size is ~1GB and KPM ~5GB and won't be a problem
    • Use all 32 nodes in parallel for the transfer preferably
    • What about non-GCS? Could use simple scp/rsync - looks like we are going ahead with GCS unless performance issues are observed
    • Require documentation to get started - should be enough for now (Unknown User (mbutler))
    • Vaikunth Thukral to test a transfer on a test setup

  • Data loading
    • Pre-sharded data means worker-chunk mapping might cause issues with metadata and replicas
    • Might be useful to try original 30 nodes rather than 40 nodes
    • Recommended cloud storage, then move to locally-attached nodes
    • Cloud storage has built in "replication"
    • Qserv-level replicas have their own benefits, will likely need more discussion
    • Reserved loading/replication discussion for next time or after transfer is complete, will most likely require some testing (Igor Gaponenko)

  • Qserv kubernetes deployment
    • Have documentation to give to Google from Fabrice Jammes
    • Should work with locally attached storage setup that is planned, might run into issues with other setups like network-attached nodes later
    • Fabrice Jammes from IN2P3 and Unknown User (mtlong2) from NCSA should be in communication with Robinson regarding our kubernetes effort and development
    • Node configuration might need some more discussion

  • Training
    • Vaikunth Thukral needs links for cloud account training, Jim is sending an email with intro
    • For technically specific areas basic training can be provided via Coursera, more can be provided based on proficiency levels

  • Other/Logistics
    • Jim Ballentine needs an account in JIRA
    • Google personnel need to be invited to #dm-google-poc channel in Slack as single-channel guests
    • Account billing from LSST is still needed

Action items/TO-DOs