Time, Date & Place

12:00 Pacific 2019-07-19 on Amazon Chime https://chime.aws/1930107527

Attendees

Wil O'Mullane

Hsin-Fang Chiang

Kian-Tat Lim

Unknown User (mbutler)

Dino

Chris

Lorena

Aaron

GregT

Todd

Sanjay 



Discussion Items

  • AWS credits are renewed to the end of this year - thank you Lorena! 
  • S3 Butler Datastore
    • Dino working on this PR and merging the changes to the LSST stack. Questions on ButlerURI design in the code. If Tim is too busy reviewing codes KT can help. 
    • Also working on using Postgres registry 
    • Tried running on AWS  single instance.  Hsin-Fang's old instances didn't work due to pytest issue so Dino had to re-install. Dino saw some errors testing with Gen2 ci_hsc 
    • Dino has S3 + RDS branches working. More work in implementations 
    • db_auth ? 
  • discussions about  authorization & credentials 
    • Can work directly from Dino's laptop talking to S3 & Postgres. Use boto3 client & stored private files for authorization. 
    • S3 has to live in ~/.aws and RDS credentials file db-auth.paf has to live in ~/.lsst
    • Chris & Aaron recommends considering AWS auth layer "instance_profile" outside of the instance . Can rotate credentials behind the scene.   
    • independent to annex credentials. condor annex has annex users. IAM roles
    • the LSST approach in general
    • A lot more cases to think about... 
  • S3 condor transfer plug in 
    • a wrapper around AWI CLI 
    • Greg will show Hsin-Fang examples 
  • scaling larger? maturity of HSC-RC2 dataset? – Not yet, but almost ready  
  • next milestones 
    • LSST-PCW Aug 12 a short demo within the DM team 
    • Boston workshop in Nov: aim for a prototype Gen3 tutorial   . Demonstrate it's easy for users to submit, generate data, can do analysis, etc on their logins 
    • https://petabytestoscience.github.io/workshop-iii
  • No labels