Time, Date & Place
12:00 Pacific 2019-07-19 on Amazon Chime https://chime.aws/1930107527
Attendees
Dino
Chris
Lorena
Aaron
GregT
Todd
Sanjay
Discussion Items
- AWS credits are renewed to the end of this year - thank you Lorena!
- S3 Butler Datastore
- Dino working on this PR and merging the changes to the LSST stack. Questions on ButlerURI design in the code. If Tim is too busy reviewing codes KT can help.
- Also working on using Postgres registry
- Tried running on AWS single instance. Hsin-Fang's old instances didn't work due to pytest issue so Dino had to re-install. Dino saw some errors testing with Gen2 ci_hsc
- Dino has S3 + RDS branches working. More work in implementations
- db_auth ?
- discussions about authorization & credentials
- Can work directly from Dino's laptop talking to S3 & Postgres. Use boto3 client & stored private files for authorization.
- S3 has to live in ~/.aws and RDS credentials file db-auth.paf has to live in ~/.lsst
- Chris & Aaron recommends considering AWS auth layer "instance_profile" outside of the instance . Can rotate credentials behind the scene.
- independent to annex credentials. condor annex has annex users. IAM roles
- the LSST approach in general
- A lot more cases to think about...
- S3 condor transfer plug in
- a wrapper around AWI CLI
- Greg will show Hsin-Fang examples
- scaling larger? maturity of HSC-RC2 dataset? – Not yet, but almost ready
- next milestones
- LSST-PCW Aug 12 a short demo within the DM team
- Boston workshop in Nov: aim for a prototype Gen3 tutorial . Demonstrate it's easy for users to submit, generate data, can do analysis, etc on their logins
- https://petabytestoscience.github.io/workshop-iii