Time, Date & Place
12:00 Pacific on Amazon Chime https://chime.aws/1930107527
Attendees
Dino, Todd, Greg Thain, MichelleG, Lorena, Greg Daues, Hsin-Fang, Miron, Wil, Sanjay, Chris Mills
Discussion Items
- Updates:
- Dino and Hsin-Fang got the ci_hsc pegasus workflow running with RDS registry and S3 datastore using stack w_2019_37.
- The repo was made using the "ci_hsc_gen3" package and transferred into RDS & S3
- The test was with mini-condor, all on one instance. Every job talks to RDS & S3 directly via Butler.
- Readiness to do large HSC-RC2 collection of data
- Planned approach:
- Using current tools from the LSST-Gen3 team, make a repo on NCSA's GPFS with a sqlite registry
- This means translating/bootstrapping from a Gen2 repo to a Gen3 repo.
- Load sqlite data into RDS
- Currently a db.t2.micro instance may be okay for initial testing. Eventually we want a larger instance (~$200/month).
- Upload science input files from GPFS to S3
- ~1.6TB, ~$38/month
- With this RDS+S3 repo, generate quantum graph and so on.
- Target LSST-stack: w_2019_38 from this weekend
- Using current tools from the LSST-Gen3 team, make a repo on NCSA's GPFS with a sqlite registry
- The cost is fine, focus on an end-to-end product first, not optimizing yet; arch review to optimize later
- Planned approach:
- Re: annex connectivity error, Todd has been looking into it. Not fully understood yet.
- Need more details logs. Should turn on finer logging.
- If see it again, keep the instance running and annex to allow further debug
- Dino building new/clean AMIs to improve reuseability and shareability
- Also add documentations of the setup process
- Boston workshop preparation
- Sanjay's idea is to have one big machine continuing running with condor, users access to this big machine on which LSST software/data are available
- Hsin-Fang will circulate a google doc to start planning.
- who will be there to help the session: 1-2 from AWS, 1-2 from Condor, 1-2 from LSST-DM
- Will need attendees' info beforehand. Think about what examples to show, etc.