Time, Date & Place
12:00 Pacific 2019-08-02 on Amazon Chime https://chime.aws/1930107527
Attendees
Chris, Sanjay, Dino, Lorena, Greg Daues Hsin-Fang Chiang Unknown User (mbutler)
regrets: Wil, K-T, Aaron
Discussion Items
- Schedule of this meeting: Cancel 8/16 (LSST Project Community Workshop at Tucson), Continue on 8/23. Hsin-Fang will send out the new calendar invite.
- Hsin-Fang will give a short demo in the LSST Project Community Workshop DM session on Aug 12. Plan to show ci_hsc workflow + condor annex + spot fleet + S3 datastore + sqlite registry on EFS; live demo a pipeline task.
- Sanjay and Dino: good to mention the scalability & shareability of the data. For example S3 across countries e.g. S3 Chile, S3 France, S3 US)
- Dino has been working on the RDS registry
- Tried both copying from sqlite3 & ingest from scratch. Debugging..
- Can try with POSIX datastore to simplify
- Unknown User (mbutler) NCSA has an object store too and can offer it for tests
- For scaling up, a larger dataset (HSC-RC2) is almost ready. A full tract of HSC-RC2 was processed successfully on NCSA with Gen3 software.
- Defer discussions with Condor team:
- HTCondor S3 plug-in for data transfer. Next step?
- annex on multiple aws regions?