Time, Date & Place

12:00 Pacific  on Amazon Chime https://chime.aws/1930107527

Attendees

Dino, Todd, Greg Thain, MichelleG, Lorena, Greg Daues, Hsin-Fang, Miron, Wil, Sanjay, Chris Mills

Discussion Items

  • Updates:
    • Dino and Hsin-Fang got the ci_hsc pegasus workflow running with RDS registry and S3 datastore using stack w_2019_37. 
    • The repo was made using the "ci_hsc_gen3" package and transferred into RDS & S3
    • The test was with mini-condor, all on one instance. Every job talks to RDS & S3 directly via Butler. 
  • Readiness to do large HSC-RC2 collection of data
    • Planned approach:
      • Using current tools from the LSST-Gen3 team, make a repo on NCSA's GPFS with a sqlite registry
        • This means translating/bootstrapping from a Gen2 repo to a Gen3 repo. 
      • Load sqlite data into RDS
        • Currently a db.t2.micro instance may be okay for initial testing. Eventually we want a larger instance (~$200/month).
      • Upload science input files from GPFS to S3  
        • ~1.6TB, ~$38/month
      • With this RDS+S3 repo, generate quantum graph and so on. 
      • Target LSST-stack: w_2019_38 from this weekend
        • Soon there will be a backward incompatible change in the Gen3 registry ( DM-17023 - Getting issue details... STATUS ). Old repos will become obsolete. 
    • The cost is fine, focus on an end-to-end product first, not optimizing yet; arch review to optimize later
  • Re: annex connectivity error, Todd has been looking into it. Not fully understood yet.
    • Need more details logs. Should turn on finer logging. 
    • If see it again, keep the instance running and annex to allow further debug 
  • Dino building new/clean AMIs to improve reuseability and shareability 
    • Also add documentations of the setup process 
  • Boston workshop preparation 
    • Sanjay's idea is to have one big machine continuing running with condor, users access to this big machine on which LSST software/data are available
    • Hsin-Fang will circulate a google doc to start planning.
    • who will be there to help the session: 1-2 from AWS, 1-2 from Condor, 1-2 from LSST-DM
    • Will need attendees' info beforehand. Think about what examples to show, etc. 




  • No labels