Time, Date & Place

12:00 Pacific  on Amazon Chime https://chime.aws/1930107527

Attendees

KTL, Sanjay Padhi, Lorena Costanza, Aaron Bucher, Greg Thain, Todd Miller, Greg Daues, Chris Mills, Dino Bektesevic, Michelle Gower, Wil O'Mullane, Michelle Butler

Discussion Items

  • Updates:
    • Dino has RDS registry working
      • makeButlerRepo needs to be tested?
        • Separate bug already fixed
      • Working on building an AMI and then getting example running
      • Potentially use S3 semantics to make things more efficient
        • Storing multiple datasets in a single object can be retrieved using range get
        • Look into cloud-based geospatial examples
    • Annex connectivity problem progress?
      • Todd Miller needs to look at VM with logs, probably in the next couple of weeks
  • Readiness to do large HSC-RC2 collection of data?
    • Data transfer
    • Registry load/ingest
      • Dino can dump SQLite or Oracle registry and import into RDS/Postgres
      • Better to use existing scripts (from Nate and Jim) to load directly

        • Michelle Gower to find and give to Dino

          Follow up: Hsin-Fang Chiang Please help flesh out details
          Needs to be run on a machine with access to an existing RC2 Gen2 repo.
          First run makeButlerRepo.py to create an empty repo using specific database backend
          Then clone https://github.com/lsst-dm/gen3-hsc-rc2.git and run bootstrap.py <repo from previous step>
          Then run any manual cleanup steps:   (Not sure if still needed see DM-18829)
              UPDATE calibration_label SET valid_last = valid_last + 1 WHERE calibration_label != 'unbounded';

      • Don't have tools to verify result yet; possible that different semantics on S3 vs. Posix may cause problems
      • S3 Events could be used to automatically trigger ingestion
    • Quantum graph generation
      • Batch Production Service will eventually work to execute multiple tasks within one Python execution/HTCondor job 
    • Anything else?
  • Cleanup
  • File transfer plugin for HTCondor
    • As long as workers are at AWS
    • May need to keep original hierarchy rather than flattening; HTCondor team will work on that
  • Aim to have things running by early October in advance of Boston meeting in November
  • No labels