2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020



Topics for discussion:

  • Project updates (Eric Bellm , Ian Sullivan ):
    • watch for email about virtual summit tour June 1 (will be recorded)
    • SLAC onboarding is coming
      • Ken likewise hit some questions
      • Meredith, Krzysztof didn't see emails.
  • Pipeline Performance sprint (Eric Bellm , John Parejko , Meredith Rawls ):
    • halfway through
    • Umbrella ticket: DM-34623 - Getting issue details... STATUS
    • made a AP-specific SFM pipeline; removing cmodel provided 30% decrease in execution time
    • well on our way to a DC2 CI dataset
    • MR: identified a larger 272 visit dataset for longer runs; now a tagged/chained collection.  Will run that and persist timing metrics.
      • spatial selection was a bit tricky due to dithering–now have some code to find detector-visits fully inside the patches.
      • home for that code is not clear at the moment.  perhaps ap_pipe-notebooks
    • JP: figured out how to turn off timing decorator so profiling will be interpretable; also trying statistical profiler (pyinstrument)
    • Eric thinking about how to use e.g. ELASTICC to make fake history for timing our timeseries features.  Could we have a pre-loaded apdb in CI?
      • KF worried about brittleness, concerned we don't know for sure where the real DIASources are.  Eric thinks we could figure out at least some confident reals.  In real ops we will have some DIASources with history, some with none, and some history with no DIASource.
  • Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
    • Krzysztof:
      • fixed a few fakes-related bugs during fakes pipeline rework
      • SSO attribution problem mostly solved, in review.
  • Precursor processing (Meredith RawlsKenneth Herner ):
    • Ken: starting work on diffim refactor review for Ian
    • Meredith: going to use Lee's new DECam skymap for HITS reprocessing.  Should it use new diffim?
  • Image differencing algorithms, DCR  ( Ian Sullivan ):
    • diffim in pretty good shape, but there are still some bugs in integrating with ap_verify due to metadata changes
    • also would be good to have comparable reprocessing runs with both old and new diffim.
      • so kick off existing diffim with just the skymap change
  • transinet (Nima Sedaghat, Eric BellmJohn Parejko ):
    • out today
  • alert distribution (Eric Bellm ):
    • no update.
  • Review CI (https://chronograf-demo.lsst.codes/)
  • Review outstanding action items
  • QA meeting in June:
    • Performance sprint update
    • Report on diffim refactor
  • AOB
    • No meeting next week due to the holiday.
    • discussion of calendar syncing

Action Items

DescriptionDue dateAssigneeTask appears on
  • Eric Bellm write up retention policy and sizing for APDB  
17 Jul 2023Eric BellmAP Pipeline Meeting, 2023-06-26
  • Eric Bellm dig into alert serialization test in ap_association and ticket ways to improve it  
11 Sep 2023Eric BellmAP Pipeline Meeting, 2023-08-21
  • Eric Bellm review broker communication channels  
25 Sep 2023Eric BellmAP Pipeline Meeting, 2023-08-14