2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020



Topics for discussion:

  • Project updates (Eric Bellm , Ian Sullivan ):
  • USDF transition status
    • /repo/main is now up and writeable
    • bps story not clear–status slides say first HSC reprocessing was done lsat week
  • PST talk outbrief (Eric Bellm )
    • as presented slides here
    • prompt processing is highest near-term priority
      • Krzysztof Findeisen : we need more effort, spawning tickets faster than can complete them.  Eric: let's talk about what kind of expertise is needed so we can drive that forward.
    • execution time: got buy-in for an initial push to 120 seconds, then further optimization once we're confident incremental template gen and AP are operating routinely
  • analysis_tools sprint (all)
    • What parquet tables should we generate, and when in the AP process should we generate them?
    • We currently produce *Diff_diaSrcTable and an optional *Diff_assocDiaSrc if DiaPipelineConfig.doWriteAssociatedSources==True, but it defaults to False; can we get most of what we need from the diaSrcTable?
      • probably not
      • needs to happen outside the 60 second loop
      • these are ephemeral data products–will get deleted in a few week, is this a problem?
    • need to make visit Summary table
    • Do we want aggregated visit-level catalogs for regular AP processing to compute visit-level metrics on? These would have to be done in post-processing on the "aggregation butler", or whatever we're calling the place we're uploading the per-detector PP runs to.
      • Eric: QA WG recommended storing metrics at lowest level granularity, could then write tasks to aggregtate metrics at higher levels
      • JP: does analysis_tools support working on metrics itself, yet?  we think not
    • What level do we aggregate DiaObject statistics at? Visit? Patch? Tract?
    • tickets:
      • write out additional parquet tables for DIAObjects, DIAForcedSources. C DM-36199 - Getting issue details... STATUS
      • parquet Source tables (after AP pipe?), visitSummary table.  JP: there are existing tasks, just have to add to appropriate pipelines.  does it go in e.g. ap_verify? DM-36198 - Getting issue details... STATUS
        • KF: not a big deal, we have ap_pipe, ap_verify, each with and without fakes.  fakes pipelines add a bunch of extra tasks to do extra
      • identified a few quantities to add to visitSummary table, e.g. # of good pixels DM-36200 - Getting issue details... STATUS
  • Review CI (https://chronograf-demo.lsst.codes/):
    • no changes
  • Review outstanding action items
  • QA meeting in October:
    • PST diffim talk: risks outline table
    • Eric: fakes completeness/detection SNR discussion
    • analysis_tools sprint results
  • AOB
    • no meeting next week (quiet week)

Action Items

DescriptionDue dateAssigneeTask appears on
  • Eric Bellm write up retention policy and sizing for APDB  
17 Jul 2023Eric BellmAP Pipeline Meeting, 2023-06-26
  • Eric Bellm dig into alert serialization test in ap_association and ticket ways to improve it  
11 Sep 2023Eric BellmAP Pipeline Meeting, 2023-08-21
  • Eric Bellm review broker communication channels  
25 Sep 2023Eric BellmAP Pipeline Meeting, 2023-08-14