AP Pipeline Meeting, 2022-09-12

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Krzysztof Findeisen 
• Erin Howard 
• Brianna Smart 
• Meredith Rawls 
• Nima Sedaghat 
• Kenneth Herner 
• John Parejko 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • ComCam is on the TMA
  • DM meeting-free week next week–no regularly-scheduled meeting
• USDF transition status
  • /repo/main is now up and writeable
  • bps story not clear–status slides say first HSC reprocessing was done lsat week
• PST talk outbrief (Eric Bellm )
  • as presented slides here
  • prompt processing is highest near-term priority
    • Krzysztof Findeisen : we need more effort, spawning tickets faster than can complete them.  Eric: let's talk about what kind of expertise is needed so we can drive that forward.
  • execution time: got buy-in for an initial push to 120 seconds, then further optimization once we're confident incremental template gen and AP are operating routinely
• analysis_tools sprint (all)
  
  • What parquet tables should we generate, and when in the AP process should we generate them?
    • Eric: anything we put in the APDB needs to get dumped to parquet as well (e.g. https://github.com/lsst/ap_association/blob/f311ae10df1a800422d8892d9c2da6d6dc07ef88/python/lsst/ap/association/diaPipe.py#L440 and following)
    • discussion of when/where to run this. 
  • We currently produce *Diff_diaSrcTable and an optional *Diff_assocDiaSrc if DiaPipelineConfig.doWriteAssociatedSources==True, but it defaults to False; can we get most of what we need from the diaSrcTable?
    • probably not
    • needs to happen outside the 60 second loop
    • these are ephemeral data products–will get deleted in a few week, is this a problem?
  • need to make visit Summary table
  • Do we want aggregated visit-level catalogs for regular AP processing to compute visit-level metrics on? These would have to be done in post-processing on the "aggregation butler", or whatever we're calling the place we're uploading the per-detector PP runs to.
    • Eric: QA WG recommended storing metrics at lowest level granularity, could then write tasks to aggregtate metrics at higher levels
    • JP: does analysis_tools support working on metrics itself, yet?  we think not
  • What level do we aggregate DiaObject statistics at? Visit? Patch? Tract?
  • tickets:
    • write out additional parquet tables for DIAObjects, DIAForcedSources. C DM-36199 - Getting issue details... STATUS
    • parquet Source tables (after AP pipe?), visitSummary table.  JP: there are existing tasks, just have to add to appropriate pipelines.  does it go in e.g. ap_verify? DM-36198 - Getting issue details... STATUS
      
      • KF: not a big deal, we have ap_pipe, ap_verify, each with and without fakes.  fakes pipelines add a bunch of extra tasks to do extra
    • identified a few quantities to add to visitSummary table, e.g. # of good pixels DM-36200 - Getting issue details... STATUS
• Review CI (https://chronograf-demo.lsst.codes/):
  • no changes
• Review outstanding action items
• QA meeting in October:
  • PST diffim talk: risks outline table
  • Eric: fakes completeness/detection SNR discussion
  • analysis_tools sprint results
• AOB
  • no meeting next week (quiet week)

Action Items