AP Pipeline Meeting, 2021-08-16

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan
• Spencer Nelson
• Krzysztof Findeisen
• Kenneth Herner
• Meredith Rawls

regrets

• Unknown User (cmorrison) (vacation)
• John Parejko  (also vacation)

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • starting to prepare for director/agency reviews in Sept/Oct
• PCW debrief:
  • fill out the feedback form
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • copying various camera-specific configs into ap_pipelines
  • converging with Jim/Meredith about some config questions
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Ken: gave one last shot to tract 442 in DC2–still running.
  • also finished reprocessing the sprint datasets.
    • Ian took a look and updated the image differencing audit for different pipeline configs
      • preconvolution and convolve science both run to completion, and have 1.4-2.5x the number of sources as in the "standard" A&L case (not apples to apples because the templates differ).  Waaay better than what we had before.
      • Eric will look at Ken's notebook and think about followup analysis tickets
    • Meredith: can we start doing this off /repo/main? 
    • these runs were intended for an apples/apples comparison with Feb.  Just need to make corresponding templates in /repo/main
    • We think having shown some improvement wrt Feb (from algorithms only) now would be a good time to move into /repo/main for the future–redo including using template gen
  • Eric showing COSMOS fake injection completeness (DM-31340)
• Image differencing algorithms  (Unknown User (gkovacs), Ian Sullivan ):
  • Ian discussed above, Gabor is out
• association/SDM/fakes/astrometry (Unknown User (cmorrison)):
  • out today
• real-bogus (Eric Bellm, John Parejko ):
  • nothing new
• alert distribution (Spencer Nelson):
  
  • mostly non-Rubin stuff
• Preops DP0.2 preparation (all):
  •  nothing short term
• Review CI (https://chronograf-demo.lsst.codes/):
  • several weird things
  • timing for Image Difference disappeared at the end of July?  Krzysztof will ticket to investigate
  • Fakes completeness all jumped between Aug 3 and 5: this is not related to taking fakes out of the source count metrics, but is connected to Chris's change to the variable source injections
    • discuss Aug 5 changes to fakes metrics with Unknown User (cmorrison)  23 Aug 2021 
  • finally the HiTS fakes completeness metrics started appearing in early August, and they are not good–we need to understand why.
• Review outstanding action items
• QA meeting in September
  • Eric COSMOS fakes/zooniverse
  • Ken & Ian on pipelines configurations
  • Meredith: optimistic (vacation), but a new gen3 (fakes) DECam run?  may also be blocking on gen3 DECam bias levels, also the multi-tract reprocessing (we think that Yusra has now merged it, right before PCW)
• AOB
  • none

Action Items