AP Pipeline Meeting, 2022-03-21

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Karlo Mrakovcic 
• Nima Sedaghat 
• Krzysztof Findeisen
• Kenneth Herner 
• Meredith Rawls 

regrets

• John Parejko (vacation)

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • lots of things happening on camera refrigeration
  • all hands scheduled
  • sprint planning next week
  • annual reviews in April
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • Prompt Processing sprint (with Eric Bellm )
    • reasonable progress over two weeks, no showstoppers.
    • Still a few more tickets to connect all the pieces together--Krzysztof Findeisen and Eric to work more this week to close out
    • Working on a draft summary
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Meredith: nothing to report
  • Ken: on hold pending diffim refactor
• Image differencing algorithms, DCR  ( Ian Sullivan ):
  • template refactor seems to produce equivalent results to existing pipeline.  Yusra to review
  • rest of the refactor is close, cleared a few bugs.  Focusing on primary code path at the moment, noting bugs/problems in other paths–currently just keeping his own notes, but will ticket when he's done
  • processing generally looks very close, except a slight difference in the background treatment–hopes it's just an afternoon away, hopefully ready for review later this week
  • will then start on RFC to allow folks time to comment on API (probably a two week comment period).  can probably merge no matter what since it doesn't replace existing code
    • will also include deprecation for existing tasks
  • Meredith: are there tickets for updating ApPipe.yaml, etc.?  A: Not yet
• real-bogus (Nima Sedaghat, Eric Bellm, John Parejko ):
  • networks are working now, adapting them to LSST from ZTF etc. 
  • Stuck right now on understanding how saturated values are handled–they are masked, but what values are they set to?  A: They are interpolated.  Threshold is set per camera?  (Per detector?)
  • Eric suggesting a little package to grab appropriate metadata in a camera-agnostic way, when shipping data for training
• alert distribution (Eric Bellm ):
  • need to get the prototype working again, start thinking about a test plan
• Cosmic rays (Karlo Mrakovcic ):
  • walking through some presentation slides
  • comparing different CR detection algorithms
    • stack on science
    • stack on difference
    • astroscrappy (LACosmic)
    • usupervised ML
  • create credible dataset of stamps labeled CRs/non CRs
  • difference CR mask not helpful due to convolution of template mask (and CRs are excluded from template anyway)
  • testing on HSC visit
  • lots of 1 pixel CRs exist
  • CRs are about 0.1% of the pixel count
  • if even 1 pixel is marked CR, called the whole 30x30 stamp CRs
  • ~26% of the CRs are associated with detected DIASources.  Longer CRs less likely to be detected?
  • few false positive CRs (although in some cases the DIASource is not actually hit by the CR).  Some CRs not detected as CRs, though
    • CR + object is messing up the detection footprint since we haven't interpolated over it
• Review CI (https://chronograf-demo.lsst.codes/)
  • some changes in DIASource count and runtime in both cameras around 3/15-3/17 (not all on both days).  Ian and Nima to investigate/annotate.
  • Nima and Ian identified a few possible tickets, but none were a smoking gun:
    • DM-34003 added explicit locations for calculating PSFs. This could change the number of diaSources, but could not affect ISR or calibration
    • DM-33645 "Add FinalizeCharacterizationTask to measure psf/apcorr with consistent stars." Could affect ISR and calibration, but does not appear to be used by AP
• Review outstanding action items
  • none!
• QA meeting in April:
  • Eric: PP sprint summary
  • Karlo: reprise CR progress report
  • (Ian: DCR?  defer to May+)
• AOB

Action Items