AP Pipeline Meeting, 2021-06-21

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan
• Krzysztof Findeisen
• Unknown User (cmorrison)
• Kenneth Herner
• Spencer Nelson
• Meredith Rawls
• Unknown User (gkovacs)
• John Parejko

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • DMLT vF2F this week
  • last week's rebaseline review seems to have gone reasonably well
  • construction teams not being allowed into Chile at the moment, so schedule is slipping day for day
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • no news
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Meredith plans to kick off BPS run for DC2 this afternoon
  • Ken: 5-year depth run on tract 443 got held on the weekend, and now wall-time is preventing it from being restarted
    • strange halts along the way–Ken looking into the logs to try to diagnose
    • probably will kill and restart
    • Eric: is our ability to search the logs good enough?  Ken: current methods are sufficient if you know what you're looking for, but hard if you aren't.  Meredith: one log per quanta better than Gen 2 mess, but hard to figure out

• Difference imaging algorithms  (Unknown User (gkovacs), Ian Sullivan ):
  • Gabor: not much, doing some large reviews.  will pose a question on Slack about PSFs
  • Ian: has been digging into failures with convolving science image
    • stack changes since the diffim sprint complicating this a bit.
      • good news is that correct PSF type is now being attached
      • but WarpedPSF types aren't persisted–so they just get silently stripped.
        • Gabor: it's just a WCS tweak on top of a normal PSF
      • Ian to ticket persisting it–requires going through the pybind11 layer
        • Krzysztof has some experience with the persistable framework, will look at the ticket
• association/SDM/fakes/astrometry (Unknown User (cmorrison)):
  • 12% of image difference failures in DP0 are due to non-overlapping tracts, which is breaking drp_association–being worked on upstream. 
  • Chris suggested they run drp_association alone, but not clear if that has been done.
  • regarding gen2/gen3 differences: Chris says gen3 now reads native types as written, whereas gen2 was coercing to float.  Next step would either be to confirm that this creates the differences we see between gen2 and gen3, or to identify other ISR etc. differences.
  • Now returning to affine astrometry–ran over all of validate_drp over the weekend.
    • close to existing fitter, although some marginal fits fail that succeeded in old.  Maybe about ~10% worse
    • worse in y-band–distortion model worse? 
    • not clear the distortion model is at the level needed to flip the switch
    • need some inputs on timeseries features for DP0.2
• real-bogus (Eric Bellm, John Parejko ):
  • another meeting with the EPO and Zooniverse folks
• alert distribution (Spencer Nelson):
  
  • has now set up a broker and schema registry in the IDF
• Preops DP0.2 preparation (all):
  • QA analysis sprint ongoing
    • Eric working on matching truth table to drp DIASources
    • Meredith trying to run ap_pipe, Ian to make follow-on notebook
      • Ian maybe will run off of the drp mode DIASources
• Review CI (https://chronograf-demo.lsst.codes/):
  • no changes
    • Eric Bellm ticket to enable fakes insertion in HiTS 28 Jun 2021 
• Review outstanding action items
• AOB
  • sprint planning timing
    • Chris, Spencer,
    • Meredith maybe reprioritize?

Action Items