AP Pipeline Meeting, 2021-11-15

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan 
• Nima Sedaghat 
• Krzysztof Findeisen 
• Meredith Rawls
• Kenneth Herner 
• Spencer Nelson 

regrets

• Unknown User (cmorrison)  (vacation)

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • stack seems now to be working on newest Mac OS
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • still working to get ap_verify datasets working fully in gen3
  • "do we still want r and i support in ap_verify_hits2015?"
    • full-sized HITS dataset has raws in g, r, and i and associated calibration data–but we are not using r and I
    • only a small handful of visits, not enough for templates etc.  Meredith thinks another dataset would be better for multi-band w/ decam (Eric: DES SN field would be the obvious choice). 
    • so we think it would be fine just to drop to g-band only. DM-32598 - Getting issue details... STATUS
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Meredith: busy with V&V, trying to make lightcurves from tract/patch data. 
  • did verify that Nate's ticket allowed her to fix overlapping tracts, sequentially
    • causes errors on later tracts due to existing DIAObjects (see DM-32246 - Getting issue details... STATUS )
      
      • Meredith Rawls check with Chris on his return about "Duplicate DiaObjects created" error with multiple tracts run 22 Nov 2021 
        
    • Ken's Saha Bulge run tests this without fakes, so we can figure out if the fakes are the issue.
    • Chris's multi-tract fakes ticket is still in progress
  • Ken: showing the convolve science and preconvolution DM-31884 - Getting issue details... STATUS
    • convolve science shows only 20% increase in DIASources relative to best seeing template (factor of 2 for good sources only)
    • preconvolution is 5-10% more in both cases
    • Ian: would be interesting to look at subset of relatively bright sources and use a flux threshold (to avoid sensitivity to exact 5 sigma cutoff)
    • Eric: preconvolution looks better than I've ever seen it before.
    • Eric: will also be worth eyeballing a reasonable number of stamps to see if the artifact distributions differ (e.g., lots of ringing in one case)
      • Ken: does show a fair number in the convolve science case (not a rigorous test, just a random check)
  • Saha Bulge–hitting multi-tract challenges.  needs to refactor analysis code some since can't load all sources into memory, but should have something for Dec.
• Image differencing algorithms, DCR  ( Ian Sullivan ):
  • building DCR coadds in gen3 is working again
• association/SDM/fakes/astrometry (Unknown User (cmorrison)):
  • out today
• real-bogus (Nima Sedaghat Eric Bellm, John Parejko ):
  • Nima: still mainly working on onboarding tasks.  brainstorming with Eric, Ian, Meredith on a private slack channel to get up to speed–others welcome as desired.  otherwise working on problem definition (just a binary classifier?), understanding where it fits in the pipeline.
  • Eric:
• alert distribution (Spencer Nelson):
  • also went to the broker technical workshop
  • working on Kafka, Kubernetes on the IDF–schema registry seems to be up and going, now working on getting the alert stream simulator going–exercises the auth system.
• Preops DP0.2 preparation (all):
  • just Meredith on V&V–notionally signing off on single-frame right now.  biggest issue is that forcedsource files are very large even for one patch
  • also running into mismatches in expectations between visit and tract/patch
• Review CI (https://chronograf-demo.lsst.codes/):
  • No change
• Review outstanding action items
• QA meeting in December:
  • Meredith: HiTS with fakes
    • can do a single tract if we can't make multi-tract work
  • Ken: convolve science ticket
  • Ken: Saha Bulge?
• AOB
  • Ian to run meeting next week

Action Items