AP Pipeline QA Meeting, 2022-06-06

4pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• John Parejko
• Kenneth Herner
• Colin Slater
• Meredith Rawls
• Krzysztof Suberlak
• Krzysztof Findeisen
• Jeffrey Carlin
• Bryce Kalmbach
• Nima Sedaghat  
• Leanne Guy 
• Yusra AlSayyad 
• Mario Juric 
• Steven Stetzler

regrets:

• Andrew Connolly  (conflicting meeting)

Topics for discussion:

• Precursor processing (AP team):
  • Performance sprint (Eric Bellm , John Parejko , Meredith Rawls )
    • May 2022 Performance Sprint Summary
    • https://docs.google.com/presentation/d/14hBEUusvTRJtQBBVHQiy7og9WbLB9f5jnD95Vtd6ocA/edit?usp=sharing (attached here)
  • Report on diffim refactor (Ian Sullivan )
    • new diffim is in the stack as of weekly 23.
    • has proper unit tests to guide future development
    • get pixel-identical results to old diffim in some configurations, in particularly better seeing in the template image.  (there are some bugs in the old diffim which we are not  trying to bring into the new version, and so which don't generate identical results)
    • still performing some validation tests.  Some metadata issues when running old datasets.
    • convolving science image may be working?
    • Eric: what are you hoping to do before a community post?  A: resolve the metadata issue to enable ap_pipe runs, run on a larger dataset to check for any major issues.
• SITCOM team updates (Andrew Connolly, Mario Juric   et al.):
  • Bryce:  showing results of optimizing the donut code and adding parallelism: ~4x improvements.
    • at 40 seconds in densest fields, need to get to 14-16 seconds.  most time in computing Zernies and getting cutouts
    • they are saving stamps to the Butler.  John P: forthcoming "in memory butler" will be helpful here.  More generally the project should make the
    • Eric wonders if there are simplifications in ISR as well–they did already turn many things off
    • Mario: are Zernike calculations in python?  A: yes, via Galsim.  M: could think of dropping to C++, or going to ML?
  • Bryce:  also looking at Zernikes in vignetted areas.  Masking vignetted regions in the model, still getting good results for all but ~5% of the intrafocal detector area (< 1.8 degrees from the center of the focal plane)
• PreOps V&V (Colin Slater ): 
  • DP0.2: all data processing is finished!  We have an enormous pile of data sitting on Google, we should figure out how to make use of that!
  • e.g., grab coadds from data-int rather than build fresh ones
  • remaining task is shaking down RSP services
• Review CI (https://chronograf-demo.lsst.codes/):
  • fakes and solar system improvements are now working.
    • one missing SSO to chase down, plus some astrometric offsets
    • fake completeness still seems a little low in HiTS
  • effects of improvements from performance sprints showing up, as expected.

• Next meetings (all at 4pm Pacific):
  • suggest we skip meeting in July: July 4 is a holiday, both Eric and Ian on vacation subsequent week
  • August 1 (tagup before PCW)
• AOB
  • none
    

Action Items