AP Pipeline Meeting, 2022-01-24

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Krzysztof Findeisen 
• Nima Sedaghat 
• John Parejko 
• Kenneth Herner 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • Welcome back from an extended meeting break!
  • Congrats on the (imminent) removal of Gen2
  • The upcoming year is going to be an exciting one
    • on-sky imaging from ComCam is expected ~ one year from now
    • AuxTel one-degree imaging survey
    • LSSTCam ISR is going to be ...interesting
    • migration from NCSA to SLAC is happening, ~ summer-early fall
    • integration of AP within Prompt Processing
    • ramp-up of activity of the AP Commissioning team (Andy → Mario, new hires...) as well as Commissioning activities with a broader team
  • We should expect that time pressure will create additional stressors. 
    • Ian and I will continue to manage workloads and set appropriate external expectations.  Not asking anyone for "hero mode" nor should it be necessary.
    • We will need to set work priorities more carefully and adjust them more frequently to unblock progress integrating with other teams.  Your flexibility will be valuable
    • Commissioning work will be driven forward, and will route around organizational blockages.  It will be important for all of us to have AP deliver promptly
    • Our work will be increasingly visible to project leadership and the community.  This provides an opportunity to build & enhance your professional reputation as someone capable who gets things done.
    • Decadal-scale observatories don't get commissioned that often!  Let's have fun.
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • almost done with review removing gen2 from ap_verify.  wants to hold off on merge until the nightly builds stabilize again
  • Eric: what is next?  Ian:  Jim is going to use a Wednesday SP meeting to outline an approach
  • KF: Tim J insists gen2 removal isn't individual devs' responsibility; however, KF created tickets for ap_pipe, etc. gen2 removal
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Ken: resubmitted the Saha Bulge with w03 with the right DECam ISR settings.  using multi-tract diffim, close to 100% success rate.
  • Ken: pair-coding preconvolution with Yusra, but haven't yet done the detailed comparison to what was done before
  • Ian: there's an old ticket to write READMEs for /datasets.  general: /datasets is gen2 only and going away!  WONTFIX,  but we need something like this... topic for future pipelines meeting?
• Image differencing algorithms, DCR  ( Ian Sullivan ):
  • with John P, sketched out a gen3 refactor of ip_diffim.  Wrote two of the sections (warping, diffim, source detection and measurement).
  • Now gives pixel-identical results in the "standard" configuration
  • going to have to live alongside ip_diffim for awhile due to API changes
  • Eric: does DRP need to have a look?  Ian: did have a pair coding session with Yusra, but was sick and so need to revisit
    • maybe also a potential Wednesday Science Pipelines meeting topic
  • John P: not aiming for feature parity, because we're trying to drop features we don't want to support
  • Eric: what about on-the-fly reconfiguration in response to changes to diffim?  Ian: definitely planned:
• real-bogus (Nima Sedaghat, Eric Bellm, John Parejko ):
  • made a new data-set of calexps and templates, transferred to GCP
  • chatted with Eric and Ian about how to organize the training problem/labels
  • looking for a yaml pipeline that will difference his input.  Meredith: https://github.com/lsst/ap_pipe/tree/main/pipelines/LsstCamImSim
• alert distribution (Spencer Nelson, Eric Bellm ):
  • Alert distribution prototype (Kafka, Schema registry, alert database), alert deployed in GCP by Spencer and community brokers are testing connection
  • https://dmtn-210.lsst.io/
• profiling (John Parejko ):
  • definitely some things we want to turn off–but he wants to wait for jim to extract configs from middleware
  • wonders how to add metrics to Chronograph
    • we're not sure that adding more metrics might not be the way
• Preops DP0.2 preparation (all):
  • Meredith: finishing up the DRP association tolerance; initial look suggested 0.5" looks okay, but trying to look a bit deeper–hitting memory errors with pandas.  Could try a "huge" container on data-int
    • are these parquet data products?  we're not actually sure about this specific data product (goodSeeingDiff_diaSrcTable)–is it FITS, or parquet?
    • KF & Eric agree that out-of-core solutions are needed
• Review CI (https://chronograf-demo.lsst.codes/):
  • no change
• Review outstanding action items
  • none!
• QA meeting in February:
  • Ken: Saha Bulge, preconvolution?
  • Meredith: maybe fake completeness?? 
• AOB
  • fill out sprint planning Doodle!
  • can we get DM and PREOPS tickets to show up in the same JIRA spots–Ian will investigate

Action Items