AP Pipeline Meeting, 2021-12-13

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Krzysztof Findeisen 
• Nima Sedaghat 
• Spencer Nelson 
• Unknown User (cmorrison) 
• John Parejko 
• Kenneth Herner 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • power issues on the summit have led to cancellations of AuxTel runs this month and next
  • no-meeting period Dec 14-Jan 14.  Next ap_pipe meeting Jan 24
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • still working on gen2 removal–things going a little slower than expected because it's harder to test individual pieces, and each run takes several hours–doesn't think we will be ready Jan. 1
  • doesn't think there's anything that we can do to speed it.
  • send tickets to Ian to flag for TCAM standup
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Meredith:
    • no updates; hopes to work more on fixing the fakes completeness issue with a ticket that Josh Meyers is working on.
    • V&V work, but hopes to get to analysis_ap, and start persisting plots that could be use the DRP viewer app
  • Ken chunked up Saha Bulge for plotting in Memory
• Image differencing algorithms, DCR  ( Ian Sullivan ):
  • digging deeper into at image differencing sprint configurations–saw some strange numbers in some visits.  Found some big failures in some CCD regions (large numbers of unsubtracted sources), trying to understand why–only has been grepping out of the logs so far, need to get into the notebooks
  • will use Christmas-NY times to think about building ip_diffim in a pure gen3 mode (with John P, others welcome)
• association, astrometry et al. (Unknown User (cmorrison)):
  • will get fakes review from Lee tonight
  • matcher completed, but was hanging during scons, and then the system was down over the weekend–should be fixed tonight
  • will make an attempt to get the minimizer in before Wednesday, but may have to ship it with instructions on how to finish
• real-bogus (Nima Sedaghat, Eric Bellm, John Parejko ):
  • got a very basic pipeline running with help from Ken, now trying to understand how to figure out what products are in the collection
• alert distribution (Spencer Nelson):
  • cluster ran out of disk space over the weekend, which broke everything in kubernetes-land...  In a 200+ reply thread with Strimzi
  • ready to invite brokers to test (after the disk space issue is resolved)
  • also wrapping up his job at UW at end of January
• Preops DP0.2 preparation (all):
  • Meredith to work on association tolerance ticket, also some technote writeup for Colin
• Review CI (https://chronograf-demo.lsst.codes/):
  • jumps in HiTS DIASource counts and characterize runtime (up to 30 seconds on one CCD!) following DM-32430
  • (a little bit of confusion because there is one CI run after the ticket merged that didn't change.  This turns out to have been the weekly run, which didn't include DM-32430).
  • DM-32962 and DM-32963 to fix these regressions
• Review outstanding action items
• QA meeting in January:
  • (no meeting)
• AOB

Action Items