AP Pipeline Meeting, 2023-10-16

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm
• Bruno Sanchez 
• Krzysztof Findeisen 
• Brianna Smart 
• Meredith Rawls 
• Kenneth Herner 
• Nima Sedaghat 
• Kurlander Jake 
• Ari Heinze 

regrets

• Ian Sullivan (vacation)
• John Parejko (vacation)
• Erin Howard (vacation)

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • Eric & Ian will be dealing with DM-SST and DMLT vF2F next week
    • DM Leadership Team Virtual Face-to-Face Meeting - 2023-Oct-24
• Auxtel processing status (all):
  • no successful PP last couple of weeks, for several reasons:
    • specifics of refcat storage
    • changes in ObsCore
    • LATISS UPS failure
  • Hsin-Fang Chiang suggests we beef up test repo, but some of this is assumptions in the middleware components that no longer agreed
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • nothing apart from above
• Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
  • Erin out, Ken swamped & nothing to report
• Satellites (Meredith Rawls ):
  • still stuck on unpacking bitpacked DIASource flags
    
    • will bring back to pair coding Wednesday
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • Bruno: created some tests for the mask planes, but Jenkins is failing.  chatted with WMWV and Shu.
  • needs to make some tickets to investigate why low thresholds break the efficiency calculation.
• alert distribution (Brianna Smart , Eric Bellm ): 
  
  • Bri: RFC-964 - Getting issue details... STATUS on adding package has stalled out, needs a comment to get things moving again
• ML (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • Nima: have been regenerating results with the updated dataset.  Also spending some time looking again at the timing, aiming to emulate planned hardware.   Wondering about paralleling over cutouts, which increases memory demands.
  • Would help to have some information about CPU and memory limits of Prompt Processing
    • KF: very virtualized system, hard to reason about specific hardware.  thinks there are a few GB of memory per container.   RFC-960 - Getting issue details... STATUS may provide a 32GB upper limit as policy
• Solar-system processing (Ari Heinze, Kurlander Jake  ):
  • Ari: not much to report, thinking about remaking DP0.3 with sorcha + interstellar objects.   working on improving heliovane
  • Jake: just getting set up at USDF
• Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • no new changes
• Review outstanding action items
• QA meeting November 6:
  • metrics sprint report
  • streak masking (though Meredith may not be available to present due to ADASS)
• AOB

Action Items