AP Pipeline Meeting, 2023-11-20

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm
• Ian Sullivan 
• Erin Howard 
• Hsin-Fang Chiang 
• Brianna Smart 
• Ari Heinze 
• Meredith Rawls 
• Kenneth Herner 
• John Parejko 
• Nima Sedaghat 
• Harshit Rai 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • All-Hands meeting near SLAC first week of February

  • Operations Rehearsal for Commissioning #3 first week in April

  • watch for sprint planning Doodle for next week
• Auxtel processing status (all):
  • AuxTel now observing 4 days/week best effort
  • no difference imaging this week–new set of fields
  • continuing to chase bugs from the observing scripts, but got to a better spot by end of the week.  Generating single frame products
• Pipelines Infrastructure (John Parejko, Krzysztof Findeisen ):
  • John: asked Merlin to review snapCombine, but need to get someone else–Ken, Ian are willing
• Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
  • no reports
• Satellites (Meredith Rawls ):
  • nothing
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • Ian: still chasing two bugs:
    • duplicate DIAObjects: still top priority, looking for pair coding partners
      • finding DIASources well off of the chip, which shouldn't happen.
      • during association on preconvolution with fakes run
      • just trimming the unphysical sources has resulted in more errors, for some reason
    • calibrateImageTask: working with John to improve astrometry (which leads to more dipoles).  Diffim kernel doesn't seem to be soaking up the shift as we'd expect it to
      • this is DC2 data, and the PSFs are weird and ugly
      • John will look into PSFs
      • Ian will look into kernels
      • John: detect has a "multiplier" keyword that lets you adjust what's in the footprint (to 5 sigma) while selecting the peak that is multiplier higher (to find peaks)
        • slack thread indicates that's a bad hack that we shouldn't be using, but it's what's being used to select sources for calibration
        • seems like the results are rather sensitive to these choices
        • John thinking about how to organize discussion with DRP folks
• alert distribution (Brianna Smart , Eric Bellm ): 
  
  • Bri: We can now send alerts to the alert stream from pipelines
  • need to deal with credential storage (Krzysztof has made some suggestions), doesn't fail gracefully
  • also some very large alert packets–we expect this is big cutouts due to big footprints
• ML (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • Nima: working on butler mode of model-packages with Tim, stay tuned
  • Harshit: looking at dipoles
• Solar-system processing (Ari Heinze, Kurlander Jake , Mario Juric ):
  • Ari: trying to find hypothesis saturation with Heliolincc.  Interior to Earth orbit, most of the missed ones are close to the Earth, considering clustering radius floor. 
    • Ari thinking of working with Steven Stezler on pipeline tasks since Steven is doing a lot of pipeline processing on DECam DEEP (on hyak)
  • Jake: there's a working version of the ephermeris generation in a notebook
    • fast except when reference times are distant from present time
    • current SkyBot lives in ap_association, not clear where precompute should live (with other SSP scripts?)
  • Mario:
    • talked some John about where ephemerides should live (in APDB?).  We need to talk to Andy S about this when he returns in Jan.
    • for now will ignore the problem and just use one big file/
• Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • just one of the COSMOS "Fraction updated" lines is wobbly?
  • possible improvement in subtract images speed, maybe due to change of kernel candidate but the merge timing seems a bit early.
  • neither are high priority for followup at this time.
• Review outstanding action items
• QA meeting December 4:
  • streak masking (Meredith)
  • status of preloading & Solar System Association?  (Jake)
  • duplicate DIAObject status? (Ian & Erin)
• AOB

Action Items