AP Pipeline Meeting, 2023-08-28

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Brianna Smart 
• Ari Heinze 
• Erin Howard 
• Nima Sedaghat 
• Krzysztof Findeisen 
• Bruno Sanchez 
• John Parejko 
• Meredith Rawls 
• Harshit Rai 
• Kenneth Herner 

regrets

• Ian Sullivan 

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
• Auxtel processing status (all):
  • run starting tomorrow
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • John: diaSource centroids (see discussion here)
    • surprisingly complicated to see what we're doing for "centroid of a DIASource." 
    • at first it looks like it comes from NaiveDipoleFit (just 3x3 box per positive/negative peak).
    • but maybe we are using the DipolePSF centroid
    • the code flow is hard to follow
    • in any case we certainty haven't calculated uncertainties from any of the dipole fits, which we need
    • Jim B. isn't sure the model is well-enough defined to have uncertainties; Eric and John think
    • Eric: we also have to figure out model selection question.  maybe Eric can lead a Weds. discussion
  • John: talked to Anastasios Tzanidakis about cutouts; realized cache for cutouts code wasn't working, which speeds the code up substantially
• Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
  • Erin:
    • CM: working with Orion to make cm_tools config files for the AP pipelines
      • cm_tools works with panda, provides a user-friendly way to deal with/list errors
    • worked with Ian and John on crowded fields–crowded fields are trying to use 50k objects for astrometry, leading to out of memory; limited to 5k via config resolved the issue
      • next bug in line is DM-36760, will work on in pair coding Weds
  • Ken: closing out dataset comparision review; also ran with quantum-backed butler without error.  filling in missing metrics this week.
• Satellites (Meredith Rawls ):
  • vacation last week, still working on streak finding implementation in single-frame
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • Bruno:
    • talked to Ian last week about flags/masks
    • comparing efficiencies between similar runs–thinking about some tooling, although may need revision for source_injection
      • Eric: what does it run on?  Bruno: matched fake catalogs
      • Eric pushing everyone to start thinking about implementing via analysis_tools metrics
• alert distribution (Brianna Smart , Eric Bellm ): 
  
  • working on updating connection info for USDF vs IDF for eg Kowl, then out for a few days
  • Roy Williams asking after testing
• TransiNet (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • Nima: working on DC2 migration to IDF.  Not fully restored on the USDF side, so working on both
• Solar-system processing (Ari Heinze ):
  • Ari: working on interstellar-capable version of the orbit fitting, struggling a bit with the deriviation–trying an alternative approach now
  • talked with Eric about how to specify covariances of trailed sources—Eric suggests Ari propose a desired combination of positions and uncertainties
• Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • no changes
• Review outstanding action items
• QA meeting September 11:
  • CalibrateImagesTask in AP (John)
  • APDB run comparison tool demo (Ken)
  • streak masking/long trail rejection?  If visuals ready (Bri & Meredith)–probably defer to October
• AOB
  • next Monday a holiday, no meeting
  • John: give opinions on task defaults on DM-39796 - Getting issue details... STATUS
    • will discuss at pipelines meeting Wednesday
    • Meredith: how much are we trying to match DRP configs?  Eric: we can diverge when we need to for speed reasons but probably should try to match otherwise.

Action Items