AP Pipeline Meeting, 2022-01-31

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• John Parejko 
• Krzysztof Findeisen 
• Nima Sedaghat 
• Kenneth Herner 
• Meredith Rawls 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • no major updates
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • KF: we are no longer running gen2! 
    • Gen2 display still is on Squash through next week's QA meeting
    • Ken & Meredith added multi-tract diffim, and will now show up in regular ap_verify runs
  • John Parejko : DM-27013 - Getting issue details... STATUS discussion
    • ticket is in review with Lauren (who will assess how using Gaia for astrometry will affect HSC).  DRP preparing for an HSC release freeze
    • similar question for us on DECam.  Meredith: I think this is our default already.
      • this is done in ap_verify, at least–what about ap_pipe?  what is being used in our ~monthly reprocessing?
      • can we check after the fact?  This now should be logged by refcat loading
        • Meredith Rawls check what astrometry refcats our monthly reprocessing are using 14 Feb 2022 
          
      • (followup: we are not using Gaia in our monthly reprocessing and should.  John P thinks maybe we just should go ahead on DECam?  Doesn't seem like Lauren is likely to push this through on HSC in the near term.)
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Ken: looking again at preconvolution w/ Yusra.  Pipeline ran to dia_pipe and then died–association claims every DIASource has NaN ra and dec values.   Now have to rerun with a current weekly
  • Ken: Saha Bulge processing with updated ISR didn't change too much
  • Meredith: no updates.  pair coding on fakes Wednesday
• Image differencing algorithms, DCR  ( Ian Sullivan ):
  • on Friday, sketched out some diffim design with John.  Trying to avoid having configs that change connections, instead make it clear which branch of the code will run.  Going to ask Krzysztof to do a design review later this week
• real-bogus (Nima Sedaghat, Eric Bellm, John Parejko ):
  • continuing to work on technote, converting ground-truth catalogs to image-like data (probably on the fly)
  • awaiting diffim
  • butler script got moved into ap_pipe-notebooks
• alert distribution (Eric Bellm ):
  • from Spencer, learned how to log into the IDF Kubernetes, eyeball the alert distribution spaceship
  • all 7 brokers have connected
• Preops DP0.2 preparation (all):
  • no substantial updates–maybe let's retire this bullet.
• Review CI (https://chronograf-demo.lsst.codes/):
  • big drop in unassociated DIASources in DECam (but not HSC) due to DM-30651 (Switch DECam overscan correction to median per row), which is good
    • this also led to as much as a 30sec improvement in calibrate runtime for some DECam CCDs???  Seems surprisingly large.
  • Switch to multi-tract image differencing (DM-32988) led to big jumps in fakes completeness.  Is this related to DM-31388?
• Review outstanding action items
• QA meeting in February:
  • Ken: Saha Bulge, preconvolution/image differencing configurations update.  will keep us posted
  • Meredith: maybe fake completeness?
• AOB

  • Karlo Mrakovcic arriving later this week–will dig into CRs
    

Action Items