AP Pipeline Meeting, 2022-04-25

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Krzysztof Findeisen 
• Ian Sullivan 
• Nima Sedaghat 
• Kenneth Herner 
• Karlo Mrakovcic 
• Meredith Rawls 
• John Parejko 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • RHL visit last week
  • likely to have meeting-free weeks in June (summer AAS?) & September
  • sprint planning next week
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • DM-33734 follow-on:
    • ran a pipeline with no processing (just loaded images and saved them again) and profiled it
    • Jim and K-T did not agree about which parts of the processing needed worrying about
    • code spent time in four places:
      • package imports (variable).  Jim thinks can reduce healpy impact on daf.butler; K-T thinks it will only happen on first run
      • in pre-execution step, gathers conda package info–takes 8 seconds!  Jim & Tim discussed caching, K-T thinks it only happens on once
        • Ian: is this dealt with on startup?  KF: no: on first pipeline run (per distinct)
      • overheads on reading raws specifically
        • we call lsstcam.getCamera()–very slow generation of Python object).  Again, Jim has an idea for improved amplifier handling; K-T again says it's a one-time penalty
          • Eric: could imagine a warm-up task during preload (ditto the conda info above)
        • IERS validation, takes several seconds of time per raw, could happen on every run of the pipeline
          • Ian: is this still the case where astropy is trying to download the table?
          • JP: can configure astropy to use an offline cache: https://docs.astropy.org/en/stable/utils/iers.html#working-offline
      • Ian:
        • we should figure out what can go into preload, and generate tickets for middleware to implement
          • potential "preload" sprint in the summer?
        • explore in-memory datastore for the butler?
  • "ap_verify in Prompt Processing": there is a successor to Squash planned for operations ("Sasquatch", still influxDB/Chronograf).  Let's discuss what it would take to push metrics from PP.
    • (deferred for two weeks)
      • Eric Bellm discuss "ap_verify in Prompt Processing" 09 May 2022 
        
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Meredith Rawls : starting reprocessing with decorrelation bug fix, aiming for next week results
  • Kenneth Herner : Processed tract 4431 with weekly 16 and 17, two patches (42 and 50) and two bands (g and r)
    • Just running step 4 and step 5
    • Variance plane currently looks exactly the same. Ian Sullivan : This is unexpected, since the variance plane should have changed
    • g-band shows ~4% reduction in sources, r-band ~90% increase in sources. We expected a slight to moderate reduction in sources in all cases, so this is unexpected.
    • Plot of the source distribution shows many sources outside of the patch boundary. 
• Image differencing algorithms, DCR  ( Ian Sullivan ):
• real-bogus (Nima Sedaghat, Eric Bellm, John Parejko ):
  • Working on training networks on simplified task of supernova detection
  • Low signal to noise SN are labeled as SN, but model cannot learn them
  • Eric: this is using too much the "god's eye view" from the truth labels.  Random noise will generate increasing numbers of objects at lower thresholds.
• alert distribution (Eric Bellm ):
• Cosmic rays (Karlo Mrakovcic ):
  • No major differences in CR algorithms that were evaluated
  • Curated a "true" sample of CRs that are labeled as CRs and not CRs
    • Every algorithm is above 99.1% completeness
    • Out of 18,000 stamps, ~600 are not labeled identically between all of the algorithms
      • LSST stack on science images appears to be algorithmically fine. Should still test true completeness by using LSST darks with CRs
      • Current stack appears to miss 0.3% of CRs that appear to be true CRs
• Review CI (https://chronograf-demo.lsst.codes/)
• Review outstanding action items
• QA meeting in May:
  
  • Diffim changes (Ian, DM-34364)
  • New reprocessing run of HiTS using new decorrelation (Meredith, DM-34475)
  • Possibly discuss Karlo's report on CRs (he will be on a plane)
  • (Ian: DCR:  defer to May+)
• AOB
  • none

Action Items