DRP Metrics Monitoring 2023-05-08

Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Yusra AlSayyad , Jim Bosch , Robert Lupton , Erfan Nourbakhsh , Lee Kelvin , 

Regrets: Arun Kannawadi

Agenda:

• Meeting recorder - (last 6 meetings were:  Arun, Clare, Lee, Colin, Jim, Keith) 
• Announcements
  • Frossie wants to decomission Squash ASAP
• Review Action items from last month
• v25 and v24 characterization drafts. Jeff asks for stories. https://dmtr-392.lsst.io/v/DM-37345/index.html
  • v23→ v24. Scarlet Lite, PIFF, repeatable star selection for PSF modeling, amp-to-amp offset matching during ISR
  • v24→ v25. analysis_tools, gbdes available (not on), image subtraction refactor, hips trees. 
• Processing Status
• Review the w_2023_15/ DM-38691 rerun:
  
  • Recall history:
    
    • w12: psfex
    • w16: piff  (bad size residuals)
    • w20: finalizeCharacterize (bad apcorr configs → bad stellar locus)
    • w22: lanczos11 + apcorr configs (better stellar locus and size residuals approx equivalent to psfEx!
    • w24: PIFF kernelSize to 25. new scarlet lite storage.
    • w28: attempt at fixing measure failures:  DM-35722 - Getting issue details... STATUS
    • w32: First RC2 at SLAC. subtractImages compatibility mode on
    • w36: subtractImages Compatibility mode off
    • w40: 
    • w44:
    • w48: 9697/7 succeeds, extra subtractImage failures gone
    • w03: Issues with using finalize characterize PSFs downstream. 
    • w07: ip_diffim. Cannot find plugin (fixed on DM-38209) run with ticket branch. Issues with finalize characterize downstream gone. 
    • w11: detectAndMeasure seg faults (fixed with hours to spare for w15) 
    • w15:
  • Squash Old Chronograf, plot-navigator review of _15 https://usdf-rsp.slac.stanford.edu/plot-navigator/dashboard_gen3
  • Sasquatch + analysis_tools chronograph
• w_2023_13 DC2:
• What do we expect next time 
  • astrometric match with reference objects improved: DM-38808 
  • Dynamic detection fix on DM-16724 - Getting issue details... STATUS merged as of w_2023_17
    • Hoping for no change. 
  • Cleared the mask plane of the template: DM-38901  (config.preserveTemplateMask = NO_DATA, BAD, SAT)
  • Proper motion and parallax will be turned on!
• Other notes:
  • DM-35336 - Getting issue details... STATUS is merged (if there's time I can share screen to show some plots)
• AOB:

Notes:

• Expected topics
  • new sky object placement (pseudo-random)
  • campaign management
  • mystery of April 12/13/14
    • Problems in stellar local width Eli:  DM-38733
    • Weekly 2023_9 to 2023_11; also 2023_15
      • Not a bug in the metric (scatter went up)
        • N.b. Jump correlated with "better" aperture corrections, GBDES
        • Nothing made sense for the drop
      • Rerunning all three weeklies
      • Still a mystery
  • (Yusra demonstrates the path to the (old squash) chronograph metrics page)
• We need to move to Sasquatch ASAP
  • There are (a few) metrics!
    • stellar locus
    • sky objects
      • sky aperture flux changed, despite being the same definition
  • Adding metrics during the sprint will dispatch to sasquatch
• Weekly 15
  • Weekly 13 is also done
  • we have metrics etc.
  • Orion:  partly interesting, partly a slog
    • step 2 and 4 problems (as usual)
      • frequently ran out of memory
        • panda no-longer does auto-retries with larger memory
        • we might need to increase the memory requests in drpPipe
    • Otherwise generally seemed fine
      • faro died with network issues 
    • squash isn't accepting metrics
  • Yusra: step 4 failures:
    • Fixed SEGV by changing LMfit stopping criterion fitting for dipoles
      • Didn't fit NaN problems
    • For weekly 19
      • update memory requests
      • fix LMfit again (it's getting NaNs)
  • Jim: are there plots we can retire?
  • Lauren: are the plots we need to port?
  • Clare:  expect big astrometric changes in the next run
    • bright sources should be better than faint – I fixed the bug
  • Reconcile rho plots between old and new code (Arun'll be working on it)
• Jeff's v24/25 characterisation reports
  
  • includes history of the last 8 releases
    • RHL: it'd be nice to draw horizontal lines for SRD thresholds (and maybe put the origin at 0.0?)
    • Yusra: Don't show older metrics (i.e. form before we changed the metric definitions)
  • Didn't expect many changes (PIFF was no-degradation;  Scarlet to Scarlet-lite was computational)
• DC2:
  • Orion:  ran normally
    • Still running _17;  _13 is done
• Erfan's plots
• Lauren has low priority for the end
  • Dynamic detection "fixed".
    • Shouldn't show up for us, matters in the LSB limit
  • Stellar locus 
    • Weekly 11 plots have many issues (analysis drp)
    • Changed colour/density conventions
    • sigma MAD was the most useful, didn't change much
    • added contour plots
      • RHL: shows up S/G problems...
• Ian: there are changes in diffim that might be relevant
  • We're now clearing the template mask plane 
    • makes output template more useful 
      • Currently preserving  NO_DATA, BAD, SAT 
        • DM-38901
• Clare:
  • will turn on proper motion and parallax fitting
    • probably not much effect on our metrics
      • (not much range in epochs; also depends on sky coverage)
• Erfan:
  • Turned on fitAffineWcs 
  • AM1 is slightly improved (post-joint cal) (20 arcmin scale)
    • This could only break things if matching failed
  • For diffim the pre-joint cal is more interesting
  • RHL: y is worse, why?
    • S/N?  Larger r_0 ?
  • Eli: does this make initial WCS fitting more stable/less crazy?
    • Need more area.  Test at USDF with HSC DR2 soon
• Dan: more plots run in CI.  Could break things
• Resource usage coming soon (Yusra)

Action Items