Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Yusra AlSayyad , Jim Bosch , Robert Lupton , Erfan Nourbakhsh , Lee Kelvin


Regrets: Arun Kannawadi

Agenda:

  • Meeting recorder - (last 6 meetings were:  Arun, Clare, Lee, Colin, Jim, Keith) 
  • Announcements
    • Frossie wants to decomission Squash ASAP
  • Review Action items from last month
  • v25 and v24 characterization drafts. Jeff asks for stories. https://dmtr-392.lsst.io/v/DM-37345/index.html
    • v23→ v24. Scarlet Lite, PIFF, repeatable star selection for PSF modeling, amp-to-amp offset matching during ISR
    • v24→ v25. analysis_tools, gbdes available (not on), image subtraction refactor, hips trees. 
  • Processing Status
  • Review the w_2023_15/ DM-38691 rerun:
    • Recall history:
      • w12: psfex
      • w16: piff  (bad size residuals)
      • w20: finalizeCharacterize (bad apcorr configs → bad stellar locus)
      • w22: lanczos11 + apcorr configs (better stellar locus and size residuals approx equivalent to psfEx!
      • w24: PIFF kernelSize to 25. new scarlet lite storage.
      • w28: attempt at fixing measure failures:  DM-35722 - Getting issue details... STATUS
      • w32: First RC2 at SLAC. subtractImages compatibility mode on
      • w36: subtractImages Compatibility mode off
      • w40: 
      • w44:
      • w48: 9697/7 succeeds, extra subtractImage failures gone
      • w03: Issues with using finalize characterize PSFs downstream. 
      • w07: ip_diffim. Cannot find plugin (fixed on DM-38209) run with ticket branch. Issues with finalize characterize downstream gone. 
      • w11: detectAndMeasure seg faults (fixed with hours to spare for w15) 
      • w15:
    • Squash Old Chronograf, plot-navigator review of _15 https://usdf-rsp.slac.stanford.edu/plot-navigator/dashboard_gen3
    • Sasquatch + analysis_tools chronograph
  • w_2023_13 DC2:
  • What do we expect next time 
    • astrometric match with reference objects improved: DM-38808 
    • Dynamic detection fix on DM-16724 - Getting issue details... STATUS merged as of w_2023_17
      • Hoping for no change. 
    • Cleared the mask plane of the template: DM-38901  (config.preserveTemplateMask = NO_DATA, BAD, SAT)
    • Proper motion and parallax will be turned on!
  • Other notes:
    • DM-35336 - Getting issue details... STATUS is merged (if there's time I can share screen to show some plots)
  • AOB:

Notes:

  • Expected topics
    • new sky object placement (pseudo-random)
    • campaign management
    • mystery of April 12/13/14
      • Problems in stellar local width Eli:  DM-38733
      • Weekly 2023_9 to 2023_11; also 2023_15
        • Not a bug in the metric (scatter went up)
          • N.b. Jump correlated with "better" aperture corrections, GBDES
          • Nothing made sense for the drop
        • Rerunning all three weeklies
        • Still a mystery
    • (Yusra demonstrates the path to the (old squash) chronograph metrics page)
  • We need to move to Sasquatch ASAP
    • There are (a few) metrics!
      • stellar locus
      • sky objects
        • sky aperture flux changed, despite being the same definition
    • Adding metrics during the sprint will dispatch to sasquatch
  • Weekly 15
    • Weekly 13 is also done
    • we have metrics etc.
    • Orion:  partly interesting, partly a slog
      • step 2 and 4 problems (as usual)
        • frequently ran out of memory
          • panda no-longer does auto-retries with larger memory
          • we might need to increase the memory requests in drpPipe
      • Otherwise generally seemed fine
        • faro died with network issues 
      • squash isn't accepting metrics
    • Yusra: step 4 failures:
      • Fixed SEGV by changing LMfit stopping criterion fitting for dipoles
        • Didn't fit NaN problems
      • For weekly 19
        • update memory requests
        • fix LMfit again (it's getting NaNs)
    • Jim: are there plots we can retire?
    • Lauren: are the plots we need to port?
    • Clare:  expect big astrometric changes in the next run
      • bright sources should be better than faint – I fixed the bug
    • Reconcile rho plots between old and new code (Arun'll be working on it)
  • Jeff's v24/25 characterisation reports
    • includes history of the last 8 releases
      • RHL: it'd be nice to draw horizontal lines for SRD thresholds (and maybe put the origin at 0.0?)
      • Yusra: Don't show older metrics (i.e. form before we changed the metric definitions)
    • Didn't expect many changes (PIFF was no-degradation;  Scarlet to Scarlet-lite was computational)
  • DC2:
    • Orion:  ran normally
      • Still running _17;  _13 is done
  • Erfan's plots
  • Lauren has low priority for the end
    • Dynamic detection "fixed".
      • Shouldn't show up for us, matters in the LSB limit
    • Stellar locus 
      • Weekly 11 plots have many issues (analysis drp)
      • Changed colour/density conventions
      • sigma MAD was the most useful, didn't change much
      • added contour plots
        • RHL: shows up S/G problems...
  • Ian: there are changes in diffim that might be relevant
    • We're now clearing the template mask plane 
      • makes output template more useful 
        • Currently preserving  NO_DATA, BAD, SAT 
          • DM-38901
  • Clare:
    • will turn on proper motion and parallax fitting
      • probably not much effect on our metrics
        • (not much range in epochs; also depends on sky coverage)
  • Erfan:
    • Turned on fitAffineWcs 
    • AM1 is slightly improved (post-joint cal) (20 arcmin scale)
      • This could only break things if matching failed
    • For diffim the pre-joint cal is more interesting
    • RHL: y is worse, why?
      • S/N?  Larger r_0 ?
    • Eli: does this make initial WCS fitting more stable/less crazy?
      • Need more area.  Test at USDF with HSC DR2 soon
  • Dan: more plots run in CI.  Could break things
  • Resource usage coming soon (Yusra)

Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
DRP Metrics Monitoring 2023-06-26
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
Yusra AlSayyadDRP Metrics Monitoring 2021-01-04
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04