DRP Metrics Monitoring 2023-04-17

Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Yusra AlSayyad, Lee Kelvin, Erfan Nourbakhsh, Fred Moolekamp, Clare Saunders, Orion Eiger, Eli Rykoff, Colin Slater, Lauren MacArthur , Hsin-Fang Chiang , Jim Bosch , Arun Kannawadi Christopher Waters Jeffrey Carlin 

Regrets:

Agenda:

• Meeting recorder -  Arun Kannawadi (last 6 meetings were:  Clare, Lee, Colin, Jim, Keith, Fred, Nate) 
• Announcements
  • analysis_tools Metric Bundles are deployed to Sasquatch!!! Behold!  https://usdf-rsp-dev.slac.stanford.edu/chronograf (Yusra if Nate not available)
    • Need to setup a dashboard before next metrics meeting (may be Dan)
• Review Action items from last month
  • Segfaults is now fixed (thanks Eli!)
  • Proper motion from Gaia makes visit-level plots a bit worse, but we don't see them in the meeting. Not much effect on the coadd plots. - Clare
  • Stellar locus width metrics had a huge jump early March, and dropped some 3 days ago. - Jeff.
    • Could be a combination of two events - GBDES and aperture corrections. We know what changes they correspond to, but don't know why. No tickets jump out. - Yusra.
    • Sky Object placement could also affect the dynamic detection (Stellar Locus Width is measured on coadds) - Lauren
    • File a ticket to investigate stellar locus RC2_SUBSET sensitivity March 8, March 10, April 14. (and remmeber action from last time to  add numbers of stars used in the fit as a metric itself) add label =  "pipelines-support" (DM-38733)
    • We need to understand why things got worse, since we always try to improve - Jim. 
    • Cannot fetch logs from butler, because these are from nightly CI jobs and need to fetch them from Jenkins Artifacts. - Jeff. 
• Processing Status
  • w_2023_11 - DM-38360
  • Stalled because of BPS bugs
  • Segfault in ip_diffim, and Eli fixed it.
  • Starting with CM means we found a whole lot of CM bugs that we now fixed.
  • Aperture corrections failing on detectors 102, 103 [is expected]. We allow some aperture corrections to fail, and we don't know why Kron is not among them - Eli
  • Even in weekly 15, we got same failures involving aperture corrections - Orion. Orion should also complain about pipeline failures. Don't know why dispatch failed. Any help in the coming days is appreciated.
  • We can't have 3 pipelines failures every RC2 by default. Check that they won't be included in the coadds per new requirements. Then figure out how to handle. File a ticket.
• Review the w_2023_11/ DM-38360 rerun:
  
  • Recall history:
    
    • w12: psfex
    • w16: piff  (bad size residuals)
    • w20: finalizeCharacterize (bad apcorr configs → bad stellar locus)
    • w22: lanczos11 + apcorr configs (better stellar locus and size residuals approx equivalent to psfEx!
    • w24: PIFF kernelSize to 25. new scarlet lite storage.
    • w28: attempt at fixing measure failures:  DM-35722 - Getting issue details... STATUS
    • w32: First RC2 at SLAC. subtractImages compatibility mode on
    • w36: subtractImages Compatibility mode off
    • w40: 
    • w44:
    • w48: 9697/7 succeeds, extra subtractImage failures gone
    • w03: Issues with using finalize characterize PSFs downstream. 
    • w07: ip_diffim. Cannot find plugin (fixed on DM-38209) run with ticket branch. Issues with finalize characterize downstream gone. 
    • w11: detectAndMeasure seg faults (fixed with hours to spare for w15) 
  • Chronograf, plot-navigator review of _11
    • Typing doesn't fetch the relevant plot - Lauren
    • Eric says Jim signed off on the naming, but it seems very noisy.
    • Jointcal in w07 → GBDES in w11, they look basically the same in astrometry plots barring scatter. - Clare
• w_2023_09 DC2:
• What do we expect next time 
  • Sky object placement (pseudorandom) 
  • Whole run in CM. - Orion
  • Solving the mystery that happened in 12,13,14
• Other notes:
• AOB:

LM: a few observations while poking around at the plots:

• w07 on the left and w11 on the right. Need to improve formatting. Nothing earth shattering otherwise - Lauren
• skyObjectHist plot labels are fixed (but we may need to rethink the numerical formatting...):
• Something is amiss with the new rho stats plots - Arun Kannawadi will create a ticket to investigate this:

analysis_drp version:

analysis_tools version:

• 
• [stellar locus plots are still "wrong" (see DM-35336 - Getting issue details... STATUS  for details ) ]

Action Items