DRP Metrics Monitoring 2023-02-13

Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Yusra AlSayyad, Sophie Reed, Lee Kelvin, Erfan Nourbakhsh, Fred Moolekamp, Christopher Waters, Dan Taranu, Clare Saunders, Keith Bechtol, Agnès Ferté, Jeffrey Carlin, Joshua Meyers, Orion Eiger, Robert Lupton, Eli Rykoff, Colin Slater, Arun Kannawadi

Regrets:

• Arun Kannawadi  (may be able to join after 12:30pm PT/3:30pm ET)
• Lauren MacArthur 

Agenda:

• Meeting recorder - (last 6 meetings were:  Lee, Colin, Jim, Keith, Fred, Nate) 
• Announcements
• Review Action items from last month
  • Orion Eiger still looking into tract 9697 patch 7 success mystery in w_2022_48.
  • Hsin-Fang Chiang solved HSC failures for analysis_tools → instrument = HSC information was not in the YAMLs;
    • Dan Taranu added instrument into DC2, but nothing changed (yet) for HSC pipeline (outstanding) → see DM-37784;
    • RC2 subset repo scripts should also be updated to account for this missing instrument information (perhaps migrating into drp_pipe) → see DM-37984;
  • Sophie Reed wrapping up new ref cat plots, TBD shortly.
• Processing Status
  • failures with finalizeCharacterization and updateVisitSummary:
    • potential cause likely linked to visit shredding;
      
      • running per-visit (i.e., dimensions = just visit); no tract constraints;
      • however, one of the input dimensions is tract, so the butler only giving detector inputs that match the tract;
      • similar issue with finalizeCharacterization, with bright source catalog being shredded on input.
    • Jim working on a longer-term solution (automatic data query manipulation) for w07 → DM-37786;
    • V&V is going to add checks that full visit data products have data points for full visits. 
• Review the cursed w_2023_03 rerun:
  • Same error messages as before, details are on the Jira ticket.  DM-37483.
  • Recall history:
    
    • w12: psfex
    • w16: piff  (bad size residuals)
    • w20: finalizeCharacterize (bad apcorr configs → bad stellar locus)
    • w22: lanczos11 + apcorr configs (better stellar locus and size residuals approx equivalent to psfEx!
    • w24: PIFF kernelSize to 25. new scarlet lite storage.
    • w28: attempt at fixing measure failures:  DM-35722 - Getting issue details... STATUS
    • w32: First RC2 at SLAC. subtractImages compatibility mode on
    • w36: subtractImages Compatibility mode off
    • w40: 
    • w44:
    • w48: 9697/7 succeeds, extra subtractImage failures gone, 
  • Chronograf, plot-navigator review of _03
    • Metrics did get loaded (thanks Hsin-Fang Chiang and Angelo Fausti!)
    • metrics look ~ok, likely because data from edge regions have not been included.
    • sky object metrics show considerable jumps, again likely due to missing data
    • New wPerp analysis_tools plots now being produced → can retire the older analysis_drp versions!
      • differing magnitude limits (and potentially differing S/N differences) from prior run making it difficult for direct comparison
• w_2023_05 DC2 was scrapped because of a ctrl_bps_panda bug. Orion running it on w_2023_06 instead. 
• What do we expect next time 
  • Duplicate analysis_drp plots to be retired: wperp
  • Following testing in DM-37141, switching to gbdes WCSFit in place of jointcal for future HSC runs
    • results are very promising, even without optimization, already reproducing current expected functionality;
    • AM metric (RMS repeatability of positional information) already better following testing updates;
    • no change in AB metrics following switch to gbdes (as expected);
    • change noted in sky object standard deviations (perhaps unexpected);
    • changes also noted in TE1 and TE2;
    • stellar locus almost the same
    • Runtime: better by a factor of 4 (worst-case scenario) :tada:
    • Memory usage: also better by a factor of 4 (worst-case scenario) :tada: :tada:
    • aim to switch this on for w11 (i.e., after a successful w07 HSC run is in place)
  • Photometry reference catalog plots → Sophie Reed (DM-37727)
• AOB:

Action Items