Skip to end of metadata
Go to start of metadata

Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom:


Yusra AlSayyad, Sophie Reed, Lee Kelvin, Erfan Nourbakhsh, Fred Moolekamp, Christopher Waters, Dan Taranu, Clare Saunders, Keith Bechtol, Agnès Ferté, Jeffrey Carlin, Joshua Meyers, Orion Eiger, Robert Lupton, Eli Rykoff, Colin Slater, Arun Kannawadi



  • Meeting recorder - (last 6 meetings were:  Lee, Colin, Jim, Keith, Fred, Nate) 
  • Announcements
  • Review Action items from last month
    • Orion Eiger still looking into tract 9697 patch 7 success mystery in w_2022_48.
    • Hsin-Fang Chiang solved HSC failures for analysis_tools → instrument = HSC information was not in the YAMLs;
      • Dan Taranu added instrument into DC2, but nothing changed (yet) for HSC pipeline (outstanding) → see DM-37784;
      • RC2 subset repo scripts should also be updated to account for this missing instrument information (perhaps migrating into drp_pipe) → see DM-37984;
    • Sophie Reed wrapping up new ref cat plots, TBD shortly.
  • Processing Status
    • failures with finalizeCharacterization and updateVisitSummary:
      • potential cause likely linked to visit shredding;
        • running per-visit (i.e., dimensions = just visit); no tract constraints;
        • however, one of the input dimensions is tract, so the butler only giving detector inputs that match the tract;
        • similar issue with finalizeCharacterization, with bright source catalog being shredded on input.
      • Jim working on a longer-term solution (automatic data query manipulation) for w07 → DM-37786;
      • V&V is going to add checks that full visit data products have data points for full visits. 
  • Review the cursed w_2023_03 rerun:
    • Same error messages as before, details are on the Jira ticket.  DM-37483.
    • Recall history:
      • w12: psfex
      • w16: piff  (bad size residuals)
      • w20: finalizeCharacterize (bad apcorr configs → bad stellar locus)
      • w22: lanczos11 + apcorr configs (better stellar locus and size residuals approx equivalent to psfEx!
      • w24: PIFF kernelSize to 25. new scarlet lite storage.
      • w28: attempt at fixing measure failures:  DM-35722 - Getting issue details... STATUS
      • w32: First RC2 at SLAC. subtractImages compatibility mode on
      • w36: subtractImages Compatibility mode off
      • w40: 
      • w44:
      • w48: 9697/7 succeeds, extra subtractImage failures gone, 
    • Chronograf, plot-navigator review of _03
      • Metrics did get loaded (thanks Hsin-Fang Chiang and Angelo Fausti!)
      • metrics look ~ok, likely because data from edge regions have not been included.
      • sky object metrics show considerable jumps, again likely due to missing data
      • New wPerp analysis_tools plots now being produced → can retire the older analysis_drp versions!
        • differing magnitude limits (and potentially differing S/N differences) from prior run making it difficult for direct comparison
  • w_2023_05 DC2 was scrapped because of a ctrl_bps_panda bug. Orion running it on w_2023_06 instead. 
  • What do we expect next time 
    • Duplicate analysis_drp plots to be retired: wperp
    • Following testing in DM-37141, switching to gbdes WCSFit in place of jointcal for future HSC runs
      • results are very promising, even without optimization, already reproducing current expected functionality;
      • AM metric (RMS repeatability of positional information) already better following testing updates;
      • no change in AB metrics following switch to gbdes (as expected);
      • change noted in sky object standard deviations (perhaps unexpected);
      • changes also noted in TE1 and TE2;
      • stellar locus almost the same
      • Runtime: better by a factor of 4 (worst-case scenario) :tada:
      • Memory usage: also better by a factor of 4 (worst-case scenario) :tada: :tada:
      • aim to switch this on for w11 (i.e., after a successful w07 HSC run is in place)
    • Photometry reference catalog plots → Sophie Reed (DM-37727)
  • AOB:

Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
DRP Metrics Monitoring 2023-06-26
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
Yusra AlSayyadDRP Metrics Monitoring 2021-01-04
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04