Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Regrets:

Agenda:

  • Announcements
  • Action items from last time:
    • we don't have a good sense right now if the plots we're looking at are showing data near the "danger zone" or not;
      • We don't know if we're approaching "danger" with the current metrics, as we're not comparing vs. thresholds.
      • JB: should have a policy that metrics are born with those thresholds (and grandfather that in for existing metrics - import from verify_metrics)
        • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS
  • Processing Status
    • Processing of w_2024_08 DC2 and  w_2024_10 RC2 (Jen/Orion)
      • CM got RC2 completed in 4 days; hats removed
  • Review the w_2024_10 DM-43178 - Getting issue details... STATUS rerun:
  • Recall what we expected for w10:
    • DM-42837 - Getting issue details... STATUS (Hopefully this changes nothing. Would only affect DC2. )
    • DM-41876 - Getting issue details... STATUS  should reduce the number of diaSources. 
    • DM-42663 - Getting issue details... STATUS moments-based star/galaxy classifier. To be tested on ComCamSim1 before March 6. Made small changes in RC2.
    • Large drop in yPerp noted in DC2 w08 from stellar locus ticket
    • wPerp dropped to zero in rc2_subset sometime between Mar 11-13:
      • Perhaps related to: https://rubinobs.atlassian.net/issues/DM-39081
      • Suspicion around arrowAstropy, but also coadd PSF metrics went bad, so maybe not (Eli tentaively assigned to investigate)
      • impacted metrics on the Chronograf:
        • wPerp_psfFlux_sigmaMAD
        • yPerp_psfFlux_sigmaMAD
        • shapeSizeFractionalDiff metrics
        • e1Diff / e2Diff
    • Per-tract AA1 – need to make a dashboard for this
    • Sophie's working on all-sky plots; not sure if these will be ready for w_10
      • Recall 2023 history:
        • w03: Issues with using finalize characterize PSFs downstream. 
        • w07: ip_diffim. Cannot find plugin (fixed on DM-38209) run with ticket branch. Issues with finalizeCharacterize downstream gone.
        • w11: GBDES on.  detectAndMeasure segfaults (fixed with hours to spare for w15) 
        • w15: quasi-random skyObject placement (DM-23781), source selectors use isPrimary (DM-39141)
        • w19: Parallax and PM on (DM-37943), astrometric match improved (DM-38808), cleared mask plan of the template. (DM-38901)
        • w23: RA/Dec columns renamed
        • w27: No major pipeline changes
        • w32: Use GAIA DR3 refcat in GBDES
        • w35: 
        • w40: scarlet_lite as a standalone package. 
        • w47: shapeHSM wraps Galsim
        • w48: shapeHSM wraps Galsim without slow down. Scarlet_lite standalone package bug fix for suspected cause of wPerp/PSF-metric regression. 
        • w50:
        • w02: 
        • w06: 
        • w10: moments-based star selector
    • New/updated plots:
      • shapeSizeFractionalDiff on ops_rehearsal_3 looks better thanks to brighter-fatter correction (but still not ideal at very bright end)
      • some discussion about reducing number of skyObjects requested in rehearsal crowded fields w/new skymap (smaller patches)
  • What pipeline changes do we expect for the w_2024_12 (DC2 and ops-rehearsal templates) or w_2024_16 (RC2)?
    • pipetask report 
    • Move objectExtended plots to Core if needed (Dan, TBD)
    • Eli is working on compensated Gaussians
  • What additional plots or metrics do we expect for the w_2024_12 (DC2) or w_2024_16(RC2)?
    • DM-41049 - Getting issue details... STATUS DC2 ref match colour diff metrics probably won't get in before w_2024_12
    • DM-37952 - Getting issue details... STATUS Lauren is adding two new per-detector metrics to check on the fidelity of the PSF model and aperture corrections across a detector.  These will have thresholds for inclusion in coadds.  The PSF model issue was uncovered in the HSC-PDR4 where negative PSF models were found in a coadd (see DM-43215 - Getting issue details... STATUS for details).
    • DM-43334 implements the dmL1AstroErr and dmL2AstroErr. The ticket is in review now, so optimistic it should be in w14, but probably not w12.
    • AA1 on the visit level is implemented but not being run. It will be turned on in RC2 and Ops Rehearsal pipelines asap, so should be available by next meeting.

ComCamSim Run:


Show and tell?

  • Actions
  • AOB:




Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-06-03
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!Sophie: I haven't made progress
DRP Metrics Monitoring 2024-05-20
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-05-20
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-04-22
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS
DRP Metrics Monitoring 2024-03-18
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
DRP Metrics Monitoring 2023-06-26
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
Yusra AlSayyadDRP Metrics Monitoring 2021-01-04