Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Invited:

Attendees:

Regrets:

Agenda:

  • Announcements
  • Action items from last time:
    • we don't have a good sense right now if the plots we're looking at are showing data near the "danger zone" or not;
      • We don't know if we're approaching "danger" with the current metrics, as we're not comparing vs. thresholds.
      • JB: should have a policy that metrics are born with those thresholds (and grandfather that in for existing metrics - import from verify_metrics)
        • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
  • Processing Status
    • Processing of w_2024_12 DC2 ( DM-43400 - Getting issue details... STATUS ),  w_2024_16 DC2 ( DM-43972 - Getting issue details... STATUS ) , and  w_2024_14 RC2 DM-43718 - Getting issue details... STATUS  
      • Brian got the w14 RC2 done over eclipse weekend. Thank you Brian. Jen got the w16 DC2 done in 22 hours. Thank you Jen.
  • Review plots and mtrics
  • Recall what we expected for w14:
    • The PSF flux issue in forced_src was fixed March 20 on DM-43376
    • AA1 on the visit level is implemented but not being run. It will be turned on in RC2 and Ops Rehearsal pipelines asap, so should be available by next meeting.
      • Tried to add this, at least for "nightlyvalidation", but now it's not there. Also, need to discuss where/how to run visit-level metrics so that they get dispatched to sasquatch.
      • Clare says she doesn't understand if she has done something wrong with the Chronograph. They did show up for the Ops Rehearsal stuff. It should be fine for the DC2 because it worked for the Ops Rehearsal.
    • Sophie and Trys are working on all-sky plots. Trys: we were thinking about some features for the automatic axis adjustments. Should be ready for the review in the next two weeks.
    • pipetask report 
    • DM-43406 - Getting issue details... STATUS done
    • DM-41049 - Getting issue details... STATUS finally merged - color difference metrics & plots now available for DC2
    • Eli is working on compensated Gaussians
    • DM-41049 - Getting issue details... STATUS DC2 ref match colour diff metrics.
    • DM-37952 - Getting issue details... STATUS Lauren is adding two new per-detector metrics to check on the fidelity of the PSF model and aperture corrections across a detector.  These will have thresholds for inclusion in coadds.  The PSF model issue was uncovered in the HSC-PDR4 where negative PSF models were found in a coadd (see DM-43215 - Getting issue details... STATUS for details).
    • DM-43334 - Getting issue details... STATUS implements the dmL1AstroErr and dmL2AstroErr. The ticket is in review now, so optimistic it should be in w14, but probably not w12. 
      • Recall 2023 history:
        • w03: Issues with using finalize characterize PSFs downstream. 
        • w07: ip_diffim. Cannot find plugin (fixed on DM-38209) run with ticket branch. Issues with finalizeCharacterize downstream gone.
        • w11: GBDES on.  detectAndMeasure segfaults (fixed with hours to spare for w15) 
        • w15: quasi-random skyObject placement (DM-23781), source selectors use isPrimary (DM-39141)
        • w19: Parallax and PM on (DM-37943), astrometric match improved (DM-38808), cleared mask plan of the template. (DM-38901)
        • w23: RA/Dec columns renamed
        • w27: No major pipeline changes
        • w32: Use GAIA DR3 refcat in GBDES
        • w35: 
        • w40: scarlet_lite as a standalone package. 
        • w47: shapeHSM wraps Galsim
        • w48: shapeHSM wraps Galsim without slow down. Scarlet_lite standalone package bug fix for suspected cause of wPerp/PSF-metric regression. 
        • w50:
        • w02: 
        • w06: 
        • w10: moments-based star selector
        • w14: the astrometry repeatability metrics are now done faster. Thus might change the metrics a little bit. Eli: the changes are much smaller than the noise on the metrics in the tests.
        • w16: DM-43885 (capping the number stars going into PSF determination) – this should have zero effects on RC2 because there are zero detectors on RC2 subset. No payload errors (our-problem errors) for this weekly (and w14).
      • w10 vs w14: Eli: the change in outliers in AF might be due to downsampling. Coadd metrics, PSF, background and deblender are pretty much the same.
      • DM-43238 was merged on w11 – no issues.
      • Yusra: We still have some analysis drp plots showing up in the plot navigator.
    • New/updated plots:
  • What pipeline changes do we expect for the w_2024_18 (RC2) or w_2024_20 (DC2)?
    • Raising to low-SNR floor for stars being used for PSF determination (w16) – from 20 to 40 (based on the recommendation from PSF science unit). We are just going to YOLO it and see what happens. This will improve the photometry relative to local background problems. Eli: RC2, DC2 and ComCam sim only. NOT TOUCHING the AuxTel.
    • Compensated Gaussians in w20? 
  • What additional plots or metrics do we expect for the w_2024_18 (RC2) or w_2024_20 (DC2)?
    • AB1/ABF1 was just merged (DM-39256), but is affected by same question as AA1. – Clare: I want to split them up by band, not sure how to do it in the Chronograph yet.
    • Yusra: will all the sky plots be ready by w20? Trys: Yes, definitely! I want to get this to work first and make additional tickets later. Yusra: maybe we should make this an analysis tool (survey-level), would be interesting for the nightly runs.

Jim: It is possible that the JSON handling of NaNs caused the problems of zeros in the Chronograph. At least it is one place to look.
Yusra: we need someone to check!


Show and tell?

  • Actions
  • AOB:
  • Lee Kelvin noticed a ~large increase in the log messages being reported for RC2 subset runs from d_2024_04_19 . 
    • Output logs went from ~65000 lines to ~3.3 million lines.
    • Of these logs, ~65% of lines are WARNING ; of all warnings, ~86% are in relation to Exception in ext_shapeHSM_HsmShapeRegauss :
    • e.g.:

      WARNING 2024-04-21T23:17:04.465-07:00 lsst.characterizeImage.measurement.ext_shapeHSM_HsmShapeRegauss (characterizeImage:{instrument: 'HSC', detector: 42, visit: 322, band: 'y', physical_filter: 'HSC-Y'})(baseMeasurement.py:396) - Exception in ext_shapeHSM_HsmShapeRegauss.measure on record 138388141244473: Unphysical situation: galaxy convolved with PSF is smaller than PSF!
      WARNING 2024-04-21T23:17:04.488-07:00 lsst.characterizeImage.measurement.ext_shapeHSM_HsmShapeRegauss (characterizeImage:{instrument: 'HSC', detector: 47, visit: 17926, band: 'z', physical_filter: 'HSC-Z'})(baseMeasurement.py:396) - Exception in ext_shapeHSM_HsmShapeRegauss.measure on record 7699259306541523: Error: NaN in calculation of adaptive moments
      WARNING 2024-04-21T23:17:04.513-07:00 lsst.characterizeImage.measurement.ext_shapeHSM_HsmShapeRegauss (characterizeImage:{instrument: 'HSC', detector: 42, visit: 23704, band: 'r', physical_filter: 'HSC-R'})(baseMeasurement.py:396) - Exception in ext_shapeHSM_HsmShapeRegauss.measure on record 10180880672751952: Error: HSM collapsed to singular moment matrix. Object is too small.

      Jim: We had a policy that the measurement plugins must raise an exception when encountering issues. Lauren: So, we were getting NaNs but with no flags raised? Jim: Yes. It seems like a good pair-coding project.

    • As far as I (LSK) can tell, the output metrics seem ~ok on the Chronograf. Example recent collection: HSC/runs/RC2_subset/d_2024_04_21 .
    • All RC2 subset log information available here: https://s3df.slac.stanford.edu/people/lskelvin/rc2_subset/

  • Dan: What's the mystery of 12/31/1969? Possibly this is Sasquatch's fault.



Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-06-03
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!Sophie: I haven't made progress
DRP Metrics Monitoring 2024-05-20
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-05-20
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-04-22
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS
DRP Metrics Monitoring 2024-03-18
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
DRP Metrics Monitoring 2023-06-26
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
Yusra AlSayyadDRP Metrics Monitoring 2021-01-04
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04