Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Invited:

Attendees:

Regrets:

Agenda:

  • Announcements
  • Action items from last time:
    • All,  reduce the plugin warnings after the Logplosion.  (Thank you Lee for coordinating)
    • Look into who is converting NaNs to 0 in the metrics that appear on sasquatch. 
    • What's the mystery of 12/31/1969? 
      • Dan Taranu: "I looked into this, it was user error"
    • We don't have a good sense right now if the plots we're looking at are showing data near the "danger zone" or not;
      • We don't know if we're approaching "danger" with the current metrics, as we're not comparing vs. thresholds.
      • JB: should have a policy that metrics are born with those thresholds (and grandfather that in for existing metrics - import from verify_metrics)
        • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
  • Processing Status
    • Processing of w_2024_18 RC2  ( DM-44183 - Getting issue details... STATUS
      • 3 flavors of problems this time:
        • DM-41507 - Getting issue details... STATUS (thank you Eli!) 
        • DM-44272 - Getting issue details... STATUS (Not reproduceable, going to see if it happens again w_2024_22)
        • DM-44333 - Getting issue details... STATUS (need someone to look into this) 
  • Review plots and mtrics
  • Recall what we expected for w18:
    • Raising to low-SNR floor for stars being used for PSF determination (w16) – from 20 to 40 (based on the recommendation from PSF science unit). We are just going to YOLO it and see what happens. This will improve the photometry relative to local background problems. Eli: RC2, DC2 and ComCam sim only. NOT TOUCHING the AuxTel.
    • Tanaka-san plots
    • AA1 on the visit level is implemented but not being run. It will be turned on in RC2 and Ops Rehearsal pipelines asap, so should be available by next meeting.
      • Tried to add this, at least for "nightlyvalidation", but now it's not there. Also, need to discuss where/how to run visit-level metrics so that they get dispatched to sasquatch.
      • Clare says she doesn't understand if she has done something wrong with the Chronograph. They did show up for the Ops Rehearsal stuff. It should be fine for the DC2 because it worked for the Ops Rehearsal.
    • DM-37952 - Getting issue details... STATUS Lauren is adding two new per-detector metrics to check on the fidelity of the PSF model and aperture corrections across a detector.  These will have thresholds for inclusion in coadds.  The PSF model issue was uncovered in the HSC-PDR4 where negative PSF models were found in a coadd (see DM-43215 - Getting issue details... STATUS for details). 
    • Recall 2023-2024 history:
      • w03: Issues with using finalize characterize PSFs downstream. 
      • w07: ip_diffim. Cannot find plugin (fixed on DM-38209) run with ticket branch. Issues with finalizeCharacterize downstream gone.
      • w11: GBDES on.  detectAndMeasure segfaults (fixed with hours to spare for w15) 
      • w15: quasi-random skyObject placement (DM-23781), source selectors use isPrimary (DM-39141)
      • w19: Parallax and PM on (DM-37943), astrometric match improved (DM-38808), cleared mask plan of the template. (DM-38901)
      • w23: RA/Dec columns renamed
      • w27: No major pipeline changes
      • w32: Use GAIA DR3 refcat in GBDES
      • w35: 
      • w40: scarlet_lite as a standalone package. 
      • w47: shapeHSM wraps Galsim
      • w48: shapeHSM wraps Galsim without slow down. Scarlet_lite standalone package bug fix for suspected cause of wPerp/PSF-metric regression. 
      • w50:
      • w02: 
      • w06: 
      • w10: moments-based star selector
      • w14: the astrometry repeatability metrics are now done faster. Thus might change the metrics a little bit. Eli: the changes are much smaller than the noise on the metrics in the tests.
      • w16: DM-43885 (capping the number stars going into PSF determination) – this should have zero effects on RC2 because there are zero detectors on RC2 subset. No payload errors (our-problem errors) for this weekly (and w14).
    • New/updated plots:
  • What pipeline changes do we expect for the w_2024_22(RC2) or w_2024_20 (DC2)?
    • Compensated Top Hats in w22?
  • What additional plots or metrics do we expect for the  w_2024_22(RC2) or w_2024_20 (DC2)?


Show and tell?

  • Actions
  • AOB:


Minutes

  • Announcements
    • Andres → community science team after paternity leave
    • Yusra: we told Princeton that they should keep us around so that we can help them when Rubin comes online.
      • Jenny Greene wants to do a precursor Swift project, light curves etc
      • Amy Secunda has been coming to pair coding and wants to work with people, we should do that, let Yusra know.
  • Action items from last time:
    • All,  reduce the plugin warnings after the Logplosion.  (Thank you Lee for coordinating) log information available here: https://s3df.slac.stanford.edu/people/lskelvin/rc2_subset/
      • Lee will update, progress is being made (see graph attached above).
      • Eli: much more storage space being used by the processing, this will effect V27
      • Jim: Backport the tickets? Eli: too many tickets to backport, maybe revert? Jim: backport the most reducive? Eli: Just undo the change that exposed the errors.
      • Jim: We are now a factor of 10 above before. Logs probably don't need to be dumped both to the submit directory and the butler.
      • Eric/Eli/Yusra: Should people run code from somewhere else?
      • Eric: Do we need the non butler version of the logs for some reason?
      • Eli: Current logs aren't useful anyway, only useful when they aren't so big that the storage space isn't an issue.
      • Lauren: sometimes the logs are the only source of the numbers we need for QA, can use butler logs for this
    • Look into who is converting NaNs to 0 in the metrics that appear on sasquatch.
    •  What's the mystery of 12/31/1969? 
      • user error
    • we don't have a good sense right now if the plots we're looking at are showing data near the "danger zone" or not;
      • We don't know if we're approaching "danger" with the current metrics, as we're not comparing vs. thresholds.
      • JB: should have a policy that metrics are born with those thresholds (and grandfather that in for existing metrics - import from verify_metrics)
        • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!Sophie: I haven't made progress
  • Processing Status
    • Yusra: streak of 4 day RC2/DC2 has been broken
      • Seg faults, source tables only included detectors that only overlapped tracts,  
    • Processing of w_2024_18 RC2  ( DM-44183 - Getting issue details... STATUS
      • 3 flavors of problems this time:
        • DM-44183 - Getting issue details... STATUS (thank you Eli!) 
        • DM-44272 - Getting issue details... STATUS (Not reproduceable, going to see if it happens again w_2024_22)
        • DM-44183 - Getting issue details... STATUS (need someone to look into this) 
    • Clare: Can we assume that source tables and pre source tables are the same size? Yusra: Yes
    • OTC died on one tract, need someone to look into this, Eli volunteered.  Eli is the best.
  • Review plots and metrics
    • https://usdf-rsp.slac.stanford.edu/plot-navigator/dashboard_gen3
      • Clare: DML2: RMS of isolated point sources after global calibration, DML1 same thing but before global calibration
        • For each star the RMS of its calibrated posisitions, metric is median of all stars on tract.
        • Need separate panel for Dec ones
      • Eli: I was looking at things for other reasons, wPerp from 14 to 18 changed, got slightly worse, photometric repeatability were basically identical
      • Yusra: shapeSizeFractionalDiff has changed between analysis_drp and analysis_tools, analysis_tools not clipping outliers
        • All scatterPlotWithtwoHists might be broken - can someone look into
      • Eli: wPerp between w14 and w18, cModel 9813 changed, PSF did not, 9613 both changed
        • yusra: looks like a shift
        • Eli: has a theory involving aperture corrections
        • Eli: aperture correction code is not good at the sub percent level, "I am very vexed", will dig more
          • Changing to higher S/N cut has not made a big difference
      • Lauren: why does forced phot ccd take different time for different bands? Yusra/JimEli: more visits
    • https://usdf-rsp-dev.slac.stanford.edu/chronograf/


  • Pipeline Changes for w22
    • Eli: compensated top hats, needs to talk to Jim
    • Clare: hoping to turn on multiband gbdes, doing some more QA
    • Lauren: two new metrics for calibration fidelity, needs to process lots of data for this, Eli: is reviewing, hopefully for next Wednesday
    • Eli: FGCM plots will be butlerised, can be plot browsered
    • Arun: focal plane plot version of shapeSizeFractionalDiff, also some others
    • Yusra: any sprint results that will influence DRP? Silence so probably not.


  • Recall what we expected for w18:
    • Raising to low-SNR floor for stars being used for PSF determination (w16) – from 20 to 40 (based on the recommendation from PSF science unit). We are just going to YOLO it and see what happens. This will improve the photometry relative to local background problems. Eli: RC2, DC2 and ComCam sim only. NOT TOUCHING the AuxTel.
    • Tanaka-san plots
    • AA1 on the visit level is implemented but not being run. It will be turned on in RC2 and Ops Rehearsal pipelines asap, so should be available by next meeting.
      • Tried to add this, at least for "nightlyvalidation", but now it's not there. Also, need to discuss where/how to run visit-level metrics so that they get dispatched to sasquatch.
      • Clare says she doesn't understand if she has done something wrong with the Chronograph. They did show up for the Ops Rehearsal stuff. It should be fine for the DC2 because it worked for the Ops Rehearsal.
    • DM-37952 - Getting issue details... STATUS Lauren is adding two new per-detector metrics to check on the fidelity of the PSF model and aperture corrections across a detector.  These will have thresholds for inclusion in coadds.  The PSF model issue was uncovered in the HSC-PDR4 where negative PSF models were found in a coadd (see DM-43215 - Getting issue details... STATUS for details). 
      • Raised S/N thresholds
      • Tanaka San plots: should be ready for next week, currently in review, was blocked by another ticket.
      • Clare has added some new metrics
      • Some metrics not coming through from RC2, sasquatch butler not being used.
    • New/updated plots:
  • What pipeline changes do we expect for the w_2024_22(RC2) or w_2024_20 (DC2)?
    • Compensated Top Hats in w22?
    • multiband gbdes 
  • What additional plots or metrics do we expect for the  w_2024_22(RC2) or w_2024_20 (DC2)?
    • FGCM QA plots are being butlerized. 
    • shapeSize fractional diff binned by focal plane position


Show and tell?

  • Actions
  • AOB:



Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-06-03
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!Sophie: I haven't made progress
DRP Metrics Monitoring 2024-05-20
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-05-20
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
DRP Metrics Monitoring 2024-04-22
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS
DRP Metrics Monitoring 2024-03-18
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
DRP Metrics Monitoring 2023-06-26
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
Yusra AlSayyadDRP Metrics Monitoring 2021-01-04
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04