Mondays 12PT (3 - 3:50pm ET)

Yusra's Zoom: https://princeton.zoom.us/my/yusra

1.1.1. Attendees: 


Regrets: Lauren MacArthur (dr.’s appointment)


1.1.2. Agenda:

  • Meeting recorder - Colin (last 6 meetings were:  Jim, Keith, Fred, Nate, Jeff, Clare) 
  • Announcements
  • Review Action items from last month
    • Lauren MacArthur :  DM-37411 - Getting issue details... STATUS  merged, so should no longer see failures in calibrate.deblender along the lines: Execution of task 'calibrate' on quantum {instrument: 'HSC', detector: 101, visit: 11706, ...} failed. Exception ValueError: PSF at (1908.3, 3245.7) has an invalid FWHM value of nan
    • Jim added PSF-based selection of images for coadds, but not in time w01 which is this week's rerun.
    • Wanted to start CharImage using PSFEx asap for NAOJ, give more time for other Piff Robustification work. Will find out how many non-robustnesses remain with w03. Were still some images that didn't have an obvious excuse for failing, will see how they turn out in w03.
    • Still outstanding why 9697 patch 7 succeeded in w48.
      • Double check that it was actually w48 that worked? Orion Eiger  double checked, but can triple check. → Slack.
    • analysis_tools failure in 48 was a mystery, but has been solved. Hsin-Fang Chiang  : see this comment on the w48 ticket for the longer explanation. For this week (w_2023_01), had to set the instrument to be HSC, then runs fine.
      • For other yamls, the instrument was built into the yaml, didn't need to add --instrument. Why is this different? Hsin-Fang Chiang : we don't have a yaml in drp_pipe for this, should we add one? Yes  (new ticket --> DM-37784)
      • Policy proposal: "never run pipelines from a yaml that's not in drp_pipe"? Maybe not never, but anything that campaign management runs should be in drp_pipe.
    • Add sky coverage plots: Lauren wanted feedback on what ones we wanted, but the capability is there. Sophie has bugfixes on the todo list.
  • Processing Status
  • Review w_2023_01 rerun:
    • Same error messages as before, details are on the Jira ticket.  DM-37483.
    • Recall history:
      • w12: psfex
      • w16: piff  (bad size residuals)
      • w20: finalizeCharacterize (bad apcorr configs → bad stellar locus)
      • w22: lanczos11 + apcorr configs (better stellar locus and size residuals approx equivalent to psfEx!
      • w24: PIFF kernelSize to 25. new scarlet lite storage.
      • w28: attempt at fixing measure failures:  DM-35722 - Getting issue details... STATUS
      • w32: First RC2 at SLAC. subtractImages compatibility mode on
      • w36: subtractImages Compatibility mode off
      • w40: 
      • w44:
      • w48: 9697/7 succeeds, extra subtractImage failures gone, 
    • Chronograf, plot-navigator review of _01
    • New Plots:
      • skyObjectHistPlot, added a one-arc-second GaaP plots. .
        • Conclusion: Both gaap and aperture underestimate the noise. S/N is per-tract but not per band? Plot title is wrong, plot is single-band.
        • Eli Rykoff: Normalize left plot as Flux per square arcsecond? Because aperture is changing between algorithms. Noise-effective area would be even better?
      • New Rho stats plot, combines three tracts. Was hoping that a bias was coming from only having small areas, but even after combining the result was still biased.
        • How did you combine the tracts? Runs on all tracts specified in the data query.
    • Interlude while waiting for plots:
      • Spoiler alert. More details on Wednesday
        • Looking at source injection, injecting quasars into HSC. Seeing z and y offsets.
    • Computational performance:
      • Jim wrote build-gather-resource-usage-qg. One row for every data ID, gives memory and runtime for every task.
      • Orion Eiger  will run after RC2/DC2, so we can make a chronograph dashboard. Monitor when things change or get worse. Hsin-Fang Chiang ran on w01. Wouldn't work in bps (have to find how to tell bps to use an existing qgraph), but worked with pipetask.
    • Next run: Sophie expects photometry vs ref cat plots.
  • AOB:
    • DC2 will be run on the version tagged this upcoming wednesday.
    • Next meeting after w03 is done.


1.1.3. Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS : We need to talk about this when Sophie is back!
 DRP Metrics Monitoring 2024-04-22
  • Sophie to add field in metric definition to hold thresholds. DM-43364 - Getting issue details... STATUS
DRP Metrics Monitoring 2024-03-18
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
 DRP Metrics Monitoring 2023-06-26
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
 DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
 DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
 Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
 Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
 Yusra AlSayyadDRP Metrics Monitoring 2021-01-04