(postponed from 2021-10-04)

Mondays 12pm - 12:50pm ET

Yusra's Zoom: https://princeton.zoom.us/my/yusra


Yusra, Arun, Nate, Jim, Leanne, Eli, Lauren, Jeff, Brock, Lee, Hsin-Fang, Chris (Waters), Clare, Dan, Merlin, Simon, Robert, Josh, Sophie


  • Announcements: Monica will be back for 0.15 FTE. Brock will continue indefinitely.
  • Review Action items from last week
  • Review w_2021_38
    • What was noteworthy in this rerun?
      • Gen2 (DM-31815)
        • LAM: lost some steam on this one with the more pressing DP0.2 issues.  Processing is ongoing and should be done with metrics dispatched in the next couple of days.
        • One change I will look into soon is Eli Rykoff 's fix on  DM-31462 - Getting issue details... STATUS  which should get us (close to) gen2/gen3 parity for fgcm
      • Gen3 (DM-31795)
        • most issues were database connection issues. 
        • Some errors in image different in step4. 
        • bps report claimed that the execution butler failed, but exited with success? Michelle Gower looked into it. Are we worried about it?
        • BB: Metrics produced and may be dispatched. Ping SK and ask why they aren't showing up on the dashboard.
        • Sophie's analysis_drp plots are produced for w34
    • What changed? Annotate
    • What do we expect from w_2021_42?
      • Post column renaming (DM-31825). 
      • faro might be run as part of the steps if tickets gets in before Wed DM-32029
      • ER & LAM: On jointcal, Gen2-Gen3 config differences have to do with column names. Gen2 uses source table. If there are ordering differences, there can be some stochasticity.
      • Changes toward Gen2-Gen3 parity:
        • We also expect some changes (towards parity) on the next run.  I will hopefully get the coadd level exposure and source IDs in sync between the two middlewares on  DM-30284 - Getting issue details... STATUS  (thanks again to Jim Bosch for his help on that one!), so we won't face random seeding difference issues. 
        • One question I have relates to jointcal: I've seen some chatter about obs_subaru gen3 configs moving around...are these configs being kept in sync with gen2?
      • What new metric can we expect next time 
        • YA's plan: raw number counts for DIA sources per unit area. I hear John added jointcal metrics a couple weeks ago
        • LG: New SRD metrics. Can take a wishlist of non-SRD metrics.
        • YA: offset with refcat and matches between band please
        • LK: Sky object metrics. 
  • Review DC2
    • w_2021_36
    • Gen3: Boring
    • Gen2: calexps(?) were same between Gen2 & Gen3
    •  w_2021_40
      • Gen2:  running with all  DM-30284 - Getting issue details... STATUS  branches set up.  Have parity up to mergeDetections, but... see that ticket for details about some nagging coadd PSF differences between gen2 & gen3.
        LAM: Jim says Scarlet can be sensitive to numerical differences. Everything except coadd PSFs are identical.
        JB: If we can't get identical results, and if we can assert that Gen3 results are good enough, we can still retire Gen2.
      • Gen3: w40 - full focal plane. Especially good for astrometric matches. 300,000ish jobs - finished all jobs except `calibrate`. `step2` and `step3` were both fine. On `step4` now.
  • AOB
    - DT showed pull (chi) plots for CModel in DC2 w26. Will show more plots on Wednesday team meeting Oct 20.

Action Items

DescriptionDue dateAssigneeTask appears on
  • Add a plot with fakes stats to the dashboard. Sophie Reed 
04 Sep 2020Sophie ReedDRP Metrics Monitoring 2020-08-07
  • Clare: add analyzeMatchedVisitsCore to drp_pipe step8
DRP Metrics Monitoring 2023-06-26
  • Sophie: make a new list for outstanding analysis_drp plots that require moving, send to Jim
DRP Metrics Monitoring 2023-06-26
  • turn catchFailures on in calibrate. Add flag to indicate that deblender failed because PSF is bad. 
DRP Metrics Monitoring 2022-10-31
  • Yusra AlSayyad Eric's account was deleted; we need to make sure he has all his logs. 
Yusra AlSayyadDRP Metrics Monitoring 2021-06-14
  • Arun Kannawadi Modify rho stats in pipe_analysis  to use debiased moments (see  DM-30751 - Getting issue details... STATUS ). 
Arun KannawadiDRP Metrics Monitoring 2021-04-19
Arun KannawadiDRP Metrics Monitoring 2021-03-01
  • Yusra AlSayyad Do a rerun with w50 PS1 refcat and one with shrunk refcat errors. 
Yusra AlSayyadDRP Metrics Monitoring 2021-01-04
  • Jeffrey Carlin Add an absolute astrometry match-to-refcat metric to dashboard  DM-34153 - Getting issue details... STATUS
Jeffrey CarlinDRP Metrics Monitoring 2021-01-04