DRP Metrics Monitoring 2021-10-11

(postponed from 2021-10-04)

Mondays 12pm - 12:50pm ET

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Yusra, Arun, Nate, Jim, Leanne, Eli, Lauren, Jeff, Brock, Lee, Hsin-Fang, Chris (Waters), Clare, Dan, Merlin, Simon, Robert, Josh, Sophie

Agenda:

• Announcements: Monica will be back for 0.15 FTE. Brock will continue indefinitely.
• Review Action items from last week
• Review w_2021_38
  • What was noteworthy in this rerun?
    • Gen2 (DM-31815)
      • LAM: lost some steam on this one with the more pressing DP0.2 issues.  Processing is ongoing and should be done with metrics dispatched in the next couple of days.
      • One change I will look into soon is Eli Rykoff 's fix on  DM-31462 - Getting issue details... STATUS  which should get us (close to) gen2/gen3 parity for fgcm

• • • Gen3 (DM-31795)
      • most issues were database connection issues. 
      • Some errors in image different in step4. 
      • bps report claimed that the execution butler failed, but exited with success? Michelle Gower looked into it. Are we worried about it?
      • BB: Metrics produced and may be dispatched. Ping SK and ask why they aren't showing up on the dashboard.
      • Sophie's analysis_drp plots are produced for w34
  • What changed? Annotate
  • What do we expect from w_2021_42?
    • Post column renaming (DM-31825). 
    • faro might be run as part of the steps if tickets gets in before Wed DM-32029
    • ER & LAM: On jointcal, Gen2-Gen3 config differences have to do with column names. Gen2 uses source table. If there are ordering differences, there can be some stochasticity.
    • Changes toward Gen2-Gen3 parity:
      • We also expect some changes (towards parity) on the next run.  I will hopefully get the coadd level exposure and source IDs in sync between the two middlewares on  DM-30284 - Getting issue details... STATUS  (thanks again to Jim Bosch for his help on that one!), so we won't face random seeding difference issues. 
      • One question I have relates to jointcal: I've seen some chatter about obs_subaru gen3 configs moving around...are these configs being kept in sync with gen2?
    • What new metric can we expect next time 
      • YA's plan: raw number counts for DIA sources per unit area. I hear John added jointcal metrics a couple weeks ago
      • LG: New SRD metrics. Can take a wishlist of non-SRD metrics.
      • YA: offset with refcat and matches between band please
      • LK: Sky object metrics. 
• Review DC2
  • w_2021_36
  • Gen3: Boring
  • Gen2: calexps(?) were same between Gen2 & Gen3
  •  w_2021_40
    • Gen2:  running with all  DM-30284 - Getting issue details... STATUS  branches set up.  Have parity up to mergeDetections, but... see that ticket for details about some nagging coadd PSF differences between gen2 & gen3.
      LAM: Jim says Scarlet can be sensitive to numerical differences. Everything except coadd PSFs are identical.
      JB: If we can't get identical results, and if we can assert that Gen3 results are good enough, we can still retire Gen2.
    • Gen3: w40 - full focal plane. Especially good for astrometric matches. 300,000ish jobs - finished all jobs except `calibrate`. `step2` and `step3` were both fine. On `step4` now.
  • Gen3 Pipeline for DP0.2 (backport-v23 label)
    • LAM: would like eyes on  DM-32128 - Getting issue details... STATUS  and  DM-32129 - Getting issue details... STATUS
    • JC: nightly ci dispatching metrics
• AOB
  - DT showed pull (chi) plots for CModel in DC2 w26. Will show more plots on Wednesday team meeting Oct 20.

Action Items