DRP Metrics Monitoring 2021-03-29

Mondays 12pm - 12:50pm EST

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

• Yusra AlSayyad, Andrés, Arun, Chris (Waters), Colin, Dan, Eli, Eric, Fred, Hsin-Fang, Ian, Jim, Josh, Keith, K-T, Lauren, Leanne, Lee, Monika, Morgan, Nate, Robert, Tim (Morton), Simon, Sophie

Agenda:

• Review Action items from last week
  
• Review w_2021_10 
  • What was noteworthy in this rerun?
    • Gen2
      • Continues to be a challenge. Few fixes need on FGCM. Ticket open for fakes reprocessing blocking the last two attempts. Last attempts haven't succeeded. Multiband not finding data. 
      • DC2 unblocked. but still going. 
    • Gen3
      • Started on Friday. ConsolidatedSourceTable and ? didn't go through bps. Next thing on to FGCM. Problem with test run, with week thirteen. 
      • Plan: Use w13 as a test for running FGCM against w10, but then proceed with w14. 
  • What changed? Annotate
    • Morgan SchmitzTake a look at whether TE1 increase is a problem. Take a look at the refcat matching plot. 
• What do we expect from w_2021_14? 
  • New features
    • Change in BF corrections (nans along the focal plane edges)
  • Changes toward Gen2-Gen3 parity:
    • DRP Gen2 to Gen3 Pipeline Migration Coordination
  • What new metric can we expect next time
• Review DC2
  • Last successful rerun was w_2021_04.
  • MatchedVisitMetrics only present for u,g,z,y. All visits. What does the tract=3828^3829 do? 
    • Hacked validate.py to ignore instrument: u_report.txt,  g_report.txt, z_report.txt,   y_report.txt
    • Do we need metrics for a DC2 instrument? https://github.com/lsst/verify_metrics/tree/master/specs/validate_drp
  • Gen3 Pipeline for DP0.2
• AOB
  • Jeffrey Carlin & I would like to discuss RC3 selection.  Please see Proposal at the end of RC3 dataset wish list, considerations
  • Regression in Faro runtime. Need help isolating issue. Can someone run it long-log? 

Updated scarlet sky source selection: see  DM-29087 - Getting issue details... STATUS

•  looks much better, but still a positive bias for aperture fluxes

Small change for next weekly related to brighter-fatter correction: see  DM-29280 - Getting issue details... STATUS

Really hoping to get this in for gen2/gen3 parity up to & including SFP (awaiting review):  DM-28936 - Getting issue details... STATUS

Matching to refer catalogs now being done to Gaia for astrometry, e.g. https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_10_qaplots/HSC-I/tract-9813/visit-1228/plot-v1228-matches_raCosDec-psfMagHist.png

Still seeing photometric differences at visit level between reruns...expected? (Eli?!), e.g. https://lsst.ncsa.illinois.edu/~emorgan2/w_2021_10_qaplots/HSC-I/tract-9813/visit-1228/compareVisit-v1228-diff_base_CircularApertureFlux_12_0-sky-gals.png

Action Items