DRP Metrics Monitoring 2020-12-07

Mondays 12pm - 12:50pm EST

Yusra's Zoom: https://princeton.zoom.us/my/yusra

Attendees:

Yusra, Keith, Arun, Hsin-Fang, Dan,  Matthias, Lee, Eric, Jeff, Josh, Lauren, Merlin, Morgan, Nate, Chris, Eli, Fred, Jim, Andres, Robert Lupton, Tim

Agenda:

• Review Action items from last week
• Review v21_0_0_rc1 (which is w_2020_46 with the 1-2% ccd psfex failure fixed) 
  • What was noteworthy in this rerun? 
    • Ran smoothly. ~6days
    • RCfakes rerun went smoothly too
  • What changed? Annotate.
    • Eli Rykoff  rerun fgcm. Investigate logs and internal output files to understand why fgcm solution changed from w42-v21.0. 
    • Unknown User (emorganson) Make sure w_2020_50 FGCM logs can be saved 
• What do we expect from w_2020_50? 
  • AssembleCcd adjusting raw WCS (not for the better)
  • ?? % of ccds fail.  
  • Because rare and not new, not a release blocker.
  • Expect fix on DM-27868
  • Expect Gaia refcat on w50 
• What new metric can we expect next time
• How goes w_2020_48's DC2?
  • matchedVisitMetric's doesn't work.
  • Look into forming an gen2-gen3 converted repo science pipelines can use for development (like RC2)
    • Debug why matchedVisitMetrics doesn't work on DC2 Yusra AlSayyad
• AOB
• plots showing differences from last (w42?) run (explained by psfex fix?)
• https://lsst.ncsa.illinois.edu/~emorgan2/v21_0_0_rc1_qaplots/HSC-I/tract-9813/visit-1228/compareVisit-v1228-diff_base_CircularApertureFlux_12_0-sky-gals.png
• https://lsst.ncsa.illinois.edu/~emorgan2/v21_0_0_rc1_qaplots/HSC-I/tract-9813/compare-t9813-HSC-I-diff_base_CircularApertureFlux_12_0-sky-gals.png
• https://lsst.ncsa.illinois.edu/~emorgan2/v21_0_0_rc1_qaplots/HSC-I/tract-9813/compare-t9813-HSC-I-psfTrace-psfMagHist.png
• https://lsst.ncsa.illinois.edu/~emorgan2/v21_0_0_rc1_qaplots/HSC-I/tract-9813/compare-t9813-HSC-I-trace-sky-stars.png

Minutes

Action items from last time

• Status of rho statistics. Work during pair coding, and probably get back to this on Wednesday. There is a notebook to demonstrate the analysis. Coadd PSF size residual (measured size - PSF model) shows some magnitude dependence at bright end. Could this be due to contamination / misclassification. Make sure not including objects that failed to get a classification. Watch out for NaNs in selection logic. Idea to compute rho_0 in different magnitude bins. Are we sure it is not just brighter fatter?
• Add a plot with fakes to dashboard. Sophie has been working on other things.
• DC2 run. We'll cover this later

v21_0_0_rc1: ran as smoothly as anything else. Dashboard metrics look mostly consistent with previous re-runs.

• Is photometry the same? The numbers got a little worse. Lauren has seen small differences in photometry? What changed? Some CCDs were failing. This was a new bug that was introduced and then removed. PA1 is stochastic, which can make some subtle problems difficult to notice/diagnose. Lauren has linked diagnostic plots. Matching catalogs between runs and measuring difference on the level of 10-20 mmag in the hotspots (spatially coherent blobs of residuals across a few CCDs). FGCM was applied in both. What has forced FGCM into a new solution? There was a brief period not reflected in these re-runs where 1% of calexps got removed. Checking the change log. Any deviations from zero on these plots is an indication that something changed. Do we have a source of truth? Some stars are held back in FGCM analysis. One place to start would be FGCM logs and internal output files. Where do these logs and internal plots go? Eric can make sure to save plots in the future. Another plot is the PSF realization. If we knew which stars were held back, we would look at the spatial structure of the residuals of the held back stars.
• Arun to change backend to database to compute PSF size residuals. Change sandbox to prod.

Lee has added Sky Object Metrics. First point on Choronagraph. Mean is negative. Visit levels are pre final sky correction that happens for coadd. Are there thresholds that we want to add?  No particular physically motivated number for threshold. It should become apparent over time what numbers of expected. Numbers are NOT changing when selecting different tracts and bands – is that a problem? Would have expected these numbers to change. Rho statistics show same patterns of NOT changing for different bands and tracts.

What do we expect from weekly 50? Gets tagged Wednesday night. Lauren discovered that we are erroneously adjusting WCS centering of search radius. Ticket to turn off erroneous adjustment. Fairly significant. Tested by not wide scale. JointCal did recover. This was causing whole CCD being lost. Probably has been in existence for years. About a year ago we changed to using boresight as initial estimate. Lauren has checked on DECam, ImSim, ..., and this effect is present for those other datasets too. It is rare, for example, did not occur in RC2. If this rare, and has been present for a while, it is not a release blocker. Release blockers are more for regressions.

Another change in using Gaia refcats by default. Yusra will post results from MatchedCatalog metrics in hsc-reprocessing. This would be using Gaia DR2.

Do we expect any more metrics? (no answer)

How is latest DC2 run going? Normal running went fine. Adding MatchedVisitMetrics didn't work. Is MatchedVisitMetrics stand alone, or does require something to be run first. For DC2, are we doing conversion to Gen3. Not being done currently, but Eric has scripts to do it.

There had been problems with previous RC2 fakes. Eric and Sophie figured it out. Hacky automatic symlink is current solution. Line in bash shell file. This is not a bug, just how to do it in Gen2. RC2 fakes need to know that FGCM results exist. Fakes re-run went smoothly once this was figured out.

Is v21 last good run? Yes.

Probably won't meet until January. Probably January 4.

Action Items