Performance Metrics - Plan of Action

Implemented in faro

10mmag including shot noise leaves no error budget

Options:

• "Liberally interpret" the requirement to subtract off shot noise
• Pick high enough SNR limit to make shot noise <<10mmag
• Ask for non-conformance (or LCR) to accept a modified definition of the requirement.

DRP proposes keeping non-statistical-uncertainty-subtracted version, useful for comparisons.

Colin

• SRD clearly states that shot noise is not part of the intended measurement.
• Should quantitatively confirm whether bright stars are insufficient for this measurement before adopting a shot-noise-subtracted version.
• Would not propose LCR. bright stars (even if limited) + shot noise subtracted version should be sufficient to demonstrate compliance with existing text.
• Choose option 2: Bump SNR threshold to 200 so that this becomes plausible. Configuration change. 
• Explain change to ppls and why the change is made. Check if they want to keep the existing configuration. 
• Setting the random seed will make it easier to recognize regressions

Faint-star Photometric repeatability, shot noise subtracted.

For each object, the observed RMS of flux measurements about the unknown "truth" flux includes statistical and systematic contributions. We want to isolate the systematic component by subtracting off the expected (shot-noise) contribution.

One remaining question: is this better calculated on PSF or aperture fluxes? (See discussion in slide 32.)

Jeff

Implement in faro. 

Move to new new metrics page. 

Come up with new naming convention, e.g PhotRepMetric-01 and note in the documentation its relation to the official PA1 metric

We need a new algorithm here to compute residuals relative to the mean, rather than pairs of differences.

• Select stars with high SNR, e.g., SNR > 200. Higher SNR is needed for this specific metric to prevent statistical shot noise from dominating the outlier fraction.
• For each object, compute the per-visit flux residuals relative to the mean of all flux measurements for that object. This will generate a distribution of per-visit flux residuals generated from a set of many objects and many visits.
• For metrics that are characterizing an outlier fraction, hold the threshold fixed and compute the fraction of objects beyond the threshold value. In other words, the thresholds are definition only, not intended to be a separate metric. In this case, we only calculate PF1 and PA2 is used as the threshold.

Keith

• Already implemented in faro, but needs to be revised.
• Implement new algorithm in faro.
• For metrics that are characterizing an outlier fraction, hold the threshold fixed and compute the fraction of objects beyond the threshold value. In other words, the thresholds are definition only, not intended to be a separate metric. In this case, we only calculate PF1 and PA2 is used as the threshold. Clarify in JIRA LVV.
  
• Explain change due to new algorithm on CMR and to pplns. 

AF1, AF2, AF3
and 

AD1, AD2, AD3

• AD1/2/3 are the 5/20/200 arcminute fixed parameter thresholds for the outlier limits associated with AF1/2/3. They are not KPMs in themselves. 
• AF1/2/3 : residuals are computed w.r.t the median separation for each source; as the goal is to understand the behaviour of outliers, computation w.r.t the mean should be looked at. 

[Leanne]

• Keep current implementation 
• Stop reporting the computed threshold at which the percentile is reached (ADx), we apply only the fixed specified value in computing AFx.
• Clarify in Jira-LVV that the ADx are fixed parameter thresholds and report only AF1/2/3 as KPMs.
• Any LCR to separate thresholds into a separate table  needs to be applied to the OSS/LSR as well. (TBD)

AB1

• Review considered several other possible interpretations of the requirement text, but concluded that the implemented metric is the correct interpretation.
• Review proposed measuring separate RA, Dec RMS values rather than RMS of a radius measurement (which isn't centered around zero)

Colin

• Add per-filter metrics – report AB1 separately for each filter, all filters must pass independently
• Add separate RA, Dec RMS, investigate differences. 
• Decide later what to report as official AB1.

Concrete actions identified in review:

1. calculate on coadds rather than matched catalogs,
2. currently it is taking the absolute value of the average correlation; modify to compute the median of the absolute values,
3. decide which convention we will adopt for ellipticity calculation (shear vs. distortion) and document our choice.

If we would consider a refactor, Pipe Analysis has a nice general implementation that can be used for multiple rho statistics.

Jeff

Update the faro implementation to meet the three conditions outlined in the discussion.

• Report distortion as TE
• Adopt naming rowe-X as convention for the shear convention

Longer term, we should refactor to be able to compute all rho statistics.

The proposed algorithm is equivalent to analyses done as part of characterization of photometric uniformity in DES and HSC (both using FGCM calibration procedure). There is a difference between predicting Gaia G versus using Gaia to predict fluxes in LSST passbands.

Keith

• Follow up with Eli to understand what he has done 
• Implement independently in faro to enable  independent verification of Eli's calculation with faro.
• u and y require more thought. Follow-up with Douglas Tucker.
• We will need to compute and persist color terms to transform between external reference catalogs and the LSST passbands. This requires some thought.

PA4: Zero point outlier limit

PF2: fraction of zero point errors that can exceed the zero point outlier limit. 

• PA4 as for AD1/2/3 is a fixed parameter threshold and should not be treated as a metrics
• Comment that he value of PA4 appears to be erroneous and not achievable. 

Leanne:

• First investigate the PA4 value together with PA3. Are these values achievable? If not, propose new submit LCR to change. 
• PA3 should be implemented first. PF2 derives from the PA3 calculation. 
• As for AD1/2/3, PA4 is not a metric and should not be computed or reported

• Calibration Scientist believed this was not in scope.
• Design goal is 10mmag, but existing best calibration efforts (CalSpec) may only be accurate to ~10mmag. Should requirement be on transformation to CalSpec?
• Is 10mmag reasonable? Was there a justification that this was possible?
• Minimum spec is 20mmag.

Colin

• "Easier" option: say this means <10mmag error relative to a state-of-the-art physical calibration source, which would be CalSpec.
• "Harder" option: sum CalSpec's reported uncertainty with the uncertainty of our tie to CalSpec.
• Low priority, do not need to decide yet

• Requires a separate study to understand how to proceed
• Punt together with PA6

This metric is clearly defined. Use reserve stars (not used for calibration) and Gaia as the external reference.

• Additional takeaway is the need to have a set of reserve stars.  

Keith

Implement in faro.

High priority - good use case to interface faro to an external reference dataset

AB2 : the color difference outlier limit for separations measured relative the r-band filter in any other filter.

ABF1 is the fraction of separations measured relative to the r-band that can exceed the outlier limit AB2 

• Related to AB1
• AB2 is a fixed parameter threshold, not a KPM in itself

[Leanne]

• As for AD1/2/3, PA4, AB2 is a fixed parameter threshold, stop computing/reporting. Defined value is used in computation of ABF1
• Implement as described 

• Same as TE1 and TE2, but for single visit vs. full survey. Can use same algorithm.
  
• Will require a summary step to compute fraction of visits in compliance.

Colin

• Implement after finishing TE1, TE2
• TE3 and TE4 are fixed thresholds. In practice, we plan to compute the correlation function for each visit and then compute the fraction of visits in compliance (TEF) with the thresholds in a summary step.

This is relatively straightforward to implement. Recommendation is to measure defects, etc. per visit, and include pixels from ghosts, glints, etc. that are masked in coaddition. 

Astrophysical contribution to the number of unusable pixels will vary with sky position – choose a nominal field location (or Galactic latitude) to measure this, but regularly check its behavior at other positions.

Jeff

• Proposal seems reasonable and can be implemented in faro - implement as described and look at results
• Further clarification of the goal needed. 

This is not in the SRD so we could descope if this is not considered useful and superseded by other more useful metrics

This analysis involves forced photometry and some careful coordinate transformations. While this analysis should be automated, it does not need to be run on every visit. Checking on a few visits periodically should sufficient. It would be interesting to compare results on different detectors / amplifiers.

Generally viewed that other crosstalk metrics

Keith

Suggest to implement this analysis first in a stand-alone script / notebook to get some intuition. Would be useful to check with a camera/ISR expert to see what has already been done.

Low priority. This is not in the SRD so we could descope if this is not considered useful, especially if they are superseded by other more useful metrics

Needs on-sky data to have realistic flux distribution of stars

• Specification is hard to parse. More information needed to understand the intent of the specification.

• Suggestion to add different metrics to show how galaxy photometry varies as a function of observing conditions, e.g seeing, and effects of blending

Leanne 

• Clarify the intent of this metric with Zeljko first

• Note that this is not explicitly a called out in the SRD but is discussed in 3.3.4.5 Further notes on photometry. 

• Can we descope this or clarify requirement flowed down from SRD discussion

• Do validation with artificial source injection?

• Problem: We have attempted to infer the mathematical operations needed to compute this metric based in discussion in LSE-40, and we need to confirm the intent. It seems that this requirement is getting at the distribution of counts in pixels for individual visits and our ability to model the background.
• There are additional related metrics using sky sources that might be more useful and have already been implemented

Keith
Discuss intent of this requirement with Zeljko

This is poorly defined/needs clarification.

"We can compute what fraction of pixels were clipped, but knowing that a pixel is clipped is not the same as knowing why the pixel was clipped."

Jeff

Discuss with DRP team and decide on interpretation/implementation.

Does this happen before or after DM attempts to correct for ghosts.