Metric Name  One line summary  Current Status  Discussion  Next Step 

PA1gri/uzy  Photometric Repeatability  Implemented in faro 10mmag including shot noise leaves no error budget  Options:  "Liberally interpret" the requirement to subtract off shot noise
 Pick high enough SNR limit to make shot noise <<10mmag
 Ask for nonconformance (or LCR) to accept a modified definition of the requirement.
DRP proposes keeping nonstatisticaluncertaintysubtracted version, useful for comparisons.  Colin  SRD clearly states that shot noise is not part of the intended measurement.
 Should quantitatively confirm whether bright stars are insufficient for this measurement before adopting a shotnoisesubtracted version.
 Would not propose LCR. bright stars (even if limited) + shot noise subtracted version should be sufficient to demonstrate compliance with existing text.
 Choose option 2: Bump SNR threshold to 200 so that this becomes plausible. Configuration change.
 Explain change to ppls and why the change is made. Check if they want to keep the existing configuration.
 Setting the random seed will make it easier to recognize regressions

 PA1 with statistical uncertainties subtracted off  proposed  For each object, the observed RMS of flux measurements about the unknown "truth" flux includes statistical and systematic contributions. We want to isolate the systematic component by subtracting off the expected (shotnoise) contribution. One remaining question: is this better calculated on PSF or aperture fluxes? (See discussion in slide 32.)  Jeff Implement in faro. Move to new new metrics page. Come up with new naming convention, e.g PhotRepMetric01 and note in the documentation its relation to the official PA1 metric 
PF1, PA2gri, PA2uzy  Photometric Repeatability outlier limit  Implemented in faro but the implementation is not consistent with the SRD intention  We need a new algorithm here to compute residuals relative to the mean, rather than pairs of differences.  Select stars with high SNR, e.g., SNR > 200. Higher SNR is needed for this specific metric to prevent statistical shot noise from dominating the outlier fraction.
 For each object, compute the pervisit flux residuals relative to the mean of all flux measurements for that object. This will generate a distribution of pervisit flux residuals generated from a set of many objects and many visits.
 For metrics that are characterizing an outlier fraction, hold the threshold fixed and compute the fraction of objects beyond the threshold value. In other words, the thresholds are definition only, not intended to be a separate metric. In this case, we only calculate PF1 and PA2 is used as the threshold.
 Keith  Already implemented in faro, but needs to be revised.
 Implement new algorithm in faro.
 For metrics that are characterizing an outlier fraction, hold the threshold fixed and compute the fraction of objects beyond the threshold value. In other words, the thresholds are definition only, not intended to be a separate metric. In this case, we only calculate PF1 and PA2 is used as the threshold. Clarify in JIRA LVV.
 Explain change due to new algorithm on CMR and to pplns.

AF1, AF2, AF3 and AD1, AD2, AD3  AF1/2/3/ : the maximum fraction of relative astrometric measurements on the 5/20/200 arcminute timescale to exceed the AD1/2/3 (5/20/200 arcminute) outlier limit.  Implemented in faro   AD1/2/3 are the 5/20/200 arcminute fixed parameter thresholds for the outlier limits associated with AF1/2/3. They are not KPMs in themselves.
 AF1/2/3 : residuals are computed w.r.t the median separation for each source; as the goal is to understand the behaviour of outliers, computation w.r.t the mean should be looked at.
 [Leanne]  Keep current implementation
 Stop reporting the computed threshold at which the percentile is reached (ADx), we apply only the fixed specified value in computing AFx.
 Clarify in JiraLVV that the ADx are fixed parameter thresholds and report only AF1/2/3 as KPMs.
 Any LCR to separate thresholds into a separate table needs to be applied to the OSS/LSR as well. (TBD)

AB1  Bandtoband astrometric registration  Implemented, could use revision.   Review considered several other possible interpretations of the requirement text, but concluded that the implemented metric is the correct interpretation.
 Review proposed measuring separate RA, Dec RMS values rather than RMS of a radius measurement (which isn't centered around zero)
 Colin  Add perfilter metrics – report AB1 separately for each filter, all filters must pass independently
 Add separate RA, Dec RMS, investigate differences.
 Decide later what to report as official AB1.

TE1, TE2  Currently these residual ellipticity correlations are being calculated using treecorr on matched visits. We will keep the general approach (with small modifications), but measure on coadds (i.e., use the object catalog).  Implemented in faro, but needs revision.  Concrete actions identified in review:  calculate on coadds rather than matched catalogs,
 currently it is taking the absolute value of the average correlation; modify to compute the median of the absolute values,
 decide which convention we will adopt for ellipticity calculation (shear vs. distortion) and document our choice.
If we would consider a refactor, Pipe Analysis has a nice general implementation that can be used for multiple rho statistics.  Jeff Update the faro implementation to meet the three conditions outlined in the discussion.  Report distortion as TE
 Adopt naming roweX as convention for the shear convention
Longer term, we should refactor to be able to compute all rho statistics. 
PA3, PA3u  Photometric zeropoint  Not implemented  The proposed algorithm is equivalent to analyses done as part of characterization of photometric uniformity in DES and HSC (both using FGCM calibration procedure). There is a difference between predicting Gaia G versus using Gaia to predict fluxes in LSST passbands.  Keith  Follow up with Eli to understand what he has done
 Implement independently in faro to enable independent verification of Eli's calculation with faro.
 u and y require more thought. Followup with Douglas Tucker.
 We will need to compute and persist color terms to transform between external reference catalogs and the LSST passbands. This requires some thought.

PF2, PA4  PA4: Zero point outlier limit PF2: fraction of zero point errors that can exceed the zero point outlier limit.  Not implemented   PA4 as for AD1/2/3 is a fixed parameter threshold and should not be treated as a metrics
 Comment that he value of PA4 appears to be erroneous and not achievable.
 Leanne:  First investigate the PA4 value together with PA3. Are these values achievable? If not, propose new submit LCR to change.
 PA3 should be implemented first. PF2 derives from the PA3 calculation.
 As for AD1/2/3, PA4 is not a metric and should not be computed or reported

PA6  Accuracy of physical fluxes (AB system)  Not implemented   Calibration Scientist believed this was not in scope.
 Design goal is 10mmag, but existing best calibration efforts (CalSpec) may only be accurate to ~10mmag. Should requirement be on transformation to CalSpec?
 Is 10mmag reasonable? Was there a justification that this was possible?
 Minimum spec is 20mmag.
 Colin  "Easier" option: say this means <10mmag error relative to a stateoftheart physical calibration source, which would be CalSpec.
 "Harder" option: sum CalSpec's reported uncertainty with the uncertainty of our tie to CalSpec.
 Low priority, do not need to decide yet

PA5, PA5u  Accuracy of absolute bandtoband color zeropoint  Not implemented  This is tied to PA6  discuss those together.  Jeff
 Requires a separate study to understand how to proceed
 Punt together with PA6

AA1 
 Not implemented  This metric is clearly defined. Use reserve stars (not used for calibration) and Gaia as the external reference.  Additional takeaway is the need to have a set of reserve stars.
 Keith Implement in faro. High priority  good use case to interface faro to an external reference dataset 
AB2, ABF1  AB2 : the color difference outlier limit for separations measured relative the rband filter in any other filter. ABF1 is the fraction of separations measured relative to the rband that can exceed the outlier limit AB2  Not implemented   Related to AB1
 AB2 is a fixed parameter threshold, not a KPM in itself
 [Leanne]  As for AD1/2/3, PA4, AB2 is a fixed parameter threshold, stop computing/reporting. Defined value is used in computation of ABF1
 Implement as described

TE3, TE4, TEF  Residual ellipticity correlations for individual visits  Not Implemented   Same as TE1 and TE2, but for single visit vs. full survey. Can use same algorithm.
 Will require a summary step to compute fraction of visits in compliance.
 Colin  Implement after finishing TE1, TE2
 TE3 and TE4 are fixed thresholds. In practice, we plan to compute the correlation function for each visit and then compute the fraction of visits in compliance (TEF) with the thresholds in a summary step.

pixFrac, sensorFraction  Fraction of "scientifically unusable" pixels per sensor, and the maximum number of sensors exceeding the threshold fraction.  Not implemented  This is relatively straightforward to implement. Recommendation is to measure defects, etc. per visit, and include pixels from ghosts, glints, etc. that are masked in coaddition. Astrophysical contribution to the number of unusable pixels will vary with sky position – choose a nominal field location (or Galactic latitude) to measure this, but regularly check its behavior at other positions.  Jeff  Proposal seems reasonable and can be implemented in faro  implement as described and look at results
 Further clarification of the goal needed.
This is not in the SRD so we could descope if this is not considered useful and superseded by other more useful metrics 
xTalk 
 Not implemented  This analysis involves forced photometry and some careful coordinate transformations. While this analysis should be automated, it does not need to be run on every visit. Checking on a few visits periodically should sufficient. It would be interesting to compare results on different detectors / amplifiers. Generally viewed that other crosstalk metrics  Keith Suggest to implement this analysis first in a standalone script / notebook to get some intuition. Would be useful to check with a camera/ISR expert to see what has already been done. Low priority. This is not in the SRD so we could descope if this is not considered useful, especially if they are superseded by other more useful metrics Needs onsky data to have realistic flux distribution of stars 
ResSource  Maximum RMS of the ratio of the error in integrated flux measurement between bright, isolated, resolved sources less than 10 arcsec in diameter and bright, isolated unresolved point sources  Not implemented   Leanne  Clarify the intent of this metric with Zeljko first
Note that this is not explicitly a called out in the SRD but is discussed in 3.3.4.5 Further notes on photometry. Can we descope this or clarify requirement flowed down from SRD discussion Do validation with artificial source injection?

SBPrec  Maximum error in the precision of the sky brightness determination. 
  Problem: We have attempted to infer the mathematical operations needed to compute this metric based in discussion in LSE40, and we need to confirm the intent. It seems that this requirement is getting at the distribution of counts in pixels for individual visits and our ability to model the background.
 There are additional related metrics using sky sources that might be more useful and have already been implemented
 Keith Discuss intent of this requirement with Zeljko

GhostAF  Fraction of image area with high gradient ghosts  Not implemented  This is poorly defined/needs clarification. "We can compute what fraction of pixels were clipped, but knowing that a pixel is clipped is not the same as knowing why the pixel was clipped."  Jeff Discuss with DRP team and decide on interpretation/implementation. Does this happen before or after DM attempts to correct for ghosts. 