Coadd Artifact Removal via Warped Image Comparison (DM-8290)

Goals 

•  Remove artifacts from coadds by Identifying and masking/Interpolating artifacts in single epoch matchedWarps (CoaddTempExps). Replacement for sigma clipped coadds or "safe-clipped" coadds. 

Background and strategic fit

LDM-151 outlines algorithms for artifact detection at various stages of processing in both AP and DRP: snap combination, single epoch processing, and coaddition. During snap subtraction, we will remove cosmic rays and satellite trails (synonymous with “linear features” in  LDM-151).   During single epoch we will model optical ghosts (ISR using dithered star fields) and identify and remove satellite trails.  During coaddition, we will use the time series of warped images to identify artifacts. Most artifacts (see below) will only appear in one of a series of observations. Therefore a stack of warped, PSF-homogenized, background-matched images provide sufficient information to identify and mask pixels affected by artifacts and produce an artifact-free coadd. This epic will address artifact removal in the coaddition stage, by identifying them masking them in the single epoch psfMatchedWarps.

In the past we've used basic sigma-clipping to remove outliers (which leave faint wings of artifacts in the coadds), and "safe-clipped" coadds. Safe-clipped coadds difference each warp with the a sigma-clipped coadd, and mask the pixels with large deviations, and coadd again. The main drawback is that the comparison is done with directWarps rather than psfMatchedWarps. Seeing variations can masquerade as artifacts.  It also doesn't handle broad but faint artifacts like faint ghosts, and pixels that have different artifacts in different observations.

Other projects use variations of the safe-clipped coadds. 

Algorithms used by other multi-epoch imaging projects and have published papers:

Dark Energy Survey “CosmoDM” (Desai et al. 2016),  removes artifacts from calexps by comparing to a catalog-generated model image (derived from measurement on a median-combined psf-matcched coadd). The algorithm consists of the following steps:

1. Make Psf-Matched (to Moffat Psf of median FWHM) Median Chi^2(i+z) Coadd

2. Run detection/measurement on Coadd.

3.  For each calexp:

   1. Produce a simulated image—via Skymaker (Bertin 2009)—of same projection/psf based on the catalogs built from measurement on the single epoch image. Exclude faintest sources.

   2.  Compare calexp to simulated calexp, by making a “deviation image:” sqrt(wieght)*(calexp - simulated_calexp)

   3.  Flag pixels with residuals above 5\sigma. (unflag pixels in vicinity of areas where coadd and coadd model image do not match).

   4. Pass through again and flag pixels adjacent to flagged pixels which deviate more than than 2.5\sigma, and again for all pixels adjacent to those which deviate more than 1.5\sigma.

Notes: They acknowledge sensitivity the background model, and failure around saturated stars--which does not surprise me because all of our algorithms fail around saturated stars too.

SWarp w/ PSFHomTest and MaskMap (Gruen et al. 2014):

1. Make sigma-clipped coadd with low threshold.  $//<![CDATA[ \begin{array}{l}| f_i - \mu | > \bar{n} \sigma_i + A |\mu|\end{array} //]]>$ where \bar{n} and A are configs to control clipping and \mu = mean of all epochs for a given pixel.
2. Using outlier maps, make masks. For squares of 3 X 3 (10X10; 50X50) pixels mask whole square if 1 (7; 35) pixels are > 4.5 (2.5; 2.5) \sigma + A\mu. 

Assumptions

1) Artifacts can be categorized as either an:

• Imaged Artifact (Artifact larger than a PSF) vs a Non-Imaged Artifact  (Artifact smaller than a PSF. Artifact has not been imaged by the camera, and is produced by CCD electronics).
• Single Epoch Artifact (appears in only one observation) vs a Persistent Artifact (appears in many observations)

Concrete examples: (key: + mean always positive; +/- could be positive or negative;  - always negative w.r.t. the "static sky")

Single Epoch Artifacts:

• Imaged ( > PSF)
  
  • Satellite trails, meteors +
  • Glints +
  • Ghosts of bright stars: will be consistent for a given pointing a rotator angle (i.e persistent in snaps but not in Warps) +
  • Astrophysical Transients (supernovae, asteroids). Though not an artifact, they are not part of the "static sky" +
• Non-imaged (CCD) (< PSF)
  • Cosmic Rays +
  • CCD defects (e.g. bad column) +/-

Persistant Artifacts:

• Imaged 
  • Diffraction Spikes +

  • Astrophysical variable sources (variable stars, quasars) +/-

• Non-imaged (CCD) (< PSF)
  

  • Cores of saturated stars (always negative?)

  • Bleed trails  +

2)  Need to use psfMatchedWarps because extreme seeing variations can appear as artifacts: e.g. wings of a star on the worst seeing night

3)  We want to mask both non-imaged artifacts and imaged artifacts, (Can’t make assumptions about what is behind an imaged artifact. its not necessarily sky.) as opposed to interpolating over the non-imaged ones. In normal single-epoch processing we interpolate over non-imaged artifacts like cosmic rays which are smaller than a PSF.   However, a psfMatchedWarp has been convolved with a kernel on order of the size of the PSF, thus broadening even non-imaged artifacts to the size of a PSF. 

4)  A masked footprint for an artifact, should be slightly larger than the identified outlier pixels (e.g. (Desai et al. 2016) checks surrounding pixels with lower and lower threshold sigmas). For an example why, recall outlines of clipped satellite trails in the sigma-clipped S13 Stripe 82 coadds. 

5) Technical definition of single-epoch artifact in time axis: Pixel is not in a detection in a robust coadd OR is detected in <10%** of observations. **Need to define in terms of probability of coincident artifacts and expected variability of real objects. 

6) Technical definition of Persistent artifact in time axis: Pixel is in detection in a robust coadd OR is detected in >10%** of observations. 

Requirements

Questions

Below is a list of questions to be addressed as a result of this requirements document:

Not Doing

• Images with guiding errors will not provide any useful information to the coadd and should be discarded.