Skip to end of metadata
Go to start of metadata
Target releaseSpring 2017
Epic

DM-8290 - Getting issue details... STATUS

Document status
DRAFT
Document owner


Goals 

  •  Remove artifacts from coadds by Identifying and masking/Interpolating artifacts in single epoch matchedWarps (CoaddTempExps). Replacement for sigma clipped coadds or "safe-clipped" coadds. 

Background and strategic fit

LDM-151 outlines algorithms for artifact detection at various stages of processing in both AP and DRP: snap combination, single epoch processing, and coaddition. During snap subtraction, we will remove cosmic rays and satellite trails (synonymous with “linear features” in  LDM-151).   During single epoch we will model optical ghosts (ISR using dithered star fields) and identify and remove satellite trails.  During coaddition, we will use the time series of warped images to identify artifacts. Most artifacts (see below) will only appear in one of a series of observations. Therefore a stack of warped, PSF-homogenized, background-matched images provide sufficient information to identify and mask pixels affected by artifacts and produce an artifact-free coadd. This epic will address artifact removal in the coaddition stage, by identifying them masking them in the single epoch psfMatchedWarps.

In the past we've used basic sigma-clipping to remove outliers (which leave faint wings of artifacts in the coadds), and "safe-clipped" coadds. Safe-clipped coadds difference each warp with the a sigma-clipped coadd, and mask the pixels with large deviations, and coadd again. The main drawback is that the comparison is done with directWarps rather than psfMatchedWarps. Seeing variations can masquerade as artifacts.  It also doesn't handle broad but faint artifacts like faint ghosts, and pixels that have different artifacts in different observations.

Other projects use variations of the safe-clipped coadds. 

Algorithms used by other multi-epoch imaging projects and have published papers:

Dark Energy Survey “CosmoDM” (Desai et al. 2016) removes artifacts from calexps by comparing to a catalog-generated model image (derived from measurement on a median-combined psf-matcched coadd). The algorithm consists of the following steps:

  1. Make Psf-Matched (to Moffat Psf of median FWHM) Median Chi^2(i+z) Coadd

  2. Run detection/measurement on Coadd.

  3.  For each calexp:

    1. Produce a simulated image—via Skymaker (Bertin 2009)—of same projection/psf based on the catalogs built from measurement on the single epoch image. Exclude faintest sources.

    2.  Compare calexp to simulated calexp, by making a “deviation image:” sqrt(wieght)*(calexp - simulated_calexp)

    3.  Flag pixels with residuals above 5\sigma. (unflag pixels in vicinity of areas where coadd and coadd model image do not match).

    4. Pass through again and flag pixels adjacent to flagged pixels which deviate more than than 2.5\sigma, and again for all pixels adjacent to those which deviate more than 1.5\sigma.

Notes: They acknowledge sensitivity the background model, and failure around saturated stars--which does not surprise me because all of our algorithms fail around saturated stars too.


SWarp w/ PSFHomTest and MaskMap (Gruen et al. 2014):

  1. Make sigma-clipped coadd with low threshold. {| f_i - \mu | > \bar{n} \sigma_i + A |\mu|} where \bar{n} and A are configs to control clipping and \mu = mean of all epochs for a given pixel.
  2. Using outlier maps, make masks. For squares of 3 X 3 (10X10; 50X50) pixels mask whole square if 1 (7; 35) pixels are > 4.5 (2.5; 2.5) \sigma + A\mu. 

Assumptions

1) Artifacts can be categorized as either an:

  • Imaged Artifact (Artifact larger than a PSF) vs a Non-Imaged Artifact  (Artifact smaller than a PSF. Artifact has not been imaged by the camera, and is produced by CCD electronics).
  • Single Epoch Artifact (appears in only one observation) vs a Persistent Artifact (appears in many observations)

Concrete examples: (key: + mean always positive; +/- could be positive or negative;  - always negative w.r.t. the "static sky")

Single Epoch Artifacts:

  • Imaged > PSF)
    • Satellite trails, meteors +
    • Glints +
    • Ghosts of bright stars: will be consistent for a given pointing a rotator angle (i.e persistent in snaps but not in Warps) +
    • Astrophysical Transients (supernovae, asteroids). Though not an artifact, they are not part of the "static sky" +
  • Non-imaged (CCD) (< PSF)
    • Cosmic Rays +
    • CCD defects (e.g. bad column) +/-

Persistant Artifacts:

  • Imaged 
    • Diffraction Spikes +
    • Astrophysical variable sources (variable stars, quasars) +/-

  • Non-imaged (CCD) (< PSF)
    • Cores of saturated stars (always negative?)

    • Bleed trails  +

2)  Need to use psfMatchedWarps because extreme seeing variations can appear as artifacts: e.g. wings of a star on the worst seeing night

3)  We want to mask both non-imaged artifacts and imaged artifacts, (Can’t make assumptions about what is behind an imaged artifact. its not necessarily sky.) as opposed to interpolating over the non-imaged ones. In normal single-epoch processing we interpolate over non-imaged artifacts like cosmic rays which are smaller than a PSF.   However, a psfMatchedWarp has been convolved with a kernel on order of the size of the PSF, thus broadening even non-imaged artifacts to the size of a PSF. 

4)  A masked footprint for an artifact, should be slightly larger than the identified outlier pixels (e.g. (Desai et al. 2016) checks surrounding pixels with lower and lower threshold sigmas). For an example why, recall outlines of clipped satellite trails in the sigma-clipped S13 Stripe 82 coadds. 

5) Technical definition of single-epoch artifact in time axis: Pixel is not in a detection in a robust coadd OR is detected in <10%** of observations. **Need to define in terms of probability of coincident artifacts and expected variability of real objects. 

6) Technical definition of Persistent artifact in time axis: Pixel is in detection in a robust coadd OR is detected in >10%** of observations. 


Requirements

#DescriptionImportanceNotes
1Identify artifacts Per Patch
  • Unit of processing with be a patch. Tract-sized descriptions of artifacts are out of scope
2Remove single-epoch artifacts from direct and PSF-Matched Coadds

3Persistent Artifacts

Four types of persistent "artifacts" have detection bit set in at least half

  • Variable Stars/Quasars will have mean flux in coadd => should not be masked in warps
  • Diffraction Spikes: typically most epochs will be masked, leaving area noisy.
  • Bleed trails: Currently interpolation in processCcd does terrible job, and the true static sky will be impossible to ascertain.
  • Cores of saturated stars: I don't know
4Mask single-epoch artifacts that coincidentally hit same pixels in multiple observations
  • Example from Bob Armstrong: It is common for satellite trails to cross each other in multiple observations

Questions

Below is a list of questions to be addressed as a result of this requirements document:

QuestionOutcome
LDM-151 says “we assume that everything we can detect morphologically will have already been detected.”  Does this mean that we will not have to detect CRs in CoaddTempExps?



Where will these masks be stored?

Currently, the only task which deals with the full stack of single epoch warps is AssembleCoaddTask. It reads in the CoaddTempExps/Warps and writes out a Coadd. It will need to write out more, specifically depth maps, which can also include the masks per visit. We need to think about the data structure for this. I'm imagining a data cube with mask-bit per visit, but I think Nate Lust's new Span Sets might have this capability too.


What are the failures of Safe-Clipped Coadds?

Per Bob Armstrong:

  • Uses directWarps (all stars look like artifacts).
  • Identifies literally single epoch artifacts (does not mask crossed satellite trails)
What behavior do we want for the cores of saturated stars?

Not Doing

  • Images with guiding errors will not provide any useful information to the coadd and should be discarded.