Overview

This page attempts to capture at a high level the software and algorithm development necessary to implement both Level 2 Calibrated Exposure Processing, though it also covers most of the code that will be used in Level 1 Calibrated Exposure Processing; the Level 1 processing should be mostly a subset of the Level 2 processing, with some slight reordering of operations and an emphasis on speed rather than precision.  As of this writing, this page is mostly focused on the gap between our code's current capabilities and the capabilities needed at first light, while the linked pages above are more concerned with fleshing out the algorithms in detail.  The long term relationship between these pages is TBD.

The rough outline for the Level 2 processing is:

  1. Run ISR on each snap exposure (sensors or even amplifiers in parallel).
  2. Coadd two snap exposures into a single visit exposure (sensors in parallel).
  3. Iterate:
    1. Model Background (sensors in parallel, at least at first).
    2. Detect (possibly Deblend) and Measure bright sources (sensors in parallel).
    3. Run Astrometric/Photometric Calibration (sensors in parallel, at least at first).
    4. Estimate PSF (all sensors together, at least during one iteration).
  4. Determine Aperture Corrections
  5. Detect, Deblend, and Measure faint sources (sensors in parallel)
  6. Apply Aperture Corrections to both bright and faint sources

It's likely we'll only need 2-3 iterations of (3), and we may not run all of (a-f) during each iteration.  The algorithms run at each step may be completely different from the algorithms we run at other steps, and the first iteration may consist almost entirely of placeholder guesses rather than real algorithms.

Current Status

We have a relatively complete pipeline for processing individual sensors in parallel, suitable for processing precursor data at a level comparable to the state-of-the-art.  Significant refactoring needed, as well as significant algorithmic work in a few areas.

ComponentStatus
Overall Processing FlowAssumes sensors are completely independent; need visit-level processing with more sophisticated parallelization to support (at least) PSF Estimation, Crosstalk-Correction. Some circular dependencies between algorithms need to be resolved via higher-level iteration, and we still need to work out the details of that.
Snap CoadditionBasic algorithms understood, prototype implemented. Not clear whether warping/PSF-matching will be necessary (and it won't be until commissioning data is available). Will need to be entirely rewritten.
ISRInterface and basic algorithms understood, prototype implemented. Interface needs refactoring (probably straightforward). Need to reconsider relationship with defect and CR interpolation (currently outside ISR pipeline) with Snap Coaddition in mind. Extensive testing needed as better simulations, commissioning data become available.
PSF EstimationPlaceholder implementation present. Final implementation will require overhaul of processing flow and interfaces as well as extensive algorithmic research.
CR Detection/RemovalBasic implementation present, will need improvements to handle modern devices. May be unnecessary after Snap Coaddition. Could benefit from design audit/review.
Single-Frame DetectionBasic implementation present, optimal for point sources. Plans for extension to faint extended sources are moderately well-understood, needs implementation and testing. Could benefit from design audit/review.
Single-Frame DeblendingPlaceholder implementation present. Needs complete rewrite and implementation of additional well-understood extensions from SDSS.
Single-Frame MeasurementBasic framework nearing completion. Individual algorithms will require minor improvements, and minor framework improvements are likely necessary. Circular dependencies with PSF Estimation and other calibration processes need to be resolved via higher-level iteration.
Single-Frame Astrometric CalibrationPlaceholder implementation present, replacement implementation underway. Should consider a multi-sensor implementation rather than a per-sensor implementation, ultimately; potentially more robust and accurate, and we'll have to do multi-sensor processing for other algorithms anyway (but it may not be important, given that this is not the final astrometric solution).
Single-Frame Photometric CalibrationPlaceholder implementation present. Algorithms are simple and well-understood, but code needs complete rewrite. Might want to consider a multi-sensor implementation, or extensions to more than a single zeropoint (but may not be important, given that this is not the final photometric solution).
Single-Frame Background ModelingPlaceholder implementation present. Algorithms are simple and well-understood, but code needs complete rewrite. Might want to consider a multi-sensor implementation, or extensions to more than a single zeropoint (but may not be important, given that this is not the final background model; that will come from background-matching in the Deep Processing).
Single-Frame Aperture CorrectionImplementation present on HSC fork, needs cleanup and minor refactoring in port to LSST codebase and the new measurement framework. Will need extensive testing.