This page attempts to capture at a high level the software and algorithm development necessary to implement both Level 2 Calibrated Exposure Processing, though it also covers most of the code that will be used in Level 1 Calibrated Exposure Processing; the Level 1 processing should be mostly a subset of the Level 2 processing, with some slight reordering of operations and an emphasis on speed rather than precision. As of this writing, this page is mostly focused on the gap between our code's current capabilities and the capabilities needed at first light, while the linked pages above are more concerned with fleshing out the algorithms in detail. The long term relationship between these pages is TBD.
The rough outline for the Level 2 processing is:
- Run ISR on each snap exposure (sensors or even amplifiers in parallel).
- Coadd two snap exposures into a single visit exposure (sensors in parallel).
- Model Background (sensors in parallel, at least at first).
- Detect (possibly Deblend) and Measure bright sources (sensors in parallel).
- Run Astrometric/Photometric Calibration (sensors in parallel, at least at first).
- Estimate PSF (all sensors together, at least during one iteration).
- Determine Aperture Corrections
- Detect, Deblend, and Measure faint sources (sensors in parallel)
- Apply Aperture Corrections to both bright and faint sources
It's likely we'll only need 2-3 iterations of (3), and we may not run all of (a-f) during each iteration. The algorithms run at each step may be completely different from the algorithms we run at other steps, and the first iteration may consist almost entirely of placeholder guesses rather than real algorithms.
We have a relatively complete pipeline for processing individual sensors in parallel, suitable for processing precursor data at a level comparable to the state-of-the-art. Significant refactoring needed, as well as significant algorithmic work in a few areas.
|Overall Processing Flow||Assumes sensors are completely independent; need visit-level processing with more sophisticated parallelization to support (at least) PSF Estimation, Crosstalk-Correction. Some circular dependencies between algorithms need to be resolved via higher-level iteration, and we still need to work out the details of that.|
|Snap Coaddition||Basic algorithms understood, prototype implemented. Not clear whether warping/PSF-matching will be necessary (and it won't be until commissioning data is available). Will need to be entirely rewritten.|
|ISR||Interface and basic algorithms understood, prototype implemented. Interface needs refactoring (probably straightforward). Need to reconsider relationship with defect and CR interpolation (currently outside ISR pipeline) with Snap Coaddition in mind. Extensive testing needed as better simulations, commissioning data become available.|
|PSF Estimation||Placeholder implementation present. Final implementation will require overhaul of processing flow and interfaces as well as extensive algorithmic research.|
|CR Detection/Removal||Basic implementation present, will need improvements to handle modern devices. May be unnecessary after Snap Coaddition. Could benefit from design audit/review.|
|Single-Frame Detection||Basic implementation present, optimal for point sources. Plans for extension to faint extended sources are moderately well-understood, needs implementation and testing. Could benefit from design audit/review.|
|Single-Frame Deblending||Placeholder implementation present. Needs complete rewrite and implementation of additional well-understood extensions from SDSS.|
|Single-Frame Measurement||Basic framework nearing completion. Individual algorithms will require minor improvements, and minor framework improvements are likely necessary. Circular dependencies with PSF Estimation and other calibration processes need to be resolved via higher-level iteration.|
|Single-Frame Astrometric Calibration||Placeholder implementation present, replacement implementation underway. Should consider a multi-sensor implementation rather than a per-sensor implementation, ultimately; potentially more robust and accurate, and we'll have to do multi-sensor processing for other algorithms anyway (but it may not be important, given that this is not the final astrometric solution).|
|Single-Frame Photometric Calibration||Placeholder implementation present. Algorithms are simple and well-understood, but code needs complete rewrite. Might want to consider a multi-sensor implementation, or extensions to more than a single zeropoint (but may not be important, given that this is not the final photometric solution).|
|Single-Frame Background Modeling||Placeholder implementation present. Algorithms are simple and well-understood, but code needs complete rewrite. Might want to consider a multi-sensor implementation, or extensions to more than a single zeropoint (but may not be important, given that this is not the final background model; that will come from background-matching in the Deep Processing).|
|Single-Frame Aperture Correction||Implementation present on HSC fork, needs cleanup and minor refactoring in port to LSST codebase and the new measurement framework. Will need extensive testing.|
This is about visit processing within Data Release Production, not Alert Production, right?
Yes, that was my original intent. But as I just discussed with K-T, I still need to figure out how this page should relate to the existing Level 1 and Level 2 Calibrated Exposure Processing pages (so far, this is about work that needs to be done, and those are more about what open questions remain), and I think I might modify this page to try to include both, since most of the code will be shared between the two productions, and that means the same will be true of the coding.