Skip to end of metadata
Go to start of metadata

This page is currently under development: text may change at any time! The content has not yet been reviewed for accuracy and completeness.

Data products are generated by the LSST Stack in a set of productions, or episodes of orchestrated data processing. The type of products, and the planned cadence on which they will be produced, reflect the survey strategy and scientific objectives of LSST. While the LSST Stack software is designed to be useful in other image processing contexts, understanding the survey data productions and products for LSST is useful for understanding how to apply the Stack for alternate purposes.

In this Chapter

LSST Data Productions

The LSST Stack is designed to produce official data products in a set of productions using dedicated, extremely high-throughput computing infrastructure. The types of products are intimately tied to the strategy and scientific objectives of the planned LSST survey. While the LSST Stack software may be used to process data in other contexts, and for other instruments, the Stack software organization and the nature of the data products is strongly influenced by LSST's concept of productions. The documents listed below provide an official, detailed description of the planned data products and how DMS will produce them.  

This chapter provides a high-level summary of the productions. Features that have been implemented for the current release of the LSST Stack are noted.

Inputs to the Productions

The following is a list of common inputs to one or more of the productions: 

Raw Exposures

The output from the camera is a set of image sections from each amplifier on each sensor in the focal plane array, including overscan

Reference Standard Catalogs

Catalogs of astrometric or photometric standards, containing coordinates, motions, and brightnesses of objects in multiple passbands, covering the survey footprint.

Calibration Reference Files

Outputs of the Calibration Products Production that are used for, e.g., instrument signature removal, to support other productions.

Image Templates

Co-added, single-band image of the sky that is deep, and where all transients, SSObjects, and artifacts have been removed. Constituent images for Image Templates may be selected from a limited range of quality parameters, such as PSF size or airmass.  

Nightly Productions

The main scientific point of productions on a diurnal cycle is to discover changes from a prior reference epoch: changes in position or brightness for known astrophysical objects, or detections of new, previously unknown sources. The production is split into two episodes: a low-latency Alert Production, and daily Moving Object Processing; in combination they generate Level-1 data products

Alert Production

The alert production (AP) is active during on-sky observing to process images obtained in each visit with low latency. Since the science goals require processing to complete within one minute of the last exposure in the visit, and the best calibration reference data will generally not be available at the time of the exposures, the processing will be of lower fidelity than for the Data Release Production

The AP has not yet been fully implemented in the Stack. However, various essential components (basic calibration, catalog generation, forced photometry) have been implemented as a part of the Data Release Production. Image differencing is in early prototype form, and the alert generation has not been implemented.

The AP operates on raw exposures obtained during a single visit, and uses the following input data (beyond that described above):

  • Catalog of DIASources, where entries may have associations to known variable DIAObjects or to known Solar System Objects (SSObjects)
  • Predicted positions of known SSObjects for the epoch of the exposure
  • Reference image templates appropriate for the region of sky, passband, and observing conditions of the exposure

The AP performs the following processing (see Alert Generation and Distribution for details) for each visit:

  1. Instrument signature removal, artifact flagging, CCD assembly, WCS and PSF characterization, and combination of the exposures. Outputs are: 
    • a calibrated visit image (persisted)
  2. Performs difference image processing with respect to an image template, including source detection and de-blending. Outputs are: 
    • A difference image (persisted)
    • A table of detected sources in the exposure, consisting of measured properties for each entry, including position and measured flux (not persisted)
  3. Associates detected sources with known DIAObjects, or SSObjects. Outputs: 
    • Updates to the DIASources catalog with: 
      • table of sources detected in the visit
      • successful spatial associations with DIAObjects, and with predicted positions of SSObjects
    • Issues alerts on DIASources
  4. Performs forced photometry (point-source only) at the locations of DIAObjects that: 
    1. overlap the visit image field of view, and
    2. for which there is no DIASource detection

The output is:

    • Updates to the DIASource catalog

MOPS

The role of the moving object processing software (MOPS) is to identify new SSObjects from unassociated DIASources

MOPS has not yet been implemented in the Stack, although an early prototype exists in the git repository.

MOPS processing includes the following:

  1. Attempts associations among unassociated DIASources by exploring the (rather large) parameter space of plausible orbits that are consistent with recent appearances (i.e., locations in space and time) of a single SSObject. Output:
    • Candidate SSObjects (not persisted)
  2. For each SSObject candidate, attempt precovery (i.e., recover past apparitions) by performing forced photometry at predicted locations in prior-epoch images, based on the derived orbit parameters. The output is: 
    • Updates to the SSObject catalog, including orbital elements flux measurements (persisted)
    • A table of tracklets, or pairs of time/location coordinates for unassociated DIASources taken at different epochs that are plausible tracks of SSObjects (not persisted)

Data Release Production

The data release production (DRP) is designed to enable static-sky science, and time-domain science that is not urgent. The DRP is designed to perform a mass reprocessing of all survey data obtained prior to some defined point in time, in multiple passes. These large-scale productions are planned annually. The DRP repeats most of the processing performed in the Nightly Productions, using the temporally best-mached calibration reference files and an updated and consistent set of algorithms.  DRP generates a variety of other products as well, but does not issue alerts

Much of the DRP has been prototyped in the LSST Stack. However, certain components (multi-object model fitting, selecting templates based on airmass or other criteria, fully correct aperture photometry) remain to be implemented.

The DRP operates on raw exposures obtained during a single visit, and uses the following input data (beyond those mentioned in the general Inputs to the Productions):

  • Catalog of DIASources, where entries may have associations to known variable DIAObjects or to known SSObjects
  • Predicted positions of known SSObjects for the epoch of the exposure
  • Reference image templates appropriate for the region of sky, passband, and observing conditions of the exposure
  • Updated calibration products, input catalogs, and templates that are temporally matched to the data 

The DRP includes following processing on the input raw images and processed images:

FIrst Pass: Single-Frame Processing

  1. As with AP, processing of individual frames consists of: instrument signature removal, artifact flagging, CCD assembly, WCS and PSF characterization, and combination of the exposures. Sources are independently detected, deblended, and measured. Outputs:
    • a calibrated visit image
    • Source table, including locations, instrumental fluxes, and shapes
  2. Relative calibration of the frames, where photometric and astrometric zero-points are determined from survey-level calibrations

Second Pass: Co-Add Creation & Measurement

  1. Co-add creation from single, overlapping frames consists of warping to the survey sky tessellation, subtracting non-astrophysical background, and co-adding the frames. Outputs: 
    1. Co-add images of various types, including Deep, seeing-optimized, and multi-color Co-Adds. 
    2. Co-add source detection and characterization. Outputs:
      • CoaddSource table 
  2. Source association, deblending, and measurements
    • Object table

Third Pass: Multi-epoch Measurement

  1. Multi-epoch object characterization of all CoaddSources, including simultaneous model fits to the position, motion, shape, and flux.
    • Object table updates
  2. Forced photometry of all Sources on the visit images, with the position, motion, shape, and deblending parameters kept fixed, to determine the flux at all epochs. Outputs: 
    • ForcedSource table

Calibration Products Production

The calibration products production is at present mostly conceptual, though some of the necessary operations will leverage the processing pipelines. In the absence of a detailed description of the processing, the major calibration reference products are listed below. Some corrections may be combined into a single file (e.g., flat-field and fringe), if appropriate. 

ProductCoverageFormatDescription
Amplifier gainamptableElectrons per detected count
Cross-talk coefficientsamptableSupports correction for electronic cross-talk between signal chains on the sensor
Linearity coefficientsamptableCorrection for non-linear pixel response
Bias residualampimageResidual DC signal after overscan correction
Bad pixel maskCCDimageBit-encoded pixels denoting known detector defects (bad columns, etc.)
Dark rateCCDimageCountrate of signal with shutter closed. Requires Bias Residual. Necessity is TBD.
Flat-fieldCCDimagePixel-to-pixel sensitivity variations
FringeCCDimageLow-level pattern from color-dependent interference. Intimately related to flat-field.
Astrometric standardsall-skycatalogStandard objects for which positions and motions are very accurately known
Photometric standardsall-skycatalogStandard stars for which absolute fluxes in all survey passbands are accurately known

Data Products from Productions

The data products that are produced and persisted in the AP, MOPS, or DRP are summarized here for quick reference. 

 

NameProductionDescription
Images  
Deep Co-AddDRP: Co-Add creationPer-band, overlapping Visit images are warped to the survey geometry, background subtracted, and co-added, with greater emphasis on depth than on quality metrics.
DifferenceAP; re-created in DRPDifference between a Visit image and a temporally appropriate Template, warped to the same geometry, and placed on the same photometric scale.
Multi-color Co-AddDRP: Co-Add creationSimilar to Deep Co-Add, but with co-addition of multiple passbands.
TemplateDRP: Co-Add creationSimilar to Deep Co-Add, except that contributing images are selected for quality attributes such as PSF size, airmass of observation. Transients, SSObjects, and artifacts are removed.
VisitAP; re-created in DRPExposures in a visit are combined into a single, calibrated image with PSF characterization and mask planes.
Catalogs  
CoaddSourceDRP:Astrophysical entities
ForcedSourceDRP: Single-epoch forced-photometry of CoaddSources that are not detected above the S/N threshold.
Object2Astrophysical entity, formed by associating Sources at multiple epochs to a common phenomenon.
Source2Single-epoch observation of an astrophysical entity, detected above the S/N threshold.
SSObjectLevel1: MOPSSolar system objects, formed by associating DIASources at different positions at multiple epochs to a plausible orbit of a solar system body.
DIASource1Single-epoch observation of an astrophysical entity, detected above the S/N threshold on a difference image.
DIAObject1Astrophysical entity, formed by associating DIASources at multiple epochs to a common phenomenon.
  • No labels