Page History

These draft notes cover the major updates made by Data Release Production to the LSST stack since release 11.0 (Summer 2015). Please record significant updates here so that they can ultimately be incorporated into the notes accompanying the Winter 2016 release.

...

Add

...

Hirata-Seljak-Mandelbaum shape measurement algorithms

The meas_extensions_shapeHSM package has been added to the distribution. This provides a series of measurement algorithms based on the work by Hirata and Seljak (2003) and Mandelbaum (2005): please cite those works if publishing results based on this code. These algorithms are disabled by default; they can be enabled by requesting ext_shapeHSM_HsmShapeBj, ext_shapeHSM_HsmShapeLinear, ext_shapeHSM_HsmShapeKsb, ext_shapeHSM_HsmShapeRegauss, ext_shapeHSM_HsmSourceMoments and/or ext_shapeHSM_HsmPsfMoments in the measurement configurationThis has the potential for removing a large number of junk detections around bright objects due to noise fluctuations in the elevated local background.  The extra subtracted interpolated background is added back in after detection.  Currently, the default setting for the config parameter doTempLocalBackround is set to False.

Add function to average coordinates

Added function lsst.afw.coord.averageCoord, which will return an average coordinate (accounting for spherical geometry) given a list of input coordinates.

Integrate support for Hyper Suprime-Cam

The obs_subaru camera package, which enables the stack to operate on data taken with the Suprime-Cam and Hyper Suprime-Cam instruments on Subaru, has been modernized, resolving build and test issues and integrating it with LSST's continuous integration system. It will now be included as part of the lsst_distrib release. (Note, though, that usage of Suprime-Cam with the stack is unsupported and unmaintained at present.)

Add option to temporarily remove an interpolated background prior to detection

This has the potential for removing a large number of junk detections around bright objects due to noise fluctuations in the elevated local background.  The extra subtracted interpolated background is added back in after detection.  Currently, the default setting for the config parameter doTempLocalBackround is set to False.

Add function to average coordinates

Added function lsst.afw.coord.averageCoord, which will return an average coordinate (accounting for spherical geometry) given a list of input coordinates.

Reimplement PSF Shapelet approximations for CModel

A new algorithm for computing multi-shapelet approximations (DoubleShapeletPsfApprox) has been added to meas_modelfit.  This is simpler and more robust than the old algorithm, which has been renamed to GeneralShapeletPsfApprox.  The new algorithm is recommended for production use, and is now the default.

Integrate support for Hyper Suprime-Cam

The obs_subaru camera package, which enables the stack to operate on data taken with the Suprime-Cam and Hyper Suprime-Cam instruments on Subaru, has been modernized, resolving build and test issues and integrating it with LSST's continuous integration system. It will now be included as part of the lsst_distrib release. (Note, though, that usage of Suprime-Cam with the stack is unsupported and unmaintained at present.)

Propagate flags from individual visits to coadds

A task has been added which can propagate flags from individual visit catalogs to coadd catalogs. This is useful, for example, to track which stars in the coadd were used for measuring PSFs on the individual visits.

Prototype Brighter-Fatter correction

Reimplement PSF Shapelet approximations for CModel

A new algorithm for computing multi-shapelet approximations (DoubleShapeletPsfApprox) has been added to meas_modelfit.  This is simpler and more robust than the old algorithm, which has been renamed to GeneralShapeletPsfApprox.  The new algorithm is recommended for production use, and is now the defaultCode for correcting for the Brighter-Fatter effect on CCDs is now available in the stack. It is enabled using the doBrighterFatter configuration option to IsrTask. It requires a pre-generated correction kernel. Calculation of this kernel is not currently performed within the stack: a prototype exists, and will be merged to the Calibration Products Pipeline in a future cycle.

Propagate flags from individual visits to coadds

A task has been added which can propagate flags from individual visit catalogs to coadd catalogs. This is useful, for example, to track which stars in the coadd were used for measuring PSFs on the individual visits.

Aperture correction on coadd measurements

Aperture corrections are now enabled for measurements performed on coadds.

Return grown Footprints from detection

Prototype Brighter-Fatter correction

Code for correcting for the Brighter-Fatter effect on CCDs is now available in the stack. It is enabled using the doBrighterFatter configuration option to IsrTask. It requires a pre-generated correction kernel. Calculation of this kernel is not currently performed within the stack: a prototype exists, and will be merged to the Calibration Products Pipeline in a future cycleBy default, Footprints return by SourceDetectionTask are now grown by a multiple of the PSF size.

Enable measurement of "sky objects" in coadd processing

Aperture correction on coadd measurements

Aperture corrections are now enabled for measurements performed on coadds.

Specification of output directory is now mandatory

Return grown Footprints from detection

By default, Footprints return by SourceDetectionTask are now grown by a multiple of the PSF sizeWhen running a command line task which produces output it is now mandatory to specify an output directory (previously, if no output location was specified, data products were written back to the input repository). Output locations may be specified with the --rerun or --output command line options. More information is available on community.lsst.org.

Bright object masks

Enable measurement of "sky objects" in coadd processing

Sky objects correspond to source properties measured at positions when no objects have been detected. This enables us to better characterize the depth of the survey. This functionality is enabled by default; it can be disabled by setting nSkySourcesPerPatch to zero in the configuration of MergeDetectionsTaskGiven an input catalog listing the known positions and sizes of bright objects, a bit is set in the mask plane for every pixel lying within the object's footprint.

Overhaul ProcessCcdTask and subtasks, including eliminating camera-specific variants

Overhauled ProcessCcdTask and CalibrateTask to simplify the code and eliminate the need for camera-specific variants. See Changes to ProceessCcdTask and Subtasks for details.

CModel fitting improvements

CModel is a model fitting approach in which a pure exponential and pure de Vaucouleur are each fit separately, and then their linear combination is fit while the ellipse parameters are held fixed. Improvements in this release make CModel fitting faster and improves results on objects which are detected with an unphysically large likelihood radius. This has been achieved in three ways:

...

Specification of output directory is now mandatory

When running a command line task which produces output it is now mandatory to specify an output directory (previously, if no output location was specified, data products were written back to the input repository). Output locations may be specified with the --rerun or --output command line options. More information is available on community.lsst.org

...

.

Astropy Table views into LSST Catalog objects

Astropy Table views into LSST catalog objects can now be created.  These views share underlying data buffers (aside from flag fields), making them read-write, but rows and columns added on either side will not be visible in the other.  Two equivalent interfaces are available:

astropy_table = lsst_catalog.asAstropy()

and (in Astropy >= v1.2):

astropy_table = astropy.table.Table(lsst_catalog)

QTable objects can also be used, resulting in columns that use Astropy's units  package to enforce unit correctness.  These interfaces have multiple options to control the details of the view, including how to handle columns that require copies; see the Python on-line help for asAstropy for more information.

While LSST's catalog objects have features that make them particularly useful in building pipelines, Astropy's are much more appropriate for most analysis tasks, and we strongly recommend using them for any analysis tasks that need to add columns to tables or combine columns from multiple tables.

Bright object masks

Given an input catalog listing the known positions and sizes of bright objects, a bit is set in the mask plane for every pixel lying within the object's footprint.

Overhaul ProcessCcdTask and subtasks, including eliminating camera-specific variants

Overhauled ProcessCcdTask and CalibrateTask to simplify the code and eliminate the need for camera-specific variants. See Changes to ProceessCcdTask and Subtasks for details.

Add an "afterburner" measurement facility

...

CModel fitting improvements

CModel is a model fitting approach in which a pure exponential and pure de Vaucouleur are each fit separately, and then their linear combination is fit while the ellipse parameters are held fixed. Improvements in this release make CModel fitting faster and improves results on objects which are detected with an unphysically large likelihood radius. This has been achieved in three ways:

• The initial approximate exponential fit that is used to determine the starting parameters and pixel region to use for the exp and dev fit now uses per-pixel variances;
• The method used to determine the pixel region used in fitting has been adjusted to make smarter choices, using fewer pixels on average for all objects and many fewer pixels for unphysically large objects;
• A new semi-empirical Bayesian prior on radius and ellipticity based on COSMOS distributions has been introduced.

Bug Fixes

The following fixes resolve problems visible to end users.

Persist LTVn headers as floating point numbers

When persisting to a FITS file, these header cards were previously, incorrectly, stored as integers.

Fix bug when identifying existing peaks in a merge

Astropy Table views into LSST Catalog objects

Astropy Table views into LSST catalog objects can now be created.  These views share underlying data buffers (aside from flag fields), making them read-write, but rows and columns added on either side will not be visible in the other.  Two equivalent interfaces are available:

astropy_table = lsst_catalog.asAstropy()

and (in Astropy >= v1.2):

astropy_table = astropy.table.Table(lsst_catalog)

QTable objects can also be used, resulting in columns that use Astropy's units  package to enforce unit correctness.  These interfaces have multiple options to control the details of the view, including how to handle columns that require copies; see the Python on-line help for asAstropy for more information.

While LSST's catalog objects have features that make them particularly useful in building pipelines, Astropy's are much more appropriate for most analysis tasks, and we strongly recommend using them for any analysis tasks that need to add columns to tables or combine columns from multiple tablesIf two separate footprints from the same catalog are merged due to an existing merged object which overlaps both of them the flags of which peaks are being detected were not being propagated. This issue caused apparent dropouts of sources and has now been fixed.

Fix situation in which the getChildren method of SourceCatalog may return the wrong information

The getChildren method requires that the result must be sorted by parent. This is naturally the case when the catalog is produced by detection or deblending tasks. However, if multiple catalogs are concatenated together this condition may no longer be true. The getChildren method was updated to raise an exception if the precondition of sorting is not met.

Fix warping when the WCS have different coordinate systems

Warping assumed that the the sky representation of both WCS was identical.

Correct bad default minInitialRadius for CModel

The minInitialRadius configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classifications.

Add an "afterburner" measurement facility

This new functionality makes it possible to register plugins to calculate quantities based on the results of pixel measurement algorithms. This might include, for example, star-galaxy separation or applying aperture corrections. Afterburners of this type are run after measurement plugins, and do not have access to pixel data.

Correct algebraic error in CModel uncertainty calculation

Bug Fixes

The following fixes resolve problems visible to end users.

Persist LTVn headers as floating point numbers

When persisting to a FITS file, these header cards were previously, incorrectly, stored as integersThere was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude.

NaiveDipoleCentroid and NaiveDipoleFlux algorithms no longer require a centroid slot

Fix bug when identifying existing peaks in a merge

If two separate footprints from the same catalog are merged due to an existing merged object which overlaps both of them the flags of which peaks are being detected were not being propagated. This issue caused apparent dropouts of sources and has now been fixedPreviously, initializing these algorithms was only possible if a centroid was already defined. That was not only unneccessary, but also made them more complicated to use, particularly in testing.

Update (some) example code to run with recent stack versions

Fix situation in which the getChildren method of SourceCatalog may return the wrong information

The getChildren method requires that the result must be sorted by parent. This is naturally the case when the catalog is produced by detection or deblending tasks. However, if multiple catalogs are concatenated together this condition may no longer be true. The getChildren method was updated to raise an exception if the precondition of sorting is not metChanges in afw::table had broken examples/calibrateTask.py in pipe_tasks. It has now been updated to comply with the latest afw::table API.

Fix a failure to appropriately log failed task execution

Fix warping when the WCS have different coordinate systems

Warping assumed that the the sky representation of both WCS was identical.When task execution fails, we add a message to the log (with level FATAL). In some cases, the very act of attempting to log this message could throw an exception, and information about the original error was lost. This has now been resolved.

Updates to Skymap packages

Correct bad default minInitialRadius for CModel

The minInitialRadius configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classificationsAdd functions to return patches and tracts which contain given coordinates, i.e. conversions between celestial coordinates and tract,patch indicies. Functions include findClosestTractPatchList, findAllTract, and findTractPatchList which finds the closets tract and patch that overlaps coordinates, finds all tracts which include the specified coordinate, and finds tracts and patches that overlap a region respectively.

Port bugfixes in meas_extensions_shapeHSM from HSC to LSST

Various bug fixes to the shapeHSM measurement algorithm were ported and include:

...

Correct algebraic error in CModel uncertainty calculation

There was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude.

NaiveDipoleCentroid and NaiveDipoleFlux algorithms no longer require a centroid slot

Previously, initializing these algorithms was only possible if a centroid was already defined. That was not only unneccessary, but also made them more complicated to use, particularly in testing.

Update (some) example code to run with recent stack versions

Changes in afw::table had broken examples/calibrateTask.py in pipe_tasks. It has now been updated to comply with the latest afw::table API.

Fix a failure to appropriately log failed task execution

When task execution fails, we add a message to the log (with level FATAL). In some cases, the very act of attempting to log this message could throw an exception, and information about the original error was lost. This has now been resolved.

Updates to Skymap packages

Add functions to return patches and tracts which contain given coordinates, i.e. conversions between celestial coordinates and tract,patch indicies. Functions include findClosestTractPatchList, findAllTract, and findTractPatchList which finds the closets tract and patch that overlaps coordinates, finds all tracts which include the specified coordinate, and finds tracts and patches that overlap a region respectively.

...

Fix variance in coadded images

...

Correct coding errors in meas_extensions_shapeHSM

Some minor errors when this code was converted to the meas_base framework left it unusable. These have been corrected.

Fix a silent failure to apply config overrides

...