Page History

These draft notes cover the major updates made by Data Release Production to the LSST stack since release 11.0 (Summer 2015). Please record significant updates here so that they can ultimately be incorporated into the notes accompanying the Winter 2016 release.

Major Functionality and Interface Changes

Dumping task configuration parameters now includes documentation

Add the option of excluding mask planes before coaddition

The user-configurable paramter removeMaskPlanes has been added to AssembleCoaddConfig. This is a list of mask planes which will not be propagated to the coadd; by default, the CROSSTALK and NOT_DEBLENDED mask planes are removed.

Dumping task configuration parameters now includes documentation

That is, running ${TASK} ... --show config displays not only the names That is, running ${TASK} ... --show config displays not only the names and values of the configuration but also associated documentation.

Clean up interpolation tasks and implement useFallbackValueAtEdge

The interpImageTask in pipe_tasks has been refactored to implement a single run function for interpolation over a list of defects in an image.  This run function will accept an afw image type of either MaskedImage or Exposure.  A defects list can be passed in directly OR the name of mask plane can be passed from which a defects list will be created.  If a PSF is attached to the image, it will be used as the (required) argument for the interpolateoverDefects  function in meas_algorithms used for the interpolation.  Otherwise a FWHM (in pixels) can be provided or the defaultFWHM value in meas_algorithms' GaussianFactory is used.  Note that while the PSF is a required argument for meas_algorithms' interpolateOverDefects, function it is currently not being used, so it is not necessary to pass in an accurate PSF.

...

HSC backport: avoid I/O race conditions config write out

This adds functionality to avoid potential I/O race conditions when running multiple simultaneous process.  This is accomplished by writing to temporary files and then renaming them to the correct destination filename in a single operation. Also, to avoid similar race conditions in the backup file creation (e.g. config.py~1, config.py~2, ...), a --no-backup-config option (to be used with --clobber-config) is added here to prevent the backup copies being made. The outcome for this option is that the config that are still recorded are for the most recent run.

HSC backport: Introduce SafeClipAssembleCoaddTask which extends AssembleCoaddTask to make clipping safer

SafeClipAssembleCoaddTask does the following, 

...

HSC backport: Allow for some fraction of PSF Candidates to be reserved from PSF fitting

This adds functionality that allows for the reservation of a fraction of the PSF Candidates from PSF fitting. The reserved fraction can then be used to test for over-fitting, do cross-validation etc. To support this functionality, the run methods of CalibrateTask and MeasurePsfTask now accept the additional keyword argument expId. Dummy versions of this keyword were added to the SdssCalibrateTask and CfhtCalibrateTask to maintain a consistent API.

Updated pipeline configuration based on Hyper Suprime-Cam experience

Configuration defaults and metadata through the LSST single-processing pipeline have been updated to match the current best practice established on HSC. Major changes include:

...

Define polygon bounds for CCDs based on vignetted regions

This adds a function in ip_isr's isrTask.py to set a "valid polygon" for a given CCD exposure as the intersection of a polygon defined in focal plane coordinates and the CCD corners.  It is currently being used in obs_subaru's isr.py to set the polygon bounds (added in DM-2981) for a CCD exposure to include the non-vignetted regions.  The settings for the vignetted region is in a separate config file so that it can be used in different places in the code.

Introduce --rerun option for command line tasks

This new command line option streamlines the process of specifying input and output repositories for command line tasks. In its simplest form, the --rerun option is a shorthand for specifying an output repository: output is written to a location relative to the input. Thus

...

Introduce framework for injecting fake sources into data processing

A new boolean option (doFakes) and a retargetable task (fakes) have been introduced into the ProcessCcdTask. This (along with a new class called BaseFakeSourcesTask) sets up a frame work that others may use to introduce known fake sources into the data processing stream. However, this framework itself does not actually insert any fake data itself, but provides an interface others may use to define their own fake source injection task. To implement a fake injection task one must create a child class of BaseFakeSourcesTask (located in lsst.pipe.tasks.fakes) and overload the run method to do the work of injecting the sources. Each source that is injected should have a corresponding bit set in the FAKE maskplane which can be accessed with the self.bitmask variable. Once a task has been created, the config field fakes in processCcd must be retarged to point to the user created task. Additionally, the task will not be run unless the doFakes configuration option in processCcd is set to True. However, if this variable is set to True, and the task is not retarged processCcd will fail.

...

Add convenience routines for working with tracts

Added two new data id containers:

...

Warp images based on an XYTransform 

Added the ability to warp images using a transformation defined by an lsst::afw::geom::XYTransform

Add getCoordSystem to Coord and add UNKNOWN CoordSystem enum

Added method getCoordSystem to lsst::afw::coord::Coord. Also added UNKNOWN=-1 as a new CoordSystem enum (the existing enums retain their existing value).

Adapt joinMatchListWithCatalog to facilitate and simplify denormalizing a match list

The match lists created when performing image calibration (astrometry and photometry) are normalized (i.e. stripped down to a list of the matched reference and source ids and their distance) prior to being persisted.  The ability to denormalize a match catalog is very useful (for post qa analysis, for example).  This can now be done using the joinMatchListWithCatalog() function in meas_algorithms's LoadReferenceObjectsTask.  It has been moved from meas_astrom's ANetBasicAstrometryTask so that it can be easily accessed (requiring only that a reference object loader be initiated) and to allow it to work with any kind of reference catalog (i.e. other than a_net).

Add a script for visualizing skymaps and CCDs

The skymap package now contains the script showVisitSkyMap.py which provides a convenient way of visualizing the tracts, patches and CCDs contained in a set of visits.

Add functions to generate "unpacked matches" to and from a catalog

Functions have been added to lsst::afw::catalogMatches to provide the ability to convert a match list into a catalog and vice versa (this can be useful for post-processing analyses: qa analysis, for example).

Add a measurement algorithm which records the focal plane coordinates of sources

The SingleFrameFPPositionPlugin measurement plugin, available in meas_base, records the positions of source centroids in focal plane coordinates (which may be convenient for plotting). This plugin is not enabled by default, but may be switched on by requesting base_FPPosition in measurement configuration.

Add a measurement algorithm which records the Jacobian at the positions of sources

The SingleFrameJacobianPlugin calculates the ratio between the nominal Jacobian determinant at the source centroid (as determined by a user-specified pixel scale) and the actual Jacobian determinant as derived from the astrometric solution. This plugin is not enabled by default, but may be switched on by requesting base_Jacobian in measurement configuration.

Add a measurement algorithm which records the number of input images contributing to a coadd sources

When run on a source detected on a coadd, SingleFrameInputCountPlugin records the number of input images which were stacked to create the coadd at the position corresponding to the source centroid. The plugin is referred to as base_CountInputs, and is enabled by default when performing measurement on coadded images. It is not appropriate to enable this plugin when processing single visit (i.e., not coadded) images.

Add a measurement algorithm which records the variance at the positions of sources

The SingleFrameVariancePlugin records the median variance in the background around the position of the source being measured. The plugin is referred to as base_Variance and is enabled by default when performing single frame measurement.

Add a measurement algorithm which records source flux in an aperture scaled to the PSF

The ScaledApertureFluxAlgorithm measures the flux in a circular aperture with radius scaled to some user-specified multiple of the PSF FWHM. This plugin is not enabled by default, but may be switched on by requesting the base_ScaledApertureFlux in measurement configuration.

Add a measurement algorithm which quantifies the amount of "blendedness" of an object

The BlendednessAlgorithm measures the amount to which an object is blendeded.  Both the flux and shape of each child object are compared to measurements at the same point on the full image.  The size of the weight function used on both images is determined from the child object.  The blendedness metric implemented is defined as 1-childFlux/parentFlux. The plugin is referred to as base_Blendedness and is not enabled by default. 

Add option to temporarily remove an interpolated background prior to detection

This has the potential for removing a large number of junk detections around bright objects due to noise fluctuations in the elevated local background.  The extra subtracted interpolated background is added back in after detection.  Currently, the default setting for the config parameter doTempLocalBackround is set to False.

Add

...

a "simple" shape measurement algorithm

The SimpleShape algorithm, provided in the meas_extensions_simpleShape package, computes the non-adaptive elliptical Gaussian-weighted moments of an image. The plugin is referred to as ext_simpleShape_SimpleShape and is not enabled by defaultAdded function lsst.afw.coord.averageCoord, which will return an average coordinate (accounting for spherical geometry) given a list of input coordinates.

Integrate support for Hyper Suprime-Cam

Add Hirata-Seljak-Mandelbaum shape measurement algorithms

The meas_extensions_shapeHSM package has been added to the distribution. This provides a series of measurement algorithms based on the work by Hirata and Seljak (2003) and Mandelbaum (2005): please cite those works if publishing results based on this code. These algorithms are disabled by default; they can be enabled by requesting ext_shapeHSM_HsmShapeBj, ext_shapeHSM_HsmShapeLinear, ext_shapeHSM_HsmShapeKsb, ext_shapeHSM_HsmShapeRegauss, ext_shapeHSM_HsmSourceMoments and/or ext_shapeHSM_HsmPsfMoments in the measurement configuration.The obs_subaru camera package, which enables the stack to operate on data taken with the Suprime-Cam and Hyper Suprime-Cam instruments on Subaru, has been modernized, resolving build and test issues and integrating it with LSST's continuous integration system. It will now be included as part of the lsst_distrib release. (Note, though, that usage of Suprime-Cam with the stack is unsupported and unmaintained at present.)

Restructure PsfFitter algorithm

Add option to temporarily remove an interpolated background prior to detection

This has the potential for removing a large number of junk detections around bright objects due to noise fluctuations in the elevated local background.  The extra subtracted interpolated background is added back in after detection.  Currently, the default setting for the config parameter doTempLocalBackround is set to False.The PsfFitter routine has been refined to better accord with the meas_base measurement framework. This includes modernized error handling.

Propagate flags from individual visits to coadds

Add function to average coordinates

Added function lsst.afw.coord.averageCoord, which will return an average coordinate (accounting for spherical geometry) given a list of input coordinatesA task has been added which can propagate flags from individual visit catalogs to coadd catalogs. This is useful, for example, to track which stars in the coadd were used for measuring PSFs on the individual visits.

Integrate support for Hyper Suprime-Cam

The obs_subaru camera package, which enables the stack to operate on data taken with the Suprime-Cam and Hyper Suprime-Cam instruments on Subaru, has been modernized, resolving build and test issues and integrating it with LSST's continuous integration system. It will now be included as part of the lsst_distrib release. (Note, though, that usage of Suprime-Cam with the stack is unsupported and unmaintained at present.)

Prototype Brigher-Fatter correction

Code for correcting for the Brighter-Fatter effect on CCDs is now available in the stack. It is enabled using the doBrighterFatter configuration option to IsrTask. It requires a pre-generated correction kernel. Calculation of this kernel is not currently performed within the stack: a prototype exists, and will be merged to the Calibration Products Pipeline in a future cycle.

Reimplement PSF Shapelet approximations for CModel

A new algorithm for computing multi-shapelet approximations (DoubleShapeletPsfApprox) has been added to meas_modelfit.  This is simpler and more robust than the old algorithm, which has been renamed to GeneralShapeletPsfApprox.  The new algorithm is recommended for production use, and is now the default.

Aperture correction on coadd measurements

Aperture corrections are now enabled for measurements performed on coadds.

Propagate flags from individual visits to coadds

A task has been added which can propagate flags from individual visit catalogs to coadd catalogs. This is useful, for example, to track which stars in the coadd were used for measuring PSFs on the individual visits.

Return grown Footprints from detection

By default, Footprints return by SourceDetectionTask are now grown by a multiple of the PSF size.

Enable measurement of "sky objects" in coadd processing

Prototype Brighter-Fatter correction

Code for correcting for the Brighter-Fatter effect on CCDs is now available in the stack. It is enabled using the doBrighterFatter configuration option to IsrTask. It requires a pre-generated correction kernel. Calculation of this kernel is not currently performed within the stack: a prototype exists, and will be merged to the Calibration Products Pipeline in a future cycleSky objects correspond to source properties measured at positions when no objects have been detected. This enables us to better characterize the depth of the survey. This functionality is enabled by setting nSkySourcesPerPatch to a non-zero value in the configuration of MergeDetectionsTask.

Bug Fixes

The following fixes resolve problems visible to end users.

Persist LTVn headers as floating point numbers

Fix bug when identifying existing peaks in a merge

Aperture correction on coadd measurements

Aperture corrections are now enabled for measurements performed on coaddsIf two separate footprints from the same catalog are merged due to an existing merged object which overlaps both of them the flags of which peaks are being detected were not being propagated. This issue caused apparent dropouts of sources and has now been fixed.

Fix situation in which the getChildren method of SourceCatalog may return the wrong information

The getChildren method requires that the result must be sorted by parent. This is naturally the case when the catalog is produced by detection or deblending tasks. However, if multiple catalogs are concatenated together this condition may no longer be true. The getChildren method was updated to raise an exception if the precondition of sorting is not met.

Return grown Footprints from detection

By default, Footprints return by SourceDetectionTask are now grown by a multiple of the PSF size.

Fix warping when the WCS have different coordinate systems

Enable measurement of "sky objects" in coadd processing

Sky objects correspond to source properties measured at positions when no objects have been detected. This enables us to better characterize the depth of the survey. This functionality is enabled by default; it can be disabled by setting nSkySourcesPerPatch to zero in the configuration of MergeDetectionsTaskWarping assumed that the the sky representation of both WCS was identical.

Correct bad default minInitialRadius for CModel

The minInitialRadius configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classifications.

Correct algebraic error in CModel uncertainty calculation

Specification of output directory is now mandatory

When running a command line task which produces output it is now mandatory to specify an output directory (previously, if no output location was specified, data products were written back to the input repository). Output locations may be specified with the --rerun or --output command line options. More information is available on community.lsst.orgThere was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude.

NaiveDipoleCentroid and NaiveDipoleFlux algorithms no longer require a centroid slot

Bright object masks

Given an input catalog listing the known positions and sizes of bright objects, a bit is set in the mask plane for every pixel lying within the object's footprintPreviously, initializing these algorithms was only possible if a centroid was already defined. That was not only unneccessary, but also made them more complicated to use, particularly in testing.

Update (some) example code to run with recent stack versions

Overhaul ProcessCcdTask and subtasks, including eliminating camera-specific variants

Overhauled ProcessCcdTask and CalibrateTask to simplify the code and eliminate the need for camera-specific variants. See Changes to ProceessCcdTask and Subtasks for detailsChanges in afw::table had broken examples/calibrateTask.py in pipe_tasks. It has now been updated to comply with the latest afw::table API.

Fix a failure to appropriately log failed task execution

When task execution fails, we add a message to the log (with level FATAL). In some cases, the very act of attempting to log this message could throw an exception, and information about the original error was lost. This has now been resolved.

Updates to Skymap packages

...

CModel fitting improvements

CModel is a model fitting approach in which a pure exponential and pure de Vaucouleur are each fit separately, and then their linear combination is fit while the ellipse parameters are held fixed. Improvements in this release make CModel fitting faster and improves results on objects which are detected with an unphysically large likelihood radius. This has been achieved in three ways:

• The initial approximate exponential fit that is used to determine the starting parameters and pixel region to use for the exp and dev fit now uses per-pixel variances;
• The method used to determine the pixel region used in fitting has been adjusted to make smarter choices, using fewer pixels on average for all objects and many fewer pixels for unphysically large objects;
• A new semi-empirical Bayesian prior on radius and ellipticity based on COSMOS distributions has been introduced.

Port bugfixes in meas_extensions_shapeHSM from HSC to LSST

Various bug fixes to the shapeHSM measurement algorithm were ported and include:

• Ensuring masks have same xy0 as images
• Fixing scaling coefficient in ellipticity error when using distortion
• Provide appropriate mask for variance image when measuring median variance around object
• Change from standard deviation to variance in the sky value when calling GalSim hsm code, as required by GalSim
• Fixes to unit tests to comply with above changes, and the port from the HSC framework to the LSST framework

Fix variance in coadded images

Warping images in order to coadd them loses variance into covariance. This is mitigated by scaling the variance plane of the coadd. The scaling was being applied incorrectly in some cases. This has now been fixed.

Astropy Table views into LSST Catalog objects

Astropy Table views into LSST catalog objects can now be created.  These views share underlying data buffers (aside from flag fields), making them read-write, but rows and columns added on either side will not be visible in the other.  Two equivalent interfaces are available:

astropy_table = lsst_catalog.asAstropy()

and (in Astropy >= v1.2):

astropy_table = astropy.table.Table(lsst_catalog)

QTable objects can also be used, resulting in columns that use Astropy's units  package to enforce unit correctness.  These interfaces have multiple options to control the details of the view, including how to handle columns that require copies; see the Python on-line help for asAstropy for more information.

While LSST's catalog objects have features that make them particularly useful in building pipelines, Astropy's are much more appropriate for most analysis tasks, and we strongly recommend using them for any analysis tasks that need to add columns to tables or combine columns from multiple tables.

Fix variance in deblended sources

The deblender incorrectly scaled the variance plane in deblended sources with the fraction of the total flux assigned to the source. This has been corrected.

Fix logic for applying aperture corrections

This fixes a but whereby the aperture corrections were being applied only after all the measurement plugins had run through, independent of their execution order. This resulted in plugins whose measurements rely on aperture corrected fluxes (i.e. with execution order > APCORR_ORDER) being applied prior to the aperture correction, leading to erroneous results. The only plugin that was affected by this at this time was base_ClassificationExtendedness.

Add an "afterburner" measurement facility

This new functionality makes it possible to register plugins to calculate quantities based on the results of pixel measurement algorithms. This might include, for example, star-galaxy separation or applying aperture corrections. Afterburners of this type are run after measurement plugins, and do not have access to pixel data.

More uniform support for assigning to catalog columns

Bug Fixes

The following fixes resolve problems visible to end users.

Persist LTVn headers as floating point numbers

When persisting to a FITS file, these header cards were previously, incorrectly, stored as integersAssignment of scalars or NumPy arrays to columns of afw.table.Catalog objects (e.g. "catalog["column"] = value") is now more uniformly supported across types (support was inconsistent before, and never allowed scalar or augmented assignment).  Flag columns still do not support column assignment, and Flag column access still returns a copy, not a view, because Flag values are stored internally as individual bits within a larger integer.

Upgraded WCSLIB to version 5.13

Fix bug when identifying existing peaks in a merge

If two separate footprints from the same catalog are merged due to an existing merged object which overlaps both of them the flags of which peaks are being detected were not being propagated. This issue caused apparent dropouts of sources and has now been fixedVersion 5.13 of WCSLIB resolves memory corruption errors that could crash the stack in some circumstances.

Fix situation in which the getChildren method of SourceCatalog may return the wrong information

The getChildren method requires that the result must be sorted by parent. This is naturally the case when the catalog is produced by detection or deblending tasks. However, if multiple catalogs are concatenated together this condition may no longer be true. The getChildren method was updated to raise an exception if the precondition of sorting is not met.

Fix warping when the WCS have different coordinate systems

Warping assumed that the the sky representation of both WCS was identical.

Correct bad default minInitialRadius for CModel

The minInitialRadius configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classifications.

Fix rotation for instrument signature removal in obs_subaru

Correct algebraic error in CModel uncertainty calculation

There was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude.

NaiveDipoleCentroid and NaiveDipoleFlux algorithms no longer require a centroid slot

Previously, initializing these algorithms was only possible if a centroid was already defined. That was not only unneccessary, but also made them more complicated to use, particularly in testing.

Update (some) example code to run with recent stack versions

Changes in afw::table had broken examples/calibrateTask.py in pipe_tasks. It has now been updated to comply with the latest afw::table API.

Fix a failure to appropriately log failed task execution

When task execution fails, we add a message to the log (with level FATAL). In some cases, the very act of attempting to log this message could throw an exception, and information about the original error was lost. This has now been resolved.

Updates to Skymap packages

Add functions to return patches and tracts which contain given coordinates, i.e. conversions between celestial coordinates and tract,patch indicies. Functions include findClosestTractPatchList, findAllTract, and findTractPatchList which finds the closets tract and patch that overlaps coordinates, finds all tracts which include the specified coordinate, and finds tracts and patches that overlap a region respectively.

Fix variance in coadded images

Warping images in order to coadd them loses variance into covariance. This is mitigated by scaling the variance plane of the coadd. The scaling was being applied incorrectly in some cases. This has now been fixed.

Fix variance in deblended sources

The deblender incorrectly scaled the variance plane in deblended sources with the fraction of the total flux assigned to the source. This has been corrected.

Fix logic for applying aperture corrections

This fixes a bug whereby the aperture corrections were being applied only after all the measurement plugins had run through, independent of their execution order. This resulted in plugins whose measurements rely on aperture corrected fluxes (i.e. with execution order > APCORR_ORDER) being applied prior to the aperture correction, leading to erroneous results. The only plugin that was affected by this at this time was base_ClassificationExtendedness.

More uniform support for assigning to catalog columns

Assignment of scalars or NumPy arrays to columns of afw.table.Catalog objects (e.g. "catalog["column"] = value") is now more uniformly supported across types (support was inconsistent before, and never allowed scalar or augmented assignment).  Flag columns still do not support column assignment, and Flag column access still returns a copy, not a view, because Flag values are stored internally as individual bits within a larger integer.

Upgraded WCSLIB to version 5.13

Version 5.13 of WCSLIB resolves memory corruption errors that could crash the stack in some circumstances.

Fix rotation for instrument signature removal in obs_subaru

Approximately half of the HSC CCDs are rotated 180 deg with respect to the others. Two others have 90 deg rotations and another two have 270 deg rotations (see HSC CCD layout). The raw images for the rotated CCDs thus need to be rotated to match the rotation of their associated calibration frames (in the context of how they have currently been ingested) prior to applying the corrections. This is accomplished by rotating the exposure using the rotated context manager function in obs_subaru's isr.py and the nQuarter specification in the policy file for each CCD. Currently, rotated uses afw's rotateImageBy90 (which apparently rotates in a counter-clockwise direction) to rotated the exposure by 4 - nQuarter turns. This turns out to be the wrong rotation for the odd nQuarter CCDs.  This issue fixes this bug, leading to much improved processing of HSC CCD's 100, 101, 102, and 103.  Note that, in the future, the ingestion of the calibration data will be updated such that no rotations are necessary (so they will then be removed from  obs_subaru accordingly).

Fix a silent failure to apply config overrides

When applying a config override, using a variable which hadn't been defined should throw a NameError, which ultimately propagages to the end user to notify them that something has gone awry. This warning was being incorrectly suppressed.

Correctly set flags for bad SdssShape measurements.

The SdssShape algorithm provides both shape and flux measurements. In some cases, a failed shape measurement could go un-noticed, resulting in an incorrect and unflagged flux measurement being associated with that source. This is now checked for, and bad fluxes are appropriately flagged.

Fix CONSTANT background interpolation of bad data

When performing interpolation over bad data (e.g. every pixel masked), all interpolation types other than CONSTANT would return NaNs; CONSTANT would throw. This has now been changed so that CONSTANT also returns NaNs.

Accommodate pixel padding when unpersisting reference catalog matches

The reference object loader in meas_algorithm's loadReferenceObjects.py grows the bbox by the config parameter pixelMargin: doc = "Padding to add to 4 all edges of the bounding box (pixels)" when setting the radius of the sky circle to be searched in the reference catalog. This is set to 50 by default but was not reflected by the radius parameter set in the metadata, which left open the possibility that some matches could reside outside the circle searched within the unpersisted radius.  Additionally, the match metadata was being set after the exposure's wcs had been updated, also leading to an inconsistency with the sky circle that was actually searched.  We now ensure that the actual sky circle that was searched for reference objects is the one set and persisted in the match metadata.

Correct misleading filter fallback error message

When failing to load a calib, if fallbackFilterName was not set, a confusing and apparently unrelated error message would be generated (Unknown value type for filter: <type 'NoneType'>). This has been corrected to properly inform the user about the issue.

Build and code improvements

These improvements should not usually be visible to end users. They may, however, be important for developers.

Work-around incompatibilities with NumPy 1.10

NumPy 1.10 introduced API changes which were incompatible with existing usage in the stack. The latter has been updated to match

When building boost warn user if user-config.jam or site-config.jam exists

Building boost can fail if a user-config.jam or site-config.jam exist and have options which conflict with the lsst build configuration process. Introduce a warning message if either of these files are found to notify the user.

Remove deprecated Task.display() method

This method has been deprecated since release 9.2 (S14). It has been removed from the codebase, and all stack code updated to directly interface with afw.display or to use helper functions defined in meas_astrom.

Efficiency improvement in converting Masks to DefectLists

The previous version of routine was extremely memory intensive when large numbers of pixels were masked.

Add a new task parallelization framework

The ctrl_pool package has been added to the LSST stack. This is a high-level parallelization framework used for distributing Task execution across a cluster, based on an MPI process pool. It is based on work carried out on Hyper Suprime-Cam. It is not intended to be the long-term solution to parallelized processing in the LSST stack, but meets our data processing needs until the fully-fledged parallelization middleware is available.

Add parallel-processing top level tasks

The new pipe_drivers package builds upon ctrl_pool, above, to provide command-line scripts which coordinate distributed execution of the single-frame, coaddition and multiband processing steps using either the Python multiprocessing module or with a SLURM batch scheduler on a cluster.

Correct coding errors in meas_extensions_shapeHSM

Adjust test tolerances to be compatible with MKL-based NumPy

Anaconda 2.5 ships, by default, with a version of NumPy built against Intel MKL rather than OpenBLAS. This can change some numerical results slightly, necessitating a change to test tolerancesSome minor errors when this code was converted to the meas_base framework left it unusable. These have been corrected.

Build and code improvements

These improvements should not usually be visible to end users. They may, however, be important for developers.

Work-around incompatibilities with NumPy 1.10

Now possible to directly get a Filter's canonical name and aliases

Added the convenience methods getCanonicalName() and getAliases() to lsst.afw.image.Filter, accessible from both C++ and Python. These return the canonical name and the aliases, respectively, of the filter. This information was previously only available through an awkward sequence of method calls.NumPy 1.10 introduced API changes which were incompatible with existing usage in the stack. The latter has been updated to match

Fix build issues with recent clang

Recent releases of the clang C/C++ compiler, as shipped with Apple XCode, caused build failures in the stack. Although we believe this may be a problem with clang, we have worked around it within the stack code. We hope to track down the source of the error and, if appropriate, report it to the clang developers in future.

When building boost warn user if user-config.jam or site-config.jam exists

Remove deprecated Task.display() method

Fix incorrect linking against Anaconda-provided libraries when using CMake

Some external packages -- mariadb, mariadbclient -- use a CMake based build system. This can incorrectly link against some libraries bundled with the Anaconda Python distribution, rather than the system-provided equivalents; in the end, this results in a build failure. We have adjusted the build process of the affected packages to work around this errorThis method has been deprecated since release 9.2 (S14). It has been removed from the codebase, and all stack code updated to directly interface with afw.display or to use helper functions defined in meas_astrom.

Efficiency improvement in converting Masks to DefectLists

Execute afw test suite when afwdata is not available

Some tests in the afw package rely on data from the afwdata package. The test suite would search for afwdata, and skip all of the afw tests if afwdata is not available. This check has been made smarter, so that only tests which actually require afwdata are now skippedThe previous version of routine was extremely memory intensive when large numbers of pixels were masked.

Add a new task parallelization framework

Disable implicit threading

Low-level threading packages (such as OpenBLAS or MKL) can implicitly use many threads. Since the LSST stack also parallelizes at a higher level (e.g. using Python's multiprocessing module), this can cause underdesireable contention. We now disable implicit threading when explicitly parallelizing at a higher level to protect the user from this. Implicit threading can be re-enabled by setting the LSST_ALLOW_IMPLICIT_THREADS environment variable.  For more details, see this CLO postThe ctrl_pool package has been added to the LSST stack. This is a high-level parallelization framework used for distributing Task execution across a cluster, based on an MPI process pool. It is based on work carried out on Hyper Suprime-Cam. It is not intended to be the long-term solution to parallelized processing in the LSST stack, but meets our data processing needs until the fully-fledged parallelization middleware is available.

Migrate to standard smart pointers

C++11 introduced new smart pointer types (std::unique_ptr, std::shared_ptr and std::weak_ptr). We have migrated from the previously used Boost smart pointers to their standard equivalents.

Adjust test tolerances to be compatible with MKL-based NumPy

Now possible to directly get a Filter's canonical name and aliases

Overhaul ProcessCcdTask and subtasks, including eliminating camera-specific variants