LSST Stack Winter 2015 Release

The Winter2015 release of the LSST stack is internally numbered v10_1 in eups and eups distrib; each git repository has been tagged with "10.1".

These release notes document the major updates since v9_2, which was a developer-only release that substitutesd for the Summer 2014 release.

Major Functionality and Interface Changes

Measurement and Table update

Significant rework was performed on the measurement plugin framework and the schemas that it uses in afw::table.  Many more details may be found at Measurement Framework Overhaul Release Notes.

One consequence of this overhaul is that measurements using the SdssCentroid algorithm, which is currently the default, require that a PSF be attached to the Exposure being measured.  This could be as simple as attaching a SingleGaussianPsf:

# Add a Gaussian PSF to the exposure.  This is needed by the SDSS Centroid Algorithm.
# For the kernel make it 3*sigma on either side of the central pixel.
    sigma = 1.0
    size = 2*int(3*sigma) + 1
    gaussianPSF = measAlg.SingleGaussianPsf(size, size, sigma)
    exposure.setPsf(gaussianPSF)

 

The alternative of using the GaussianCentroid algorithm, which does not require a PSF, is not recommended.

(DM-958, DM-1058, DM-1070, DM-419, DM-836, DM-241, DM-1281, DM-1282, DM-1218, DM-964)

Image origins now default to PARENT

The Stack has long had the concept of a sub-image or a bounding box that is indexed in the coordinate space of the image it was derived from, indicated by PARENT, or indexed in its own, new coordinate space, indicated by LOCAL.  The offset between these coordinate spaces is stored in the XY0 attribute of the sub-image.  The difference between these two concepts has long been a source of confusion, and the fact that the default was LOCAL often led to errors.

The default used when creating a sub-image or bounding box is now PARENT, which is almost always correct.  Changes were made throughout the code to implement this; in some cases these changes have fixed pre-existing bugs.

(DM-841, DM-845, DM-840, DM-846, DM-848, DM-843, DM-763, DM-7, DM-1684, DM-1682, DM-1176, DM-1680, DM-1076)

Astrometry refactoring

The meas_astrom package has been overhauled and the astrometry solving tasks refactored. 

Reference object catalogs have a new schema, which is documented in lsst.meas.algorithms.LoadReferenceObjectsTask. Some important differences:

 LoadReferenceObjectsTask is a new abstract base class for reference object loaders; presently we offer one concrete implementation: LoadAstrometryNetTask which loads data from astrometry.net index files.

The default astrometry task is ANetAstrometryTask. This is a refactored version of lsst.pipe.tasks.AstrometryTask (which no longer exists). The user interface is basically the same, though the argument formerly named "sources" has been renamed to "sourceCat" for clarity. The internals have been updated to use LoadAstrometryNetTask, and thus to return reference objects using the new schema.

We also offer a new astrometry task AstrometryTask, which is designed to be more configurable drop-in replacement for ANetAstrometryTask. AstrometryTask is a simple driver that calls three sub-tasks:

Unfortunately the new AstrometryTask is not providing as accurate astrometry as the older ANetAstrometryTask. Dominique Boutigny provided a demonstration of this using CFHT data: /lsst8/boutigny/valid_cfht on lsst-dev. That is why the new AstrometryTask is not yet used by default.

Note: the new astrometry task does not compute distances in the output match list (this will be fixed in the next release, using DM-2511).  They can be computed manually using this Python function:

 

def setMatchDistance(matches):
    """Set the distance field of the matches in a match list to the distance in radians on the sky

 

    @warning the coord field of the source in each match must be correct

 

    @param[in,out] matches a list of matches, an instance of lsst.afw.table.ReferenceMatch
    """
    if len(matches) < 1:
        return

 

    sourceCoordKey = matches[0].first.schema["coord"].asKey()
    refObjCoordKey = matches[0].second.schema["coord"].asKey()
    for match in matches:
        sourceCoord = match.first.get(sourceCoordKey)
        refObjCoord = match.second.get(refObjCoordKey)
        match.distance = refObjCoord.angularSeparation(sourceCoord).asRadians()

 

(DM-1576, DM-1600, DM-1580, DM-1577, DM-1672, DM-2185)

DistortedTanWcs class now available

This class contains both a distortion model and a TanWcs.  It is appropriate for describing this combination of atmospheric and optical coordinate transformations, particularly for use in astrometric calibration.  The class is not yet persistable, however.  Documentation for the class is available.

(DM-1969)

Utilities to generate a Camera object from FITS headers

Creating an afw.cameraGeom.Camera is typically done in an obs_ package and involves understanding both the class and the instrument.  A utility library is now provided in lsst.afw.cameraGeom.fitsUtils that allows the object creation to be done based on FITS headers.  Documentation of the DetectorBuilder class is available.

(DM-1296)

RGB image generation moved into afw.display

The afw_extensions_rgb package contained code to properly create RGB images based on three input Images, including replacement of saturated pixels.  This code has now been moved into afw.display.  As a result, afw now has an optional dependency on matplotlib, which is used to output certain kinds of RGB images.

Documentation of the mapping classes and RGB image generation convenience functions is available.

(DM-2343, DM-2363, DM-2436)

SeparableXYTransform and Functor added to afw::geom; subclass added to obs_lsstSim to model edge rolloff effects

A subclass of XYTransform was added to afw::geom to enable computation of inverses and derivatives.  This is used to model edge rolloff effects in obs_lsstSim.

Documentation is available for SeparableXYTransform and Functor.

(DM-2033)

Added include method for Footprints; used in meas_deblender

This method allows a Footprint to be merged with its contiguous neighbors.  It solved a problem with artifacts in noise-replaced images.

Documentation on the method is available.

(DM-1738)

Optimized grow operations on Footprints

The previous code used a slow algorithm; this has been updated with a much better one based on the Kim 2005 paper cited in the issue, particularly for isotropic grows.

(DM-1128)

Allow calling Catalog.extend with a SchemaMapper as second argument

The second argument is normally the deep flag, but it is common to want to use a SchemaMapper, so this syntax was enabled (in addition to previous support for passing the SchemaMapper via the mapper keyword argument). 

(DM-1514)

Add ability to stringify a lsst.afw.cameraGeom.CameraPoint

C++ and Python methods were added to allow text output of this object.

(DM-351)

Inactivity timeouts when receiving ctrl_events now disabled

When a message takes a long time to be delivered, the system was sometimes timing out before the receiving process wanted.

(DM-1185)

Provided getElement and setElement for CovarianceMatrixKey in afw.table

When CovarianceMatrixKey was added, we neglected to add methods to allow direct access to elements of the covariance matrix, instead requiring users to get or set the full matrix in order to retrieve individual elements.  These new methods were added to allow direct element access, which was necessary to support astrometry code being backported from the HSC fork..

(DM-1671)

Arithmetic operators that mix integer and floating-point Point and Extent

Most arithmetic operators between integer and floating point coordinate objects now work, producing floating point outputs.  More see the more complete documentation for the full list of supported operators and explanations for the operators that are not supported.

(DM-1197)

Bug Fixes

The following fixes resolve problems visible to end users.

afw background.py test failed on some platforms

An approximate comparison was needed instead of an exact one.

(DM-1374)

ExtentI division fixed

When Python 3-compatible division is used, ExtentI caused unexpected exceptions.  This has been fixed.

Enabled writing uint64 for display via ds9

This used to throw an exception.

(DM-1537)

FITS HISTORY and COMMENT cards were not being read

They are now read into the metadata PropertyList.

(DM-1651)

Extending a Catalog had the wrong default for the deep argument

This sometimes caused exceptions.

(DM-1710, DM-1083)

Fixed segmentation fault in FootprintFunctor

A reference data member with an incorrect lifetime was causing invalid memory accesses, particularly when the code was built with gcc 4.8.

(DM-1793)

Fixed segmentation fault when writing FITS headers with "."

A problem in PropertySet caused this.

(DM-882)

Fixed equality/inequality semantics of Coords

Sometimes Coords that were equal were also unequal.

(DM-2347)

Fail gracefully when Psf type is unknown when reading an Exposure

Previously, the failure was confusing because outdated exception state was being kept.

(DM-2435)

Throw an exception for overflowingly large Images

We now check at Image construction time.

(DM-527)

Avoid infinite loops when a CameraMapper outputRoot is the same as the root

We now avoid creating a looping _parent symlink.

(DM-1513)

Avoid error when printing command-line task help

While the help message was printed, the error could be confusing.

(DM-383)

Avoid exception when command-line task fails with doRaise=False and doReturnResults=True

Now, the task will returna Struct containing a result of None instead.

(DM-1558, DM-1559)

Avoid exception when meas.photocal.colorterms.transformMags is called when color terms do not exist

The cached primary set is now used, as expected.

(DM-2170)

Process ids greater than 65535 don't crash ctrl_events

It used to be assumed that pids fit in 16 bits.

(DM-1351)

Decided that empty defects files should be required for defect-free CCDs

Missing defects files continue to cause an error.

(DM-1699)

Build and Code Improvements

These improvements should not usually be visible to end users. They may be important for developers, however.

Adoption of C++11

The entire stack is now built with C++11 features enabled in the C++ compiler, and the code has been modified to begin to use these.  Since our default compiler is still gcc 4.4, the number of available features is still quite limited.  Incompatibilities with C++11 have been fixed.  No interface changes have yet occurred, but they should be expected in the future.

(DM-1209, DM-1302, DM-1371, DM-1361, DM-765, DM-1486)

Adoption of SWIG 3

The Stack now relies on SWIG 3 for making C++ interfaces available to Python.  Some old-style SWIG code using "{" and "}" was updated to use "%{" and "%}".

(DM-768, DM-2321)

obs_test camera and data update

The camera description and data in the obs_test package was updated to make it useful for Task-level tests.  The data used is half of each of the 16 amplifiers from a single simulated LSST CCD.  This update was then used to create a test case for the processCcd Task.

(DM-1291, DM-1292, DM-1471, DM-1293, DM-1115)

Preset RNG seeds and unique output files for tests

Preset seeds were implemented for random number generators used in tests, making the results more reproducible and avoiding rare cases where results were out of tolerance.  Cases in which multiple tests were writing to the same file or the same event topic, causing problems if they were run in parallel, have also been fixed.

(DM-1019, DM-1439, DM-1202)

Removal of sqlite package

This package used to be included when we used older versions of Python.  It is no longer necessary and can conflict with the built-in version on MacOS X.  It has been removed from the Stack as an external dependency.

(DM-2428)

Table files use envPrepend

All eups table files now use envPrepend to ensure that their changes to path environment variables take precedence over any pre-existing values.

(DM-868)

Build system fixes

The lsstsw package is now more robust on MacOS X.  In particular, it installs the Mac version of anaconda and properly detects whether unit tests passed.

(DM-1219)

lsst_build was patched to take into account changes to eups related to the ShellShock vulnerability.

(DM-1367)

Renamed exceptions

ErrorExceptions in pex_exceptions were renamed to just Error.

(DM-764)

Improve translation of C++ exceptions to Python

Custom LSST exceptions thrown in C++ are now converted directly to their Python counterparts, and the old LsstCppException class, which used to represent all LSST C++ exceptions in Python, has been removed.

(DM-827)

Code cleanup

Unsafe using namespace statements in pex_policy were removed.

(DM-2386)

Multiple-include guards in various places were made consistent with the LSST coding standards.

(DM-1406)

Unused local typedefs that caused warnings with newer compilers were removed.

(DM-1802)

The obsolete afw/coord/Utils.h file was removed.

(DM-2332)

JSON is now used for marshaling in ctrl_events.

(DM-1349)

References to pex_harness have been removed from ctrl_orca.

(No issue number)