These draft notes cover the major updates made by Data Release Production to the LSST stack since release 12 (Winter/Extra 2016). Please add material here which should ultimately be incorporated into the notes accompanying the Fall 2016 release.
Major Functionality and Interface Changes
Add a measurement extension which performs Kron photometry
The KronFluxAlgorithm measures the flux in an elliptical aperture as defined by Kron (1980). The plugin is not enabled by default, but may be switched on by requesting ext_photometryKron_KronFlux in measurement configuration.
- DM-3440Getting issue details... STATUS - DM-6561Getting issue details... STATUS
Package version tracking and checking ("stack provenance")
A package version tracking and checking capability has been included in the stack and integrated with the ``CmdLineTask`` system. This keeps track of what versions of packages are being used in order to prevent the user from unwittingly processing data with multiple, subtly different, versions of the code. This provides a provisional implementation of a provenance tracking
system: it is essential for short-term use, but will be rendered obsolete before we move to operations. A more detailed description of this system is available on the LSST Community Forum.
- DM-3372Getting issue details... STATUS
ngmix-based measurement
Erin Sheldon's ngmix has been integrated with the LSST codebase: it can be installed as package ngmix, and meas_extensions_ngmix package provides an interface between it and the LSST measurement framework. At time of writing, these packages are purely optional, and have not been included in any of the standard Science Pipelines top-level packages (lsst_apps, lsst_distrib). Currently, we provide an ngmix-based plugins for approximating the PSF and for fitting sources using a mixture of Gaussians in both single frame and forced measurement.
- DM-5429Getting issue details... STATUS - DM-6123Getting issue details... STATUS - DM-6124Getting issue details... STATUS - DM-6125Getting issue details... STATUS - DM-2253Getting issue details... STATUS - DM-5867Getting issue details... STATUS - DM-5432Getting issue details... STATUS - DM-6719Getting issue details... STATUS - DM-6724Getting issue details... STATUS - DM-6300Getting issue details... STATUS - DM-6398Getting issue details... STATUS - DM-6788Getting issue details... STATUS
Multiband coadd processing now saves metadata
Metadata describing the operations of algorithms within the multiband coadd processing framework is now correctly persisted along with their outputs.
- DM-4991Getting issue details... STATUS
"Mosaic" calibration
The meas_mosaic package for performing simultaneous astrometric calibration across multiple overlapping exposures has been ported from Hyper Suprime-Cam. While this is an essential capability for LSST, we do not expect meas_mosaic itself to survive to operations: it is slated for replacement by Jointcal. However, meas_mosaic provides an essential point of comparison for Jointcal, and is a key capability until Jointcal is fully available.
- DM-2674Getting issue details... STATUS
Disabled oversampling in PSFEx
Experiments with Hyper Suprime-Cam have established that oversampling does not work well in PSFEx. It has therefore been disabled in the default configuration used for LSST.
- DM-6982Getting issue details... STATUS
Add support for the HSC I2 filter
This filter is now defined in the obs_subaru camera package.
- DM-7045Getting issue details... STATUS
Multiband coadd processing does not select flagged bands as reference
The algorithm used to select the reference band in multi-band coadd processing has been updated to avoid bands with certain user-specified flags set. By default, these flags include the source having an interpolated centre, or errors detected with PSF, Kron or CModel fluxes.
- DM-7044Getting issue details... STATUS
Renamed the "afterburner" system to CatalogCalculation
The "afterburner" system the stack in the previous (v12) release; it provides a capability to run algorithms akin to our regular measurements, but which only take inputs from catalogs (i.e. not pixel data). To avoid ambiguity, this system has been renamed CatalogCalculation
.
- DM-6919Getting issue details... STATUS - DM-7421Getting issue details... STATUS
Centroid position checker
Centroid finding measurement algorithms were updated to ensure that the centroids they find fall within the Footprint
being centroided. If not, the returned value is forced to correspond to the position of the most significant peak within the Footprint
, and a flag is set to indicate the error.
- DM-4926Getting issue details... STATUS
Added ability to run ctrl_pool processes without a batch system
The option --batch-type none
(or None
) runs the command in your process as normal. This means that e.g. the pipe_drivers calibration scripts need not rely on MPI.
- DM-7900Getting issue details... STATUS
HSC R2 filter
The obs_subaru package now includes support for the R2 filter on Hyper Suprime-Cam.
- DM-7510Getting issue details... STATUS
Added DirectMatchTask
The new DirectMatchTask
matches sources against a reference catalog without performing any rotation, offset or WCS fitting. This provides a simpler and less error-prone option for matching sources for quality assessment. The MeasureMergedCoaddSourcesTask
has been modified to make use of this new DirectMatchTask
.
- DM-7117Getting issue details... STATUS
Bug Fixes
Corrected configuration dictionary comparisons
Resolved a bug whereby comparison of (intrinsically unordered) configuration dictionaries incorrectly compared their ordering rather than just their contents. This lead to unexpected failure of re-runs due to incorrectly identified “mismatched configurations”.
- DM-6661Getting issue details... STATUS
Correct CModel table schema units
Units for the time field changed from seconds to second to comply with the Astropy convention.
- DM-6282Getting issue details... STATUS
Restore capability of calculating temporary local backgrounds in source detection
Our SourceDetectionTask
provides the optional capability of fitting and temporarily removing a local background before detecting sources. This capability was accidentally disabled in the X/W16 release (v12); it has now been restored. It is disabled by default.
- DM-5323Getting issue details... STATUS
Resolve a memory error in convolution
A memory handling error could cause our convolution code to generate incorrect results on some platforms (problems observed on OS X when the code is built without optimization; no errors seen on Linux, or when building with -O2
or -O3
on OS X). This appears to originate in the Boost Generic Image Library, an external dependency of the LSST stack. We have been unable to isolate the problem in Boost, but have worked around it for our particular use case.
- DM-5822Getting issue details... STATUS
Patched GalSim to raise an exception in some error conditions
On certain error conditions, GalSim, a third party package used by the LSST stack, generates an assertion failure which is hard for our code to handle appropriately. LSST's version of GalSim has been patched to raise a more appropriate exception instead. This change has been submitted for inclusion in GalSim itself.
- DM-7114Getting issue details... STATUS
Ensure star selector results are reproducible
Resolved an issue whereby the star selection could change depending on the number of CPU cores in use.
- DM-7040Getting issue details... STATUS
Single frame measurement exploits multiple CPU cores
Resolved an issue whereby the SingleFrameDriverTask
failed to use more than one CPU core regardless of configuration.
- DM-7134Getting issue details... STATUS
Fix typo in CoaddSrcTransformTask
A typo which rendered CoaddSrcTransformTask inoperable was resolved. In the process, the tests for this task were significantly upgraded to avoid a repeat of this, or other, problems.
- DM-6635Getting issue details... STATUS
Do not write duplicate AR_NAME
headers when persisting data
When persisting objects to FITS, we were incorrectly writing duplicate AR_NAME
headers. This has been resolved.
- DM-7221Getting issue details... STATUS
Be more rigorous about units when persisting angles
We now correctly write TUNIT
headers describing the units used when storing angles, and are more careful about checking units when reading them.
- DM-7221Getting issue details... STATUS
Infinite recursion in afw table
A bug has been discovered in the afw table library which could lead to infinite recursion. While the ultimate cause of the bug is still under investigation, a temporary workaround has been put in place to mitigate its impact.
- DM-7377Getting issue details... STATUS - DM-7385Getting issue details... STATUS
Fixed a division by zero error in measurement of aperture corrections
Attempting to calculate an aperture correction for a measurement of zero flux would previously result in an unhandled division by zero.
- DM-7180Getting issue details... STATUS
Fix dataset lookup in multibandDriver.py
The multiband driver script was previously checking for the existence of the pre-calexp version of the coadd. This is not guaranteed to exist if the coadds were built using the driver scripts (where there is no need to persist both the pre-calexp and calexp versions) and it is only the calexp coadds that are used in the multiband tasks, so it is the calexp version whose existence should be checked for in multibandDriver.py.
- DM-7716Getting issue details... STATUS
Build & Code Improvements
Migration from Boost to standard library equivalents
Continuing work started in the previous release, we have audited the stack for use of the Boost libraries. Where possible, Boost has been replaced with C++11 standard library equivalents.
- DM-5880Getting issue details... STATUS - DM-6089Getting issue details... STATUS - DM-6090Getting issue details... STATUS - DM-6091Getting issue details... STATUS - DM-6092Getting issue details... STATUS - DM-6093Getting issue details... STATUS - DM-6094Getting issue details... STATUS - DM-6095Getting issue details... STATUS - DM-6096Getting issue details... STATUS - DM-6097Getting issue details... STATUS - DM-6325Getting issue details... STATUS
Properly log warnings in ObjectSizeStarSelector
Previously, non-fatal NumPy warnings of the form RuntimeWarning: invalid value encountered in less could appear on the console, but not be caught and logged anywhere. It was therefore easy to miss problems arising when running the algorithm. These warnings are now ingested into the LSST logging system, and are hence recorded properly.
- DM-5428Getting issue details... STATUS
Unit tests for colorterm framework
A set of unit tests have been written which properly exercise the colorterm framework which was adapted from Hyper Suprime-Cam in the S15 cycle.
- DM-2837Getting issue details... STATUS
FlagHandler accessible from Python
The FlagHandler system for efficiently and safely setting flags corresponding to measurement algorithm failures on source records is now accessible from Python. Previously, it was only available for use in C++ code.
- DM-4009Getting issue details... STATUS
Exact image comparison utility code
The assertImagesEqual
and assertMaskedImagesEqual
methods have been added to added to LSST's test cases. These provide explicit mechanisms for checking for exact equality of image data.
- DM-6641Getting issue details... STATUS
Add integration test for stellaricity of sources used to measure PSF
Added a test to the ci_hsc package to monitor the fraction of sources used to model the PSF in when processing a CCD image and ensure that the vast majority of them are ultimately classified as stars. This provides an automated check that we are not inadvertantly estimating the PSF based on extended objects.
- DM-6051Getting issue details... STATUS
Use CatalogCalculation to handle aperture corrections
The CatalogCalculation facility was added to the stack in the previous (v12) release (where it was referred to as "afterburners"); it provides a capability to run algorithms akin to our regular measurements, but which only take inputs from catalogs (i.e. not pixel data). This has now been used to clean up the logic used to apply aperture corrections within our regular data processing flow. This old method for handling aperture corrections was a persistent source of bugs and confusion: these should be avoided by the new approach.
- DM-5877Getting issue details... STATUS
Builds are now optimized by default
Following an RFC, sconsUtils was updated to enable optimization by default.
- RFC-202Getting issue details... STATUS - DM-7010Getting issue details... STATUS
Fixed a build failure of mpi4py on some versions of Mac OS X
Added extra logic to ensure we target a version of the OS X SDK which is compatible both with the version of Mac OS installed and the Python interpreter in use.
- DM-6133Getting issue details... STATUS
Centralised Butler datasets
Butler dataset definitions which are common to multiple cameras have been consolidated in a central location in daf_butlerUtils rather than being repeated in each camera package.
- DM-6858Getting issue details... STATUS
Order in which SConscript files are read is configurable
The sconsUtils build system now reads SConscript
files, which describe the build, in a user configurable order. This makes it possible to specify that some components of a package must be built before others.
- DM-7179Getting issue details... STATUS
Ndarray is now an external package
The ndarray package in the LSST stack now installs vanilla upstream version 1.3.0.
DM-2005, DM-7519