These draft notes cover the major updates made by Data Release Production to the LSST stack since release 11.0 (Summer 2015). Please record significant updates here so that they can ultimately be incorporated into the notes accompanying the Winter 2016 release.
Major Functionality and Interface Changes
Dumping task configuration parameters now includes documentation
That is, running ${TASK} ... --show config displays not only the names and values of the configuration but also associated documentation.
DM-3811 - Getting issue details... STATUS
Cleanup interpolation tasks and implement useFallbackValueAtEdge
The interpImageTask in pipe_tasks has been refactored to implement a single run function for interpolation over a list of defects in an image. This run function will accept an afw image type of either MaskedImage or Exposure. A defects list can be passed in directly OR the name of mask plane can be passed from which a defects list will be created. If a PSF is attached to the image, it will be used as the (required) argument for the
function in interpolateoverDefects
meas_algorithms used for the interpolation. Otherwise a FWHM (in pixels) can be provided or the defaultFWHM value in meas_algorithms' GaussianFactory is used. Note that while the PSF is a required argument for meas_algorithms' interpolateOverDefects, function it is currently not being used, so it is not necessary to pass in an accurate PSF.
The useFallbackValueAtEdge option is now implemented. This tapers the interpolation to a fallbackValue towards the image edge. The fallbackValue can be set via config parameters to be either computed as a statistical representation of the image data (MEAN, MEDIAN, or MEANCLIP) or set by providing a specific value. Allowance for a negative fallbackValue is also controlled though a config parameter.
DM-3677 - Getting issue details... STATUS
HSC backport: avoid I/O race conditions config write out
This adds functionality to avoid potential I/O race conditions when running multiple simultaneous process. This is accomplished by writing to temporary files and then renaming them to the correct destination filename in a single operation. Also, to avoid similar race conditions in the backup file creation (e.g. config.py~1, config.py~2, ...), a --no-backup-config option (to be used with --clobber-config) is added here to prevent the backup copies being made. The outcome for this option is that the config that are still recorded are for the most recent run.
DM-3911 - Getting issue details... STATUS
HSC backport: Introduce SafeClipAssembleCoaddTask which extends AssembleCoaddTask to make clipping safer
SafeClipAssembleCoaddTask does the following,
- build both clipped and unclipped coadds and difference them first
- detect on the difference
- identify difference footprints that overlap appreciably with only one input image
- use identified difference footprints as a mask into a final coadd
- set a "clipped" bit on the final coadd for any pixel that did not include all input frames within the valid polygons.
To support the new coadd task, the pixel flags measurement plugin has been modified to accept two new configuration parameters. These new parameters, named masksFpCenter and masksFpAnywhere, each accept a list of mask planes. When the pixel flags measurement plugin searches for mask planes to set corresponding flags, it will now additionally search the user supplied mask plane lists. The masksFpCenter parameter specifies mask planes that, if found within the center of a footprint, will have a corresponding pixel flag set. The masksFpCenter parameter specifies mask planes that, if found anywhere in a footprint, will have a corresponding pixel flag set. The masksFpAnywhere parameter now has it's defaults set within the stack which specify the clipped mask plane created by SafeClipAssembleCoaddTask.
SafeClipAssebleCoaddTask is now the default method for building a coadd within the LSST stack. The assembleCoadd.py CommandLineTask now supports the --legacy command line flag which will allow the original AssembleCoaddTask to be run. If the legacy task is run, the clipped mask plane must be removed from the masksFpAnywhere configuration parameter.
DM-2915 - Getting issue details... STATUS
Preliminaries for the LSST vs. HSC pipeline comparison through single-frame processing
This involved a detailed audit of all the functionality currently used in HSC single-frame processing to ensure all relevant features have been pulled over to the LSST stack such that the comparisons to be done in DM-2984 will be meaningful and informative. It led to the porting of a number of recent HSC tickets and and standalone commits. For the most part they are updates to add support for narrow band filters now (or soon-to-be) in regular use in HSC observations, update CCD defect lists, optimizations (to decrease processing time) to default settings in deblender flags, and to ward against potential failure cases (e.g. operating on zero-length arrays, taking log10 of zero).
DM-3942 - Getting issue details... STATUS
Define polygon bounds for CCDs based on vignetted regions
This adds a function in ip_isr's isrTask.py to set a "valid polygon" for a given CCD exposure as the intersection of a polygon defined in focal plane coordinates and the CCD corners. It is currently being used in obs_subaru's isr.py to set the polygon bounds (added in DM-2981) for a CCD exposure to include the non-vignetted regions. The settings for the vignetted region is in a separate config file so that it can be used in different places in the code.
DM-3259 - Getting issue details... STATUS
Bug Fixes
The following fixes resolve problems visible to end users.
Correct bad default minInitialRadius for CModel
The minInitialRadius configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classifications.
DM-3821 - Getting issue details... STATUS
Correct algebraic error in CModel uncertainty calculation
There was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude.
DM-3821 - Getting issue details... STATUS
NaiveDipoleCentroid and NaiveDipoleFlux algorithms no longer require a centroid slot
Previously, initializing these algorithms was only possible if a centroid was already defined. That was not only unneccessary, but also made them more complicated to use, particularly in testing.
DM-3940 - Getting issue details... STATUS
Update (some) example code to run with recent stack versions
Changes in afw::table had broken examples/calibrateTask.py in pipe_tasks. It has now been updated to comply with the latest afw::table API.
DM-4125 - Getting issue details... STATUS
Fix a failure to appropriately log failed task execution
When task execution fails, we add a message to the log (with level FATAL). In some cases, the very act of attempting to log this message could throw an exception, and information about the original error was lost. This has now been resolved.
DM-4218 - Getting issue details... STATUS
Build and code improvements
These improvements should not usually be visible to end users. They may, however, be important for developers.
Work-around incompatibilities with NumPy 1.10
NumPy 1.10 introduced API changes which were incompatible with existing usage in the stack. The latter has been updated to match
DM-4063 - Getting issue details... STATUS DM-4071 - Getting issue details... STATUS DM-4238 - Getting issue details... STATUS
When building boost warn user if user-config.jam or site-config.jam exists
Building boost can fail if a user-config.jam or site-config.jam exist and have options which conflict with the lsst build configuration process. Introduce a warning message if either of these files are found to notify the user.
DM-4198 - Getting issue details... STATUS