These draft notes cover the major updates made by Data Release Production to the LSST stack since release 11.0 (Summer 2015). Please record significant updates here so that they can ultimately be incorporated into the notes accompanying the Winter 2016 release.
Major Functionality and Interface Changes
Dumping task configuration parameters now includes documentation
That is, running ${TASK} ... --show config displays not only the names and values of the configuration but also associated documentation.
DM-3811 - Getting issue details... STATUS
Cleanup interpolation tasks and implement useFallbackValueAtEdge
The interpImageTask in pipe_tasks has been refactored to implement a single run function for interpolation over a list of defects in an image. This run function will accept an afw image type of either MaskedImage or Exposure. A defects list can be passed in directly OR the name of mask plane can be passed from which a defects list will be created. If a PSF is attached to the image, it will be used as the (required) argument for the
function in interpolateoverDefects
meas_algorithms used for the interpolation. Otherwise a FWHM (in pixels) can be provided or the defaultFWHM value in meas_algorithms' GaussianFactory is used. Note that while the PSF is a required argument for meas_algorithms' interpolateOverDefects, function it is currently not being used, so it is not necessary to pass in an accurate PSF.
The useFallbackValueAtEdge option is now implemented. This tapers the interpolation to a fallbackValue towards the image edge. The fallbackValue can be set via config parameters to be either computed as a statistical representation of the image data (MEAN, MEDIAN, or MEANCLIP) or set by providing a specific value. Allowance for a negative fallbackValue is also controlled though a config parameter.
DM-3677 - Getting issue details... STATUS
HSC backport: avoid I/O race conditions config write out
This adds functionality to avoid potential I/O race conditions when running multiple simultaneous process. This is accomplished by writing to temporary files and then renaming them to the correct destination filename in a single operation. Also, to avoid similar race conditions in the backup file creation (e.g. config.py~1, config.py~2, ...), a --no-backup-config option (to be used with --clobber-config) is added here to prevent the backup copies being made. The outcome for this option is that the config that are still recorded are for the most recent run.
DM-3911 - Getting issue details... STATUS
HSC backport: Introduce SafeClipAssembleCoaddTask which extends AssembleCoaddTask to make clipping safer
SafeClipAssembleCoaddTask does the following,
- build both clipped and unclipped coadds and difference them first
- detect on the difference
- identify difference footprints that overlap appreciably with only one input image
- use identified difference footprints as a mask into a final coadd
- set a "clipped" bit on the final coadd for any pixel that did not include all input frames within the valid polygons.
To support the new coadd task, the pixel flags measurement plugin has been modified to accept two new configuration parameters. These new parameters, named masksFpCenter and masksFpAnywhere, each accept a list of mask planes. When the pixel flags measurement plugin searches for mask planes to set corresponding flags, it will now additionally search the user supplied mask plane lists. The masksFpCenter parameter specifies mask planes that, if found within the center of a footprint, will have a corresponding pixel flag set. The masksFpCenter parameter specifies mask planes that, if found anywhere in a footprint, will have a corresponding pixel flag set. The masksFpAnywhere parameter now has it's defaults set within the stack which specify the clipped mask plane created by SafeClipAssembleCoaddTask.
SafeClipAssebleCoaddTask is now the default method for building a coadd within the LSST stack. The assembleCoadd.py CommandLineTask now supports the --legacy command line flag which will allow the original AssembleCoaddTask to be run. If the legacy task is run, the clipped mask plane must be removed from the masksFpAnywhere configuration parameter.
DM-2915 - Getting issue details... STATUS DM-4961 - Getting issue details... STATUS
HSC backport: Allow for some fraction of PSF Candidates to be reserved from PSF fitting
This adds functionality that allows for the reservation of a fraction of the PSF Candidates from PSF fitting. The reserved fraction can then be used to test for over-fitting, do cross-validation etc. To support this functionality, the run methods of CalibrateTask and MeasurePsfTask now accept the additional keyword argument expId. Dummy versions of this keyword were added to the SdssCalibrateTask and CfhtCalibrateTask to maintain a consistent API.
DM-3692 - Getting issue details... STATUS
Updated pipeline configuration based on Hyper Suprime-Cam experience
Configuration defaults and metadata through the LSST single-processing pipeline have been updated to match the current best practice established on HSC. Major changes include:
- Support for narrow band filters;
- Updated CCD defect lists;
- Optimized (in terms of CPU time) deblender settings;
- Avoiding failure in certain corner cases (e.g. operating on zero-length arrays, taking log10 of zero).
DM-3942 - Getting issue details... STATUS
Define polygon bounds for CCDs based on vignetted regions
This adds a function in ip_isr's isrTask.py to set a "valid polygon" for a given CCD exposure as the intersection of a polygon defined in focal plane coordinates and the CCD corners. It is currently being used in obs_subaru's isr.py to set the polygon bounds (added in DM-2981) for a CCD exposure to include the non-vignetted regions. The settings for the vignetted region is in a separate config file so that it can be used in different places in the code.
DM-3259 - Getting issue details... STATUS
Introduce --rerun option for command line tasks
This new command line option streamlines the process of specifying input and output repositories for command line tasks. In its simplest form, the --rerun option is a shorthand for specifying an output repository: output is written to a location relative to the input. Thus
CmdLineTask /path/to/inputroot --rerun useroutput
is equivalent to
CmdLineTask /path/to/inputroot --output /path/to/inputroot/rerun/useroutput
Often, one task will process the output of a previous rerun. For this situation, --rerun provides a two-valued form which specifies relative locations of both input and output. In this mode,
CmdLineTask /path/to/inputroot --rerun process1:useroutput
is equivalent to
CmdLineTask /path/to/inputroot --input /path/to/inputroot/rerun/process1 --output /path/to/inputroot/rerun/useroutput
RFC-95 - Getting issue details... STATUS
DM-3371 - Getting issue details... STATUS
Introduce framework for injecting fake sources into data processing
A new boolean option (doFakes) and a retargetable task (fakes) have been introduced into the ProcessCcdTask. This (along with a new class called BaseFakeSourcesTask) sets up a frame work that others may use to introduce known fake sources into the data processing stream. However, this framework itself does not actually insert any fake data itself, but provides an interface others may use to define their own fake source injection task. To implement a fake injection task one must create a child class of BaseFakeSourcesTask (located in lsst.pipe.tasks.fakes) and overload the run method to do the work of injecting the sources. Each source that is injected should have a corresponding bit set in the FAKE maskplane which can be accessed with the self.bitmask variable. Once a task has been created, the config field fakes in processCcd must be retarged to point to the user created task. Additionally, the task will not be run unless the doFakes configuration option in processCcd is set to True. However, if this variable is set to True, and the task is not retarged processCcd will fail.
DM-3380 - Getting issue details... STATUS
Add tract conveniences
Added two new data id containers:
PerTractCcdDataIdContainer: determines the set of tracts each visit touches and adds a data reference with those tracts for each of the visit components. This allows for the user to run a command line taskforcedPhotCcd.pyfor a given visit without having to know which tracts overlap the visit. [Note this will also be used bymeas_mosaicif/when it gets incorporated into the LSST stack.]TractDataIdContainer: generates a list of data references for patches within a given tract (effectively a "data reference list" that points to the entire tract). [Note that, at the time of writing, this is only being used by a qa analysis script currently under development.]
DM-4373 - Getting issue details... STATUS
Warp images based on an XYTransform
Added the ability to warp images using a transformation defined by an lsst::afw::geom::XYTransform
DM-4162 - Getting issue details... STATUS
Add getCoordSystem to Coord and add UNKNOWN CoordSystem enum
Added method getCoordSystem to lsst::afw::coord::Coord. Also added UNKNOWN=-1 as a new CoordSystem enum (the existing enums retain their existing value).
DM-4606 - Getting issue details... STATUS
Adapt joinMatchListWithCatalog to facilitate and simplify denormalizing a match list
The match lists created when performing image calibration (astrometry and photometry) are normalized (i.e. stripped down to a list of the matched reference and source ids and their distance) prior to being persisted. The ability to denormalize a match catalog is very useful (for post qa analysis, for example). This can now be done using the joinMatchListWithCatalog() function in meas_algorithms's LoadReferenceObjectsTask. It has been moved from meas_astrom's ANetBasicAstrometryTask so that it can be easily accessed (requiring only that a reference object loader be initiated) and to allow it to work with any kind of reference catalog (i.e. other than a_net).
DM-3633 - Getting issue details... STATUS
Add a script for visualizing skymaps and CCDs
The skymap package now contains the script showVisitSkyMap.py which provides a convenient way of visualizing the tracts, patches and CCDs contained in a set of visits.
DM-4095 - Getting issue details... STATUS
Add functions to generate "unpacked matches" to and from a catalog
Functions have been added to lsst::afw::catalogMatches to provide the ability to convert a match list into a catalog and vice versa (this can be useful for post-processing analyses: qa analysis, for example).
DM-4729 - Getting issue details... STATUS
Add a measurement algorithm which records the focal plane coordinates of sources
The SingleFrameFPPositionPlugin measurement plugin, available in meas_base, records the positions of source centroids in focal plane coordinates (which may be convenient for plotting). This plugin is not enabled by default, but may be switched on by requesting base_FPPosition in measurement configuration.
DM-4234 - Getting issue details... STATUS
Add a measurement algorithm which records the Jacobian at the positions of sources
The SingleFrameJacobianPlugin calculates the ratio between the nominal Jacobian determinant at the source centroid (as determined by a user-specified pixel scale) and the actual Jacobian determinant as derived from the astrometric solution. This plugin is not enabled by default, but may be switched on by requesting base_Jacobian in measurement configuration.
DM-4234 - Getting issue details... STATUS
Add a measurement algorithm which records the number of input images contributing to a coadd sources
When run on a source detected on a coadd, SingleFrameInputCountPlugin records the number of input images which were stacked to create the coadd at the position corresponding to the source centroid. The plugin is referred to as base_CountInputs, and is enabled by default when performing measurement on coadded images. It is not appropriate to enable this plugin when processing single visit (i.e., not coadded) images.
DM-4235 - Getting issue details... STATUS
Add a measurement algorithm which records the variance at the positions of sources
The SingleFrameVariancePlugin records the median variance in the background around the position of the source being measured. The plugin is referred to as base_Variance and is enabled by default when performing single frame measurement.
DM-4235 - Getting issue details... STATUS DM-5050 - Getting issue details... STATUS
Add a measurement algorithm which records source flux in an aperture scaled to the PSF
The ScaledApertureFluxAlgorithm measures the flux in a circular aperture with radius scaled to some user-specified multiple of the PSF FWHM. This plugin is not enabled by default, but may be switched on by requesting the base_ScaledApertureFlux in measurement configuration.
DM-3257 - Getting issue details... STATUS
Add option to temporarily remove an interpolated background prior to detection
This has the potential for removing a large number of junk detections around bright objects due to noise fluctuations in the elevated local background. The extra subtracted interpolated background is added back in after detection. Currently, the default setting for the config parameter doTempLocalBackround is set to False.
DM-4821 - Getting issue details... STATUS
Add function to average coordinates
Added function lsst.afw.coord.averageCoord, which will return an average coordinate (accounting for spherical geometry) given a list of input coordinates.
DM-4933 - Getting issue details... STATUS
Integrate support for Hyper Suprime-Cam
The obs_subaru camera package, which enables the stack to operate on data taken with the Suprime-Cam and Hyper Suprime-Cam instruments on Subaru, has been modernized, resolving build and test issues and integrating it with LSST's continuous integration system. It will now be included as part of the lsst_distrib release. (Note, though, that usage of Suprime-Cam with the stack is unsupported and unmaintained at present.)
DM-3518 - Getting issue details... STATUS DM-4358 - Getting issue details... STATUS DM-5007 - Getting issue details... STATUS
Restructure PsfFitter algorithm
The PsfFitter routine has been refined to better accord with the meas_base measurement framework. This includes modernized error handling.
DM-4830 - Getting issue details... STATUS
Propagate flags from individual visits to coadds
A task has been added which can propagate flags from individual visit catalogs to coadd catalogs. This is useful, for example, to track which stars in the coadd were used for measuring PSFs on the individual visits.
DM-4878 - Getting issue details... STATUS DM-5084 - Getting issue details... STATUS
Bug Fixes
The following fixes resolve problems visible to end users.
Persist LTVn headers as floating point numbers
When persisting to a FITS file, these header cards were previously, incorrectly, stored as integers.
DM-4133 - Getting issue details... STATUS
Fix bug when identifying existing peaks in a merge
If two separate footprints from the same catalog are merged due to an existing merged object which overlaps both of them the flags of which peaks are being detected were not being propagated. This issue caused apparent dropouts of sources and has now been fixed.
DM-2978 - Getting issue details... STATUS
Fix situation in which the getChildren method of SourceCatalog may return the wrong information
The getChildren method requires that the result must be sorted by parent. This is naturally the case when the catalog is produced by detection or deblending tasks. However, if multiple catalogs are concatenated together this condition may no longer be true. The getChildren method was updated to raise an exception if the precondition of sorting is not met.
DM-2976 - Getting issue details... STATUS
Fix warping when the WCS have different coordinate systems
Warping assumed that the the sky representation of both WCS was identical.
DM-4162 - Getting issue details... STATUS
Correct bad default minInitialRadius for CModel
The minInitialRadius configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classifications.
DM-3821 - Getting issue details... STATUS
Correct algebraic error in CModel uncertainty calculation
There was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude.
DM-3821 - Getting issue details... STATUS
NaiveDipoleCentroid and NaiveDipoleFlux algorithms no longer require a centroid slot
Previously, initializing these algorithms was only possible if a centroid was already defined. That was not only unneccessary, but also made them more complicated to use, particularly in testing.
DM-3940 - Getting issue details... STATUS
Update (some) example code to run with recent stack versions
Changes in afw::table had broken examples/calibrateTask.py in pipe_tasks. It has now been updated to comply with the latest afw::table API.
DM-4125 - Getting issue details... STATUS
Fix a failure to appropriately log failed task execution
When task execution fails, we add a message to the log (with level FATAL). In some cases, the very act of attempting to log this message could throw an exception, and information about the original error was lost. This has now been resolved.
DM-4218 - Getting issue details... STATUS
Updates to Skymap packages
Add functions to return patches and tracts which contain given coordinates, i.e. conversions between celestial coordinates and tract,patch indicies. Functions include findClosestTractPatchList, findAllTract, and findTractPatchList which finds the closets tract and patch that overlaps coordinates, finds all tracts which include the specified coordinate, and finds tracts and patches that overlap a region respectively.
DM-3775 - Getting issue details... STATUS
Port bugfixes in meas_extensions_shapeHSM from HSC to LSST
Various bug fixes to the shapeHSM measurement algorithm were ported and include:
- Ensuring masks have same xy0 as images
- Fixing scaling coefficient in ellipticity error when using distortion
- Provide appropriate mask for variance image when measuring median variance around object
- Change from standard deviation to variance in the sky value when calling GalSim hsm code, as required by GalSim
- Fixes to unit tests to comply with above changes, and the port from the HSC framework to the LSST framework
DM-3384 - Getting issue details... STATUS
Fix variance in coadded images
Warping images in order to coadd them loses variance into covariance. This is mitigated by scaling the variance plane of the coadd. The scaling was being applied incorrectly in some cases. This has now been fixed.
DM-4798 - Getting issue details... STATUS
Fix variance in deblended sources
The deblender incorrectly scaled the variance plane in deblended sources with the fraction of the total flux assigned to the source. This has been corrected.
DM-4845 - Getting issue details... STATUS
Fix logic for applying aperture corrections
This fixes a but whereby the aperture corrections were being applied only after all the measurement plugins had run through, independent of their execution order. This resulted in plugins whose measurements rely on aperture corrected fluxes (i.e. with execution order > APCORR_ORDER) being applied prior to the aperture correction, leading to erroneous results. The only plugin that was affected by this at this time was base_ClassificationExtendedness.
DM-4836 - Getting issue details... STATUS
More uniform support for assigning to catalog columns
Assignment of scalars or NumPy arrays to columns of afw.table.Catalog objects (e.g. "catalog["column"] = value") is now more uniformly supported across types (support was inconsistent before, and never allowed scalar or augmented assignment). Flag columns still do not support column assignment, and Flag column access still returns a copy, not a view, because Flag values are stored internally as individual bits within a larger integer.
DM-4856 - Getting issue details... STATUS
Upgraded WCSLIB to version 5.13
Version 5.13 of WCSLIB resolves memory corruption errors that could crash the stack in some circumstances.
DM-4904 - Getting issue details... STATUS RFC-89 - Getting issue details... STATUS DM-4946 - Getting issue details... STATUS DM-3793 - Getting issue details... STATUS
Fix rotation for instrument signature removal in obs_subaru
Approximately half of the HSC CCDs are rotated 180 deg with respect to the others. Two others have 90 deg rotations and another two have 270 deg rotations (see HSC CCD layout). The raw images for the rotated CCDs thus need to be rotated to match the rotation of their associated calibration frames (in the context of how they have currently been ingested) prior to applying the corrections. This is accomplished by rotating the exposure using the rotated context manager function in obs_subaru's isr.py and the nQuarter specification in the policy file for each CCD. Currently, rotated uses afw's rotateImageBy90 (which apparently rotates in a counter-clockwise direction) to rotated the exposure by 4 - nQuarter turns. This turns out to be the wrong rotation for the odd nQuarter CCDs. This issue fixes this bug, leading to much improved processing of HSC CCD's 100, 101, 102, and 103. Note that, in the future, the ingestion of the calibration data will be updated such that no rotations are necessary (so they will then be removed from obs_subaru accordingly).
DM-4998 - Getting issue details... STATUS
Build and code improvements
These improvements should not usually be visible to end users. They may, however, be important for developers.
Work-around incompatibilities with NumPy 1.10
NumPy 1.10 introduced API changes which were incompatible with existing usage in the stack. The latter has been updated to match
DM-4063 - Getting issue details... STATUS DM-4071 - Getting issue details... STATUS DM-4238 - Getting issue details... STATUS
When building boost warn user if user-config.jam or site-config.jam exists
Building boost can fail if a user-config.jam or site-config.jam exist and have options which conflict with the lsst build configuration process. Introduce a warning message if either of these files are found to notify the user.
DM-4198 - Getting issue details... STATUS
Remove deprecated Task.display() method
This method has been deprecated since release 9.2 (S14). It has been removed from the codebase, and all stack code updated to directly interface with afw.display or to use helper functions defined in meas_astrom.
DM-4428 - Getting issue details... STATUS
Efficiency improvement in converting Masks to DefectLists
The previous version of routine was extremely memory intensive when large numbers of pixels were masked.
DM-4800 - Getting issue details... STATUS
Add a new task parallelization framework
The ctrl_pool package has been added to the LSST stack. This is a high-level parallelization framework used for distributing Task execution across a cluster, based on an MPI process pool. It is based on work carried out on Hyper Suprime-Cam. It is not intended to be the long-term solution to parallelized processing in the LSST stack, but meets our data processing needs until the fully-fledged parallelization middleware is available.
DM-2983 - Getting issue details... STATUS DM-4835 - Getting issue details... STATUS