These draft notes cover the major updates made by Data Release Production to the LSST stack since release 11.0 (Summer 2015). Please record significant updates here so that they can ultimately be incorporated into the notes accompanying the Winter 2016 release.
That is, running ${TASK} ... --show config
displays not only the names and values of the configuration but also associated documentation.
The interpImageTask
in pipe_tasks
has been refactored to implement a single run
function for interpolation over a list of defects in an image. This run function will accept an afw image type of either MaskedImage or Exposure. A defects list can be passed in directly OR the name of mask plane can be passed from which a defects list will be created. If a PSF is attached to the image, it will be used as the (required) argument for the
function in interpolateoverDefects
meas_algorithms
used for the interpolation. Otherwise a FWHM (in pixels) can be provided or the defaultFWHM value in meas_algorithms
' GaussianFactory
is used. Note that while the PSF is a required argument for meas_algorithms'
interpolateOverDefects
, function it is currently not being used, so it is not necessary to pass in an accurate PSF.
The useFallbackValueAtEdge option is now implemented. This tapers the interpolation to a fallbackValue towards the image edge. The fallbackValue can be set via config parameters to be either computed as a statistical representation of the image data (MEAN, MEDIAN, or MEANCLIP) or set by providing a specific value. Allowance for a negative fallbackValue is also controlled though a config parameter.
This adds functionality to avoid potential I/O race conditions when running multiple simultaneous process. This is accomplished by writing to temporary files and then renaming them to the correct destination filename in a single operation. Also, to avoid similar race conditions in the backup file creation (e.g. config.py~1, config.py~2, ...), a --no-backup-config option (to be used with --clobber-config) is added here to prevent the backup copies being made. The outcome for this option is that the config that are still recorded are for the most recent run.
SafeClipAssembleCoaddTask does the following,
To support the new coadd task, the pixel flags measurement plugin has been modified to accept two new configuration parameters. These new parameters, named masksFpCenter and masksFpAnywhere, each accept a list of mask planes. When the pixel flags measurement plugin searches for mask planes to set corresponding flags, it will now additionally search the user supplied mask plane lists. The masksFpCenter parameter specifies mask planes that, if found within the center of a footprint, will have a corresponding pixel flag set. The masksFpCenter parameter specifies mask planes that, if found anywhere in a footprint, will have a corresponding pixel flag set. The masksFpAnywhere parameter now has it's defaults set within the stack which specify the clipped mask plane created by SafeClipAssembleCoaddTask.
SafeClipAssebleCoaddTask is now the default method for building a coadd within the LSST stack. The assembleCoadd.py CommandLineTask now supports the --legacy command line flag which will allow the original AssembleCoaddTask to be run. If the legacy task is run, the clipped mask plane must be removed from the masksFpAnywhere configuration parameter.
This adds functionality that allows for the reservation of a fraction of the PSF Candidates from PSF fitting. The reserved fraction can then be used to test for over-fitting, do cross-validation etc. To support this functionality, the run methods of CalibrateTask
and MeasurePsfTask
now accept the additional keyword argument expId
. Dummy versions of this keyword were added to the SdssCalibrateTask
and CfhtCalibrateTask
to maintain a consistent API.
Configuration defaults and metadata through the LSST single-processing pipeline have been updated to match the current best practice established on HSC. Major changes include:
This adds a function in ip_isr's isrTask.py to set a "valid polygon" for a given CCD exposure as the intersection of a polygon defined in focal plane coordinates and the CCD corners. It is currently being used in obs_subaru's isr.py to set the polygon bounds (added in DM-2981) for a CCD exposure to include the non-vignetted regions. The settings for the vignetted region is in a separate config file so that it can be used in different places in the code.
This new command line option streamlines the process of specifying input and output repositories for command line tasks. In its simplest form, the --rerun
option is a shorthand for specifying an output repository: output is written to a location relative to the input. Thus
CmdLineTask /path/to/inputroot --rerun useroutput |
is equivalent to
CmdLineTask /path/to/inputroot --output /path/to/inputroot/rerun/useroutput |
Often, one task will process the output of a previous rerun. For this situation, --rerun
provides a two-valued form which specifies relative locations of both input and output. In this mode,
CmdLineTask /path/to/inputroot --rerun process1:useroutput |
is equivalent to
CmdLineTask /path/to/inputroot --input /path/to/inputroot/rerun/process1 --output /path/to/inputroot/rerun/useroutput |
A new boolean option (doFakes) and a retargetable task (fakes) have been introduced into the ProcessCcdTask. This (along with a new class called BaseFakeSourcesTask) sets up a frame work that others may use to introduce known fake sources into the data processing stream. However, this framework itself does not actually insert any fake data itself, but provides an interface others may use to define their own fake source injection task. To implement a fake injection task one must create a child class of BaseFakeSourcesTask (located in lsst.pipe.tasks.fakes) and overload the run method to do the work of injecting the sources. Each source that is injected should have a corresponding bit set in the FAKE maskplane which can be accessed with the self.bitmask variable. Once a task has been created, the config field fakes in processCcd must be retarged to point to the user created task. Additionally, the task will not be run unless the doFakes configuration option in processCcd is set to True. However, if this variable is set to True, and the task is not retarged processCcd will fail.
Added two new data id containers:
PerTractCcdDataIdContainer
: determines the set of tracts each visit touches and adds a data reference with those tracts for each of the visit components. This allows for the user to run a command line task forcedPhotCcd.py
for a given visit without having to know which tracts overlap the visit. [Note this will also be used by meas_mosaic
if/when it gets incorporated into the LSST stack.]TractDataIdContainer
: generates a list of data references for patches within a given tract (effectively a "data reference list" that points to the entire tract). [Note that, at the time of writing, this is only being used by a qa analysis script currently under development.]Added the ability to warp images using a transformation defined by an lsst::afw::geom::XYTransform
Added method getCoordSystem to lsst::afw::coord::Coord. Also added UNKNOWN=-1 as a new CoordSystem enum (the existing enums retain their existing value).
The match lists created when performing image calibration (astrometry and photometry) are normalized (i.e. stripped down to a list of the matched reference and source ids and their distance) prior to being persisted. The ability to denormalize a match catalog is very useful (for post qa analysis, for example). This can now be done using the joinMatchListWithCatalog()
function in meas_algorithms
's LoadReferenceObjectsTask. It has been moved from meas_astrom
's ANetBasicAstrometryTask so that it can be easily accessed (requiring only that a reference object loader be initiated) and to allow it to work with any kind of reference catalog (i.e. other than a_net
).
The skymap
package now contains the script showVisitSkyMap.py
which provides a convenient way of visualizing the tracts, patches and CCDs contained in a set of visits.
Functions have been added to lsst::afw::catalogMatches to provide the ability to convert a match list into a catalog and vice versa (this can be useful for post-processing analyses: qa analysis, for example).
The SingleFrameFPPositionPlugin
measurement plugin, available in meas_base
, records the positions of source centroids in focal plane coordinates (which may be convenient for plotting). This plugin is not enabled by default, but may be switched on by requesting base_FPPosition
in measurement configuration.
The SingleFrameJacobianPlugin
calculates the ratio between the nominal Jacobian determinant at the source centroid (as determined by a user-specified pixel scale) and the actual Jacobian determinant as derived from the astrometric solution. This plugin is not enabled by default, but may be switched on by requesting base_Jacobian
in measurement configuration.
The following fixes resolve problems visible to end users.
LTVn
headers as floating point numbersWhen persisting to a FITS file, these header cards were previously, incorrectly, stored as integers.
If two separate footprints from the same catalog are merged due to an existing merged object which overlaps both of them the flags of which peaks are being detected were not being propagated. This issue caused apparent dropouts of sources and has now been fixed.
getChildren
method of SourceCatalog
may return the wrong informationThe getChildren
method requires that the result must be sorted by parent. This is naturally the case when the catalog is produced by detection or deblending tasks. However, if multiple catalogs are concatenated together this condition may no longer be true. The getChildren
method was updated to raise an exception if the precondition of sorting is not met.
Warping assumed that the the sky representation of both WCS was identical.
minInitialRadius
for CModelThe minInitialRadius configuration parameter had a default that is too small, causing many galaxies to be fit with point source models, leading to bad star/galaxy classifications.
There was a simple but important algebra error in the uncertainty calculation, making the uncertainty a strong function of magnitude.
NaiveDipoleCentroid
and NaiveDipoleFlux
algorithms no longer require a centroid slotPreviously, initializing these algorithms was only possible if a centroid was already defined. That was not only unneccessary, but also made them more complicated to use, particularly in testing.
Changes in afw::table
had broken examples/calibrateTask.py
in pipe_tasks
. It has now been updated to comply with the latest afw::table
API.
When task execution fails, we add a message to the log (with level FATAL
). In some cases, the very act of attempting to log this message could throw an exception, and information about the original error was lost. This has now been resolved.
Add functions to return patches and tracts which contain given coordinates, i.e. conversions between celestial coordinates and tract,patch indicies. Functions include findClosestTractPatchList, findAllTract, and findTractPatchList which finds the closets tract and patch that overlaps coordinates, finds all tracts which include the specified coordinate, and finds tracts and patches that overlap a region respectively.
Various bug fixes to the shapeHSM measurement algorithm were ported and include:
These improvements should not usually be visible to end users. They may, however, be important for developers.
NumPy 1.10 introduced API changes which were incompatible with existing usage in the stack. The latter has been updated to match
Building boost can fail if a user-config.jam or site-config.jam exist and have options which conflict with the lsst build configuration process. Introduce a warning message if either of these files are found to notify the user.
Task.display()
methodThis method has been deprecated since release 9.2 (S14). It has been removed from the codebase, and all stack code updated to directly interface with afw.display
or to use helper functions defined in meas_astrom
.
Mask
s to DefectList
sThe previous version of routine was extremely memory intensive when large numbers of pixels were masked.