At the SST F2F meeting on 5 November 2018, we reviewed the contents of the Source and DIASource tables. It was not possible to capture all the decisions about the columns in these tables in the meeting. The tables below contain the contents of the Source and DIASource tables transcribed from the DPDD. Three options were identified for each column, keep as is, keep but move to another (currently unspecified) table, or remove entirely.
We should also consider the possibility of using a Gaia-style approach of storing everything and choosing what to publish
MWV: Overall Comments.
- The SST and the DPDD are about what the user sees and can access as a column. That's separate from how they are stored or generated. The lower-level Science Data Model will benefit from much of these discussions.
- I do think that
trail
* anddip
* columns need to be revisited. Here is a case where implementation is much more closely tied to the column definitions and where accepted practice is much less settled.trail
* More work should be done to come up with a planned model of how to do this photometry and related measurements and then that can be turned into a proposal for what the columns should be fore the DPDDdip
* These columns are the least certain to me. They are largely for characterizing our ability to do image subtraction.
Definition of Time in the Source/Object tables
Robert Lupton to clarify the meaning of time in the object table. (See DM-SST action 2018-11-05 DM SST F2F Agenda and Meeting notes)
RHL: I believe that the object table field in question is {{psRadecTai}}. I propose that we replace this field with an epoch that applies to all measurements in the object table. The epoch could be c. 2024 to coincide with the start of operations, or 2030 to minimise the amount of extrapolation required; we could use a different epoch for different releases, but that would probably be a little confusing.
LPG: Gaia choose this approach also and also has a different reference epoch for each DR. The reference epoch for Gaia DR3 is J2016.0. It is J2015.5 for Gaia DR2 and J2015.0 for Gaia DR1.
RHL: For DIASource table, midPointTai
also needs to be defined. We know the time profile for the shutter at every point in the focal plane from the camera's Hall effect sensors, and I propose that we define the midPointTai to be the mean of this time profile on the assumption that the source does not vary during the exposure. If we take data using snaps, this would be the average of the two half-exposures, and I propose that we do not weight these two measurements by the flux in the two snaps as this would require us to measure fluxes on the two snaps, which is not the current plan, and also change the definition of midPointTai
depending on whether the data is taken with- or without- snaps (and thus potentially depending on the band).
DIASource Table
Column | Description | Retain/Move/Delete? (Robert Lupton ) | Comments |
---|---|---|---|
diaSourceId | Unique source identifier |
| |
ccdVisitId | ID of CCD and visit where this source was measured |
| |
diaObjectId | ID of the DIAObject this source was associated with if any |
| |
ssObjectId | ID of the SSObject this source has been linked to if any |
| RHL: I haven't thought about how to normalise/denormalise these tables, and would expect the ssObjectId to come via a join to diaObject LPG: I thought a DIASource was associated with either a DIAObject or an SSObject, but not both? 2022-03-04: SSSource Table join on diaSourceId MWV: If SSObjects are not DIAObjects (which is my understanding of the current written plan), then both of these columns need to be retained. |
parentDiaSourceId | ID of the parent DIASource this object has been deblended from if any |
| RHL: I'm hoping not to have to deal with blended diaSources, but for completeness we need this. |
midPointTai | Time of mid-exposure for this DIASource. The visit mid-exposure time generally depends on the position of the source relative to the shutter blade motion trajectory. |
| RHL: or move to a lookup based on the known shutter timing. |
radec | Centroid |
| |
radecCov | radec covariance matrix. |
| |
xy | Column and row of the centroid. |
| |
xyCov | Centroid covariance matrix. |
| |
apFlux | the flux difference between the template and the visit image. |
| RHL define radius to be used. Is one enough? |
apFluxErr | Estimated uncertainty of apFlux. |
| |
SNR | The signal-to-noise ratio at which this source was detected in the difference image. |
| RHL this is not that well defined. I'd expect to use the S/N in the PSF flux (correct for point sources such as SNe and asteroids; not for comets), so we don't need this too MWV: Keep. But define what it means. Phase 1 detection threshold? Phase 2 detection threshold? abs(psFlux)/psFluxErr? LPG: The detection threshold will be in provenance and the same for all source detections. The signal-to-noise ratio at which this source was detected is abs(psFlux)/psFluxErr and can be computed from the following 2 columns. We will remove this. ECB: note that the detection threshold is not abs(psFlux)/psFluxErr–detection SNR and measured SNR are different. See DM-31811 |
psFlux | Calibrated flux for point source model. Note this actually measures the flux difference between the template and the visit image. |
| LPG: Change to psfFlux if not done already. |
psFluxErr |
| RHL not present in the DPDD MWV: Agreed. Users will want this very simple quantity, even if it can be extracted from LPG: psfFluxErr if not done already | |
psRadec | Centroid for point source model. |
| RHL We already have raDec MWV: Keep for now, but if implementation makes this redundant with LPG: Potentially confusing - will be the same forpoint sources. |
psCov | Covariance matrix for point source model parameters. |
| RHL In theory we need the full position x flux covariance, but in practice I think that we can separate them (and then they're in already) MWV: I think we should keep, even if some of the first implementations are only diagonal or block diagonal. LPG: psfCov. Not currently computed. Could compute but do not publish |
psLnL | Natural log likelihood of the observed data given the point source model. |
| RHL This was intended to be some sort of S/G number, I suspect, but I expect to use extendedness instead. This covers the next two entries too (psChi2, psNdata) MWV: I agree that LnL and Chi2 need some further thought. It's not clear to me that LnL is what we will want quote – are we including an astrophysical prior on the point source vs. extended source distribution as part of our point source model? ECB: I think all of these are trying to enable model comparison/goodness of fit for the point source vs trail vs dipole models. We should convey that information somehow... |
psChi2 | chi^2 statistic of the model fit. |
| MWV: I would argue to keep. |
psNdata | The number of data points (pixels) used to fit the model. |
| MWV: I'm ambivalent. I can't figure out if this is usefully and consistently defined. Is this useful to understand performance of de-blended objects? I certainly can't really imagine using this column across different visits. |
trailFlux | Calibrated flux for a trailed source model. A Trailed Source Model attempts to fit a (PSF-convolved) model of a point source that was trailed by a certain amount in some direction (taking into account the two-snap nature of the visit |
| RHL We want some sort of trailed number, probably. However, I'm not sure that we need it for every entry in the diaSource table, and I'm not convinced that we need all these numbers. The orientation is given by the second moments (Ixx etc. below), so a flux, fluxError and length is probably enough. So I'm voting for retain, but not to be taken literally. RHL We also need trailFluxErr MWV: I agree. More work needed here to develop a specific plan for trailing measurements and then return and figure out what all of these LPG: All tail* fields need to be reviewed with Mario |
trailRadec | Centroid for trailed source model. |
| RHL use raDec |
trailLength | Maximum likelihood fit of trail length (Note that we'll likely measure trailRow and trailCol) |
| |
trailAngle | Maximum likelihood fit of the angle between the meridian through the centroid and the trail direction (bearing |
| |
trailCov | Covariance matrix of trailed source model parameters. |
| |
trailLnL | Natural log likelihood of the observed data given the trailed source model. |
| |
trailChi2 | chi^2 statistic of the model fit. |
| |
trailNdata | The number of data points (pixels) used to fit the model. |
| |
dipMeanFlux | Maximum likelihood value for the mean absolute flux of the two lobes for a dipole model. A Dipole Model attempts to fit a (PSF-convolved) model of two point sources of opposite signs. |
| RHL Remove all the dipole measurements. We will need some numbers to drive true/bogus measurements LPG: I understood we would move all dipole measurements to another table as they were useful for debugging (Simon Krughoff ?) MWV: I think I agree that the dipole measurements likely should be removed from the DPDD DiaSource Table. This isn't a particularly stable or well-defined quantity. LPG: Remove all dipole measurements frome the DPDD. Check with UW as to whether this is still useful and if it is the afw tables. ECB: are we sure there are no science uses for these? early identification of proper motion stars? |
dipFluxDiff | Maximum likelihood value for the difference of absolute fluxes of the two lobes for a dipole model. |
| |
dipRadec | Centroid for dipole model. |
| |
dipLength | Maximum likelihood value for the lobe separation in dipole model. |
| |
dipAngle | Maximum likelihood fit of the angle between the meridian through the centroid and the dipole direction (bearing |
| |
dipCov | Covariance matrix of dipole model parameters. |
| |
dipLnL | Natural $log$ likelihood of the observed data given the dipole source model. |
| MWV: I think LnL for dipole makes particularly no sense. |
dipChi2 | chi^ statistic of the model fit. |
| |
dipNdata | The number of data points (pixels) used to fit the model. |
| |
totFlux | Calibrated flux for point source model measured on the visit image centered at the centroid measured on the difference image (forced photometry flux) |
| RHL: unmeasurable in crowded fields. I believe that this was added to break degeneracies between dipole separation and amplitude. ECB: No, there's a science use--we need some way to go from difference image flux to total flux for variable stars. Forced photometry on the visit image seems preferable to me to the only alternative I'm aware of, which is figuring out the flux at the position of the DIASource in the template and adding it. The latter requires the same operations, has the same crowding problem, and creates more complications. MWV: This is a doable measurement. It just doesn't necessarily always mean what you might want it to mean. But we should still do and provide it, even though it means something different in crowded fields. LPG: Retain but change name to totPsfFlux |
totFluxErr | Estimated uncertainty of totFlux. |
| LPG: Retain but change name to totPsfFluxErr |
diffFlux | Calibrated flux for point source model centered on radec but measured on the difference of snaps comprising this visit. This flux can be used to detect sources changing on timescales comparable to snap exposure length ~15 |
| LPG: Snaps are still in the baseline & the DPDD must agree with the baseline. Change name to snapDiffPsfFlux. Will remove if snaps are removed. |
diffFluxErr | Estimated uncertainty of diffFlux. |
| LPG: Change to snapDiffPsfFluxErr |
fpBkgd | Estimated background at the position (centroid) of the object in the template image. |
| LPG: What is fp? |
fpBkgdErr | Estimated uncertainty offpBkgd. |
| |
Ixx | Adaptive second moment of the source intensity. See Bernstein & Jarvis [2] for detailed discussion of all adaptive-moment related quantities47 . |
| |
Iyy | Adaptive second moment of the source intensity. See Bernstein & Jarvis [2] for detailed discussion of all adaptive-moment related quantities47 . |
| |
Ixy | Adaptive second moment of the source intensity. |
| |
Icov | Ixx, Iyy, Ixy covariance matrix |
| |
IxxPSF | Adaptive second moment for the PSF |
| |
IyyPSF | Adaptive second moment for the PSF. |
| |
IxyPSF | Adaptive second moment for the PSF. |
| |
IxxDeconv, IxyDeconv, IyyDeconv | Estimate of PSF-deconvolved second moments. Initial implementation may be quadrature difference of second moments and |
| LPG: Follow up with SNAPs/SSSC about the usefulness of these. |
extendedness | A measure of extendedness, computed using a combination of available moments, or from a likelihood ratio of point/trailed source models (exact algorithm TBD). extendedness = 1 implies a high degree of confidence that the source is extended. extendedness = 0 implies a high degree of confidence that the source is pointlike. |
| |
spuriousness | A measure of spuriousness, computed using information from the source and image characterization, as well as the information on the Telescope and Camera system (e.g., ghost maps, defect maps, etc.). |
| RHL some sort of true/bogus number, I expect. Maybe more than one. LPG Measures (>1) to separate morph from other measures. Write desciprtion better. |
Flags | Various useful flags. |
|
Source Table
Column | Description | Retain/Move/Delete? (Robert Lupton ) | Comments |
---|---|---|---|
sourceId | Unique source identifier |
| |
ccdVisitId | ID of CCD and visit where this source was measured |
| |
objectId | ID of the Object this source was associated with, if any |
| |
ssObjectId | ID of the SSObject this source has been linked to, if any |
| LPG: Same reasoning as above - can join tables |
parentSourceId | ID of the parent Source this source has been deblended from, if any |
| |
xy | Position of the object (centroid) |
| LPG: See discussion above |
xyCov | Covariance matrix for xy |
| LPG: See discussion above |
radec | Calibrated RA, Dec |
| |
radecCov | Covariance matrix for radec. |
| |
apFlux | Calibrated aperture flux. |
| LPG: Calibration ap flux? Flux to which all else is tied? check with Eli/Robert. E.g. compensated aperture flux. |
apFluxErr | Estimated uncertainty of apFlux. |
| |
sky | Estimated background (sky) surface brightness at the position (centroid) of the source. |
| LPG: change to fpBkgd / fpBkgdErr. Remove sky from description. Majke same as description in DIASource. |
skyErr | Estimated uncertainty of sky. |
| LPG: change to fpBkgd / fpBkgdErr |
psRadec | Point source model ra/dec |
| LPG: See above |
psFlux | Calibrated point source model flux. |
| LPG: Change to psfFlux/psfFluxErr |
psFluxErr | Calibrated point source model flux uncertainty. |
| MWV: Breaking out the LPG: Change to psfFlux/psfFluxErr? To reconsider. |
psCov | Point-source model covariance matrix. |
| LPG: psfCov. Not currently computed. Could compute but do not publish. Will revise at a later date |
psLnL | Natural log likelihood of the observed data given the point source model. |
| MWV: What priors will we use for the log likelihood? Are we including astrophysical priors? Astrophysical priors that are a function of Galactic (l, b)? I think the answer has to be no, but if it is, then this isn't a likelihood. |
psChi2 | $\chi^2$ statistic of the model fit. |
| |
psNdata | The number of data points (pixels) used to fit the model. |
| LPG: See above |
Ixx | Adaptive second moment of the source intensity. See \citet{2002AJ....123..583B} for detailed discussion of all adaptive-moment related quantities\footnote{Or \url{http://ls.st/5f4} for a brief summary.}. |
| |
Iyy | Adaptive second moment of the source intensity. |
| |
Ixy | Adaptive second moment of the source intensity. |
| |
Icov | Ixx, Iyy, Ixy covariance matrix |
| |
IxxPSF | Adaptive second moment for the PSF. |
| |
IyyPSF | Adaptive second moment for the PSF. |
| |
IxyPSF | Adaptive second moment for the PSF. |
| LPG: Add deconvolved PSF - as per above. |
apNann | Number of elliptical annuli (see below). |
| MWV: I think this name is confusing. My brain keeps pulling a LPG: Not currently calculating elliptical apertures. Put forward changing elliptical to circular. Change to apNAnn. Need to check where the geometry of all the annulus is stored. |
apMeanSb | Mean surface brightness within an annulus. |
| LPG: 1 for each annulus. Flux v SB ? Assueme we want SB |
apMeanSbSigma | Standard deviation of apMeanSb |
| LPG: 1 for each annulus |
extendedness | A measure of extendedness (exact algorithm TBD). $extendedness=1$ implies a high degree of confidence that the source is extended. $extendedness=0$ implies a high degree of confidence that the source is point-like. |
| |
flags | Various useful flags. |
|