At the SST F2F meeting on 5 November 2018, we reviewed the contents of the Source and DIASource tables.  It was not possible to capture all the decisions about the columns in these tables in the meeting. The tables below contain the contents of the Source and DIASource tables transcribed from the DPDD. Three options were identified for each column, keep as is, keep but move to another (currently unspecified) table,  or remove entirely.  

We should also consider the possibility of using a Gaia-style approach of storing everything and choosing what to publish

MWV: Overall Comments.

  1. The SST and the DPDD are about what the user sees and can access as a column.  That's separate from how they are stored or generated.  The lower-level Science Data Model will benefit from much of these discussions.
  2. I do think that trail* and dip* columns need to be revisited.  Here is a case where implementation is much more closely tied to the column definitions and where accepted practice is much less settled.
    1. trail* More work should be done to come up with a planned model of how to do this photometry and related measurements and then that can be turned into a proposal for what the columns should be fore the DPDD
    2. dip*  These columns are the least certain to me.  They are largely for characterizing our ability to do image subtraction.

Definition of Time in the Source/Object tables

Robert Lupton  to clarify the meaning of time in the object table.  (See DM-SST action 2018-11-05 DM SST F2F Agenda and Meeting notes)

RHL: I believe that the object table field in question is {{psRadecTai}}.  I propose that we replace this field with an epoch that applies to all measurements in the object table.  The epoch could be c. 2024 to coincide with the start of operations, or 2030 to minimise the amount of extrapolation required;  we could use a different epoch for different releases, but that would probably be a little confusing.

LPG: Gaia choose this approach also and also has a different reference epoch for  each DR. The reference epoch for Gaia DR3 is J2016.0. It is J2015.5 for Gaia DR2 and J2015.0 for Gaia DR1.

RHL: For DIASource table, midPointTai also needs to be defined.  We know the time profile for the shutter at every point in the focal plane from the camera's Hall effect sensors, and I propose that we define the midPointTai to be the mean of this time profile on the assumption that the source does not vary during the exposure.   If we take data using snaps, this would be the average of the two half-exposures, and I propose that we do not weight these two measurements by the flux in the two snaps as this would require us to measure fluxes on the two snaps, which is not the current plan, and also change the definition of midPointTai depending on whether the data is taken with- or without- snaps (and thus potentially depending on the band).

DIASource Table

ColumnDescription

Retain/Move/Delete? (Robert Lupton )

Comments

diaSourceId

 Unique source identifier 

  • Retain in table
  • Move to another table
  • Remove entirely

ccdVisitId

 ID of CCD and visit where this source was measured 

  • Retain in table
  • Move to another table
  • Remove entirely

diaObjectId

 ID of the DIAObject this source was associated with  if any

  • Retain in table
  • Move to another table
  • Remove entirely

ssObjectId

 ID of the SSObject this source has been linked to if any

  • Retain in table
  • Move to another table
  • Remove entirely

RHL: I haven't thought about how to normalise/denormalise these tables, and would expect the ssObjectId to come via a join to diaObject

LPG: I thought a DIASource was associated with either a DIAObject or an SSObject, but not both? 2022-03-04: SSSource Table join on diaSourceId

MWV: If SSObjects are not DIAObjects (which is my understanding of the current written plan), then both of these columns need to be retained.

parentDiaSourceId

 ID of the parent DIASource this object has been deblended from if any

  • Retain in table
  • Move to another table
  • Remove entirely
RHL: I'm hoping not to have to deal with blended diaSources, but for completeness we need this.

midPointTai

 Time of mid-exposure for this DIASource. The visit mid-exposure time generally depends on the position of the source relative to the shutter blade motion trajectory.

  • Retain in table
  • Move to another table
  • Remove entirely
RHL: or move to a lookup based on the known shutter timing.

radec

 Centroid

  • Retain in table
  • Move to another table
  • Remove entirely

radecCov

 radec covariance matrix.

  • Retain in table
  • Move to another table
  • Remove entirely

Note full cov matrix not just Err. 

DM-7101 - Getting issue details... STATUS

xy

 Column and row of the centroid.

  • Retain in table
  • Move to another table
  • Remove entirely


xyCov

 Centroid covariance matrix. 

  • Retain in table
  • Move to another table
  • Remove entirely


apFlux

the flux difference between the template and the visit image. 

  • Retain in table
  • Move to another table
  • Remove entirely
RHL define radius to be used. Is one enough?

apFluxErr

  Estimated uncertainty of apFlux.

  • Retain in table
  • Move to another table
  • Remove entirely

SNR

 The signal-to-noise ratio at which this source was detected in the difference image.

  • Retain in table
  • Move to another table
  • Remove entirely

RHL this is not that well defined. I'd expect to use the S/N in the PSF flux (correct for point sources such as SNe and asteroids; not for comets), so we don't need this too

MWV: Keep. But define what it means. Phase 1 detection threshold? Phase 2 detection threshold? abs(psFlux)/psFluxErr?

LPG: The detection threshold will be in provenance and the same for all source detections.  The signal-to-noise ratio at which this source was detected is abs(psFlux)/psFluxErr and can be computed from the following 2 columns. We will remove this. 

ECB: note that the detection threshold is not abs(psFlux)/psFluxErr–detection SNR and measured SNR are different.  See DM-31811

psFlux

 Calibrated flux for point source model. Note this actually measures the flux difference between the template and the visit image.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: Change to psfFlux if not done already.
psFluxErr
  • Retain in table
  • Add to table
  • Move to another table
  • Remove entirely

RHL not present in the DPDD

MWV: Agreed. Users will want this very simple quantity, even if it can be extracted from psCov.

LPG: psfFluxErr if not done already

psRadec

 Centroid for point source model.

  • Retain in table
  • Move to another table
  • Remove entirely

RHL We already have raDec

MWV: Keep for now, but if implementation makes this redundant with raDec consider removing at that time.

LPG: Potentially confusing  - will be the same forpoint sources. 

psCov

 Covariance matrix for point source model parameters.

  • Retain in table
  • Move to another table
  • Remove entirely

RHL In theory we need the full position x flux covariance, but in practice I think that we can separate them (and then they're in already)

MWV: I think we should keep, even if some of the first implementations are only diagonal or block diagonal.

LPG: psfCov.  Not currently computed. Could compute but do not publish 

psLnL

 Natural log likelihood of the observed data given the point source model.

  • Retain in table
  • Move to another table
  • Remove entirely

RHL This was intended to be some sort of S/G number, I suspect, but I expect to use extendedness instead. This covers the next two entries too (psChi2, psNdata)

MWV: I agree that LnL and Chi2 need some further thought. It's not clear to me that LnL is what we will want quote – are we including an astrophysical prior on the point source vs. extended source distribution as part of our point source model?

ECB: I think all of these are trying to enable model comparison/goodness of fit for the point source vs trail vs dipole models.  We should convey that information somehow...

psChi2

chi^2 statistic of the model fit.

  • Retain in table
  • Move to another table
  • Remove entirely
MWV: I would argue to keep.

psNdata

 The number of data points (pixels) used to fit the model.

  • Retain in table
  • Move to another table
  • Remove entirely
MWV: I'm ambivalent. I can't figure out if this is usefully and consistently defined. Is this useful to understand performance of de-blended objects? I certainly can't really imagine using this column across different visits.

trailFlux

 Calibrated flux for a trailed source model. A Trailed Source Model attempts to fit a (PSF-convolved) model of a point source that was trailed by a certain amount in some direction (taking into account the two-snap nature of the visit

  • Retain in table
  • Move to another table
  • Remove entirely

RHL We want some sort of trailed number, probably. However, I'm not sure that we need it for every entry in the diaSource table, and I'm not convinced that we need all these numbers. The orientation is given by the second moments (Ixx etc. below), so a flux, fluxError and length is probably enough. So I'm voting for retain, but not to be taken literally.

RHL We also need trailFluxErr

MWV: I agree. More work needed here to develop a specific plan for trailing measurements and then return and figure out what all of these trail* columns should be.

LPG: All tail* fields need to be reviewed with Mario

trailRadec

 Centroid for trailed source model.

  • Retain in table
  • Move to another table
  • Remove entirely
RHL use raDec

trailLength

 Maximum likelihood fit of trail length (Note that we'll likely measure trailRow and trailCol)

  • Retain in table
  • Move to another table
  • Remove entirely

trailAngle

 Maximum likelihood fit of the angle between the meridian through the centroid and the trail direction (bearing

  • Retain in table
  • Move to another table
  • Remove entirely

trailCov

 Covariance matrix of trailed source model parameters.

  • Retain in table
  • Move to another table
  • Remove entirely

trailLnL

Natural log likelihood of the observed data given the trailed source model.

  • Retain in table
  • Move to another table
  • Remove entirely

trailChi2

chi^2 statistic of the model fit.

  • Retain in table
  • Move to another table
  • Remove entirely

trailNdata

The number of data points (pixels) used to fit the model.

  • Retain in table
  • Move to another table
  • Remove entirely

dipMeanFlux

 Maximum likelihood value for the mean absolute flux of the two lobes for a dipole model. A Dipole Model attempts to fit a (PSF-convolved) model of two point sources of opposite signs.

  • Retain in table
  • Move to another table
  • Remove entirely

RHL Remove all the dipole measurements. We will need some numbers to drive true/bogus measurements

LPG: I understood we would move all dipole measurements to another table as they were useful for debugging (Simon Krughoff ?)

MWV: I think I agree that the dipole measurements likely should be removed from the DPDD DiaSource Table. This isn't a particularly stable or well-defined quantity.

LPG: Remove all dipole measurements frome the DPDD. Check with UW as to whether this is still useful and if it is the afw tables.

ECB: are we sure there are no science uses for these?  early identification of proper motion stars?

dipFluxDiff

Maximum likelihood value for the difference of absolute fluxes of the two lobes for a dipole model.

  • Retain in table
  • Move to another table
  • Remove entirely

dipRadec

Centroid for dipole model.

  • Retain in table
  • Move to another table
  • Remove entirely

dipLength

Maximum likelihood value for the lobe separation in dipole model.

  • Retain in table
  • Move to another table
  • Remove entirely

dipAngle

 Maximum likelihood fit of the angle between the meridian through the centroid and the dipole direction (bearing

  • Retain in table
  • Move to another table
  • Remove entirely

dipCov

 Covariance matrix of dipole model parameters.

  • Retain in table
  • Move to another table
  • Remove entirely

dipLnL

 Natural $log$ likelihood of the observed data given the dipole source model.

  • Retain in table
  • Move to another table
  • Remove entirely
MWV: I think LnL for dipole makes particularly no sense.

dipChi2

chi^ statistic of the model fit.

  • Retain in table
  • Move to another table
  • Remove entirely

dipNdata

 The number of data points (pixels) used to fit the model.

  • Retain in table
  • Move to another table
  • Remove entirely

totFlux

 Calibrated flux for point source model measured on the visit image centered at the centroid measured on the difference image (forced photometry flux)

  • Retain in table
  • Move to another table
  • Remove entirely

RHL: unmeasurable in crowded fields. I believe that this was added to break degeneracies between dipole separation and amplitude.

ECB: No, there's a science use--we need some way to go from difference image flux to total flux for variable stars.  Forced photometry on the visit image seems preferable to me to the only alternative I'm aware of, which is figuring out the flux at the position of the DIASource in the template and adding it.  The latter requires the same operations, has the same crowding problem, and creates more complications.

MWV: This is a doable measurement. It just doesn't necessarily always mean what you might want it to mean. But we should still do and provide it, even though it means something different in crowded fields.

LPG: Retain but change name to totPsfFlux

totFluxErr

 Estimated uncertainty of totFlux.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: Retain but change name to totPsfFluxErr

diffFlux

Calibrated flux for point source model centered on radec but measured on the difference of snaps comprising this visit. This flux can be used to detect sources changing on timescales comparable to snap exposure length ~15

  • Retain in table
  • Move to another table
  • Remove entirely

LPG: Snaps are still in the baseline & the DPDD must agree with the baseline.   Change name to  snapDiffPsfFlux.  Will remove if snaps are removed. 

diffFluxErr

 Estimated uncertainty of diffFlux.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: Change to snapDiffPsfFluxErr

fpBkgd

 Estimated background at the position (centroid) of the object in the template image.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: What is fp? 

fpBkgdErr

 Estimated uncertainty offpBkgd.

  • Retain in table
  • Move to another table
  • Remove entirely

Ixx

Adaptive second moment of the source intensity. See Bernstein & Jarvis [2] for detailed discussion of all adaptive-moment related quantities47 .

  • Retain in table
  • Move to another table
  • Remove entirely

Iyy

Adaptive second moment of the source intensity. See Bernstein & Jarvis [2] for detailed discussion of all adaptive-moment related quantities47 .

  • Retain in table
  • Move to another table
  • Remove entirely

Ixy

Adaptive second moment of the source intensity.

  • Retain in table
  • Move to another table
  • Remove entirely

Icov

Ixx, Iyy, Ixy covariance matrix

  • Retain in table
  • Move to another table
  • Remove entirely

IxxPSF

Adaptive second moment for the PSF

  • Retain in table
  • Move to another table
  • Remove entirely

IyyPSF

 Adaptive second moment for the PSF.

  • Retain in table
  • Move to another table
  • Remove entirely

IxyPSF

 Adaptive second moment for the PSF.

  • Retain in table
  • Move to another table
  • Remove entirely

IxxDeconv, IxyDeconv, IyyDeconvEstimate of PSF-deconvolved second moments.  Initial implementation may be quadrature difference of second moments and 
  • Retain in table
  • Add to table
  • Remove entirely
LPG: Follow up with SNAPs/SSSC about the usefulness of these. 

extendedness

A measure of extendedness, computed using a combination of available moments, or from a likelihood ratio of point/trailed source models (exact algorithm TBD). extendedness = 1 implies a high degree of confidence that the source is extended. extendedness = 0 implies a high degree of confidence that the source is pointlike.

  • Retain in table
  • Move to another table
  • Remove entirely

spuriousness

A measure of spuriousness, computed using information from the source and image characterization, as well as the information on the Telescope and Camera system (e.g., ghost maps, defect maps, etc.).

  • Retain in table
  • Move to another table
  • Remove entirely

RHL some sort of true/bogus number, I expect. Maybe more than one.

LPG Measures (>1) to separate morph from other measures. Write desciprtion better. 

Flags

Various useful flags.

  • Retain in table
  • Move to another table
  • Remove entirely


Source Table


ColumnDescription

Retain/Move/Delete? (Robert Lupton )

Comments

sourceId

Unique source identifier

  • Retain in table
  • Move to another table
  • Remove entirely

ccdVisitId

ID of CCD and visit where this source was measured

  • Retain in table
  • Move to another table
  • Remove entirely

objectId

 ID of the Object this source was associated with, if any 

  • Retain in table
  • Move to another table
  • Remove entirely

ssObjectId

 ID of the SSObject this source has been linked to, if any 

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: Same reasoning as above - can join tables 

parentSourceId

 ID of the parent Source this source has been deblended from, if any

  • Retain in table
  • Move to another table
  • Remove entirely

xy

 Position of the object (centroid)

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: See discussion above

xyCov

 Covariance matrix for xy

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: See discussion above

radec

 Calibrated RA, Dec

  • Retain in table
  • Move to another table
  • Remove entirely

radecCov

 Covariance matrix for radec.

  • Retain in table
  • Move to another table
  • Remove entirely

apFlux

 Calibrated aperture flux.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: Calibration ap flux? Flux to which all else is tied? check with Eli/Robert. E.g. compensated aperture flux. 

apFluxErr

  Estimated uncertainty of apFlux.

  • Retain in table
  • Move to another table
  • Remove entirely

sky

Estimated background (sky) surface brightness at the position (centroid) of the source.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: change to fpBkgd / fpBkgdErr. Remove sky from description. Majke same as description in DIASource. 

skyErr

 Estimated uncertainty of sky.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: change to fpBkgd / fpBkgdErr

psRadec

 Point source model ra/dec

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: See above 

psFlux

 Calibrated point source model flux.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: Change to psfFlux/psfFluxErr
psFluxErr Calibrated point source model flux uncertainty.
  • Retain in table
  • Move to another table
  • Add to table
  • Remove entirely

MWV: Breaking out the psFluxErr from psCov will substantially help the user.

LPG: Change to psfFlux/psfFluxErr? To reconsider. 

psCov

Point-source model covariance matrix.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: psfCov.  Not currently computed. Could compute but do not publish. Will revise at a later date

psLnL

 Natural log likelihood of the observed data given the point source model.

  • Retain in table
  • Move to another table
  • Remove entirely
MWV: What priors will we use for the log likelihood? Are we including astrophysical priors? Astrophysical priors that are a function of Galactic (l, b)? I think the answer has to be no, but if it is, then this isn't a likelihood.

psChi2

$\chi^2$ statistic of the model fit.

  • Retain in table
  • Move to another table
  • Remove entirely

psNdata

 The number of data points (pixels) used to fit the model.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: See above

Ixx

Adaptive second moment of the source intensity. See \citet{2002AJ....123..583B} for detailed discussion of all adaptive-moment related quantities\footnote{Or \url{http://ls.st/5f4} for a brief summary.}.

  • Retain in table
  • Move to another table
  • Remove entirely

Iyy

 Adaptive second moment of the source intensity.

  • Retain in table
  • Move to another table
  • Remove entirely

Ixy

 Adaptive second moment of the source intensity.

  • Retain in table
  • Move to another table
  • Remove entirely

Icov

 Ixx, Iyy, Ixy covariance matrix

  • Retain in table
  • Move to another table
  • Remove entirely

IxxPSF

 Adaptive second moment for the PSF.

  • Retain in table
  • Move to another table
  • Remove entirely

IyyPSF

 Adaptive second moment for the PSF.

  • Retain in table
  • Move to another table
  • Remove entirely

IxyPSF

 Adaptive second moment for the PSF.

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: Add deconvolved PSF - as per above. 

apNann

 Number of elliptical annuli (see below).

  • Retain in table
  • Move to another table
  • Remove entirely

MWV: I think this name is confusing. My brain keeps pulling a NaN out of this name and then not being sure what to do with the other letters. The simplest would perhaps be just to spell it out: apNannuli

LPG: Not currently calculating elliptical apertures. Put forward changing elliptical to circular. 

Change to apNAnn. 

Need to check where the geometry of all the annulus is stored. 

apMeanSb

 Mean surface brightness within an annulus.

  • Retain in table
  • Move to another table
  • Remove entirely

LPG: 1 for each annulus.  

Flux v SB ? Assueme we want SB

apMeanSbSigma

 Standard deviation of apMeanSb

  • Retain in table
  • Move to another table
  • Remove entirely
LPG: 1 for each annulus

extendedness

 A measure of extendedness (exact algorithm TBD). $extendedness=1$ implies a high degree of confidence that the source is extended. $extendedness=0$ implies a high degree of confidence that the source is point-like.

  • Retain in table
  • Move to another table
  • Remove entirely

flags

Various useful flags.

  • Retain in table
  • Move to another table
  • Remove entirely