Attending

"Fluffy" issues

  • DPDD Reviewers at FDR suggested: replace "Source" with "Detection" everywhere. Due to cost/benefit trade off, the decision is to not make this change.
  • Do we want to still include maggies? Included in SDSS to distinguish relative calibration of the survey from absolute calibration to a physical scale. Do we want to drop the distinction and switch everything to Janskys? Will require slight changes in fluxes between data releases (Jansky to Jansky conversion), which may seem odd. Unresolved, deferred to a written argument between Zeljko and Robert.
  • Negative fluxes? Discussion deferred.

Section 4.2.1

  • Why don't we say we will report dipole measurements? Are dipole measurements going to be useful to the science users? They will be useful for internal diagnostics, but internal quantities don't need to be reported in the DPDD. We might not know all of the cases that science users will want to extract information from. Robert argues that dipole measurements are not "fundamental" quantities; we should really be reporting an astrometric shift. Zeljko argues that dipole measurement is a pragmatic method towards this. Agreement that we should put it in now, but it will require further thought (who will do the thinking?).
  • Item 10: Precovery PSF measurements will be performed on the past 30 days of imaging. Suggestion from science users that this window is too short. K-T says this window is set by the cost of keeping lots of data. We were not completely understanding the science case for this. Suggestion that a more formalized path for users to provide suggested revisions and the associated science drivers would be more useful to us. We (Zeljko?) will request a specific science case (at SAC?). Melissa reports that supernovae surrounded by circumstellar material are likely to have precursor events, and are then very high priority targets for follow-up.
    • Zeljko Ivezic request specific science case concerning size of precovery window.

General Level 1

  • Discussion of the difference between the "Live" Level 1 database and the "Replica" Level 1 database. Jacek proposes not having full table scans on the "live" database (https://jira.lsstcorp.org/browse/RFC-133), but the these would still be supported on the replica for science users. Requirements say that after an alert is produced, it will immediately be available in the database (available to users). Our current design might not completely reflect this yet. Database design is still in process.
  • Are the Level 1 events available to be queried in a database? We don't think we've designed specifically whether we will store the verbatim alert itself or provide ways to recreate it. We will provide the ability to retrieve historical alerts somehow. K-T realizes now that we will have to store the alerts, because the Level1 database will be updated by DRP.
  • How does the alert stream get its Quality Assurance information? Many use cases for QA on alerts, but where should the description of this process live?
    • Zeljko Ivezic will add text to DPDD on use cases for QA on alerts.
  • Question also came up of providing feedback to the Observatory; what information do we need to provide to them?
    • Andrew Connolly will follow-up with PST on what information from L1 should be fed back to Observatory.
  • David Schlegel suggests we should report inverse covariances instead of covariances. RHL disagrees, no change.
  • We say we will report log likelihood and chi-squared. Why do we include both? They might differ, in which case log likelihood is the right answer, but users might be more comfortable with chi-squared. Deemed acceptable to keep both.
  • We have fluffy words that say we will not use the two snaps to determine the direction (sign) of motion of trailed sources by using the two snaps. This is also relevant for discussion of dropping snaps in favor of single exposures. Will receive some analysis with MOPS.
  • We say we will report "extendedness" of DIA Sources, but the text describing the measurement is very fluffy. Removefluff in favor of saying TBD.
  • "curved trail model". Delete.
  • How do we compute proper motion and parallax for DIAObject? No description in LDM-151 of how/when we generate this. If we do this, we need both parallax and PM to avoid biases. We have a linkage between DIAObject and Object, is that good enough? This might be useful for nearby objects, or objects in crowded regions/overlapping galaxies.
    • Zeljko Ivezic  will write down this science case for computing proper motion and parallax for DIAObjects.
  • Footnote 47: We assume a flat prior when matching DIAObject with Object. This is dangerous when there are many more faint sources than bright sources and one doesn't take brightness into account. Update to "appropriate" prior. Discussion that we only report the nearest 3 Objects to a DIAObject, and if one is trying to find the galaxy associated with a supernova, you might end up with three stars on the outskirts rather than the galaxy itself. Agreement that we should do slightly more sophisticated matching than just distance. Implementation left to further discussion.
  • Page 29 says we will bin templates by airmass and seeing. This is a questionable strategy, and doesn't belong in DPDD. Remove.
  • DPDD says that multifit will be used on variable sources, but this is a serious challenge. We should probably remove details of this implementation.
  • Should we do forced photometry of all Objects on either difference images or on direct images? Deferred discussion to the afternoon.


  • Questions about Level 1 reprocessing in Level 2. Are the DPDD plans still up to date? What are the uses cases for the detailed "catch-up" plans, or changing algorithms? Generally we will want to enable software updates more frequently than annually, but this will affect downstream users of the L1 stream (e.g. a user has a tuned ML algorithm that depends on quantities we might change). Do we need to hold off on algorithm updates, or is it sufficient to provide notification? Do we need to provide a reprocessed "training set" to users before the on-line change? No clear decision here.
  • Can we include the full associated Object record in alert stream? This is a data rights issue, since L1 will be world-public while L2 won't. Decision from Kahn this past Monday that we are allowed to include the full Object. This might be a lot of data, so there may be technical problems. We only provide alert footprints in the same filter as the DIA Source; will science users by unhappy about this?

Afternoon - Level 2

  • Our coadd algorithms are now significantly improved over the general discussion of different coadd types described in the DPDD. Jim says this text should all be replaced. Need a study of how well we can detect objects of different SEDs by using (multiple?) combinations of multi band coadds.
  • The timescale for coadds may also differ based on both the start of survey bootstrapping process and on science drivers for timescales shorter than DRP. 
  • page 32, Jim: "The master list of Objects" won't actually be generated from single epoch source detections; instead will be done from coadds and DIA sources. Lots of discussion of the distinction between sources and objects. I think Jim will have to fill out this section.
  • Star galaxy separation. Jim suggests the full bayesian likelihood ratio comparison between a bulge+disk galaxy fit and a moving point source fit. Sensitive to priors. Needs to be included in the text.
  • Section 5.2.1, Jim: should be doing sampling in point source model fit. Want a joint galaxy and star model, where a source can smoothly vary between a star and a galaxy depending on how the data drives the parameters. Important for galaxy shear? Zeljko argues that this needs significant testing before committing a large fraction of someone's time to this.
  • Section 5.2.1, the 200 samples we quote has a significant impact on database and compute sizing, and we are not sure of the justification for this number. Jim says that getting a better estimate of this number requires having a shear fitting code running at LSST performance levels, and we don't have that yet. K-T says this is not a huge space constraint since Objects is smaller than the Sources and ForcedSources tables. The number of realizations that are computed is also different from the number of samples saved.
  • Section 5.2.1, Variability characterization. Jim wants to switch to force photometry on difference images. How do we track the effect of the template? We think we can figure this out. In the case of blended sources, photometry on the difference images is superior. In the uncrowded case, you lose a bit of SNR.
  • Adaptive moments. Are they useful for star-galaxy separation at the faint end? Not the best. Other issue with the moments description is that table says e1 e2, switch to Ixx, Iyy. Should also clarify that we will use PSF matched images (double check this?).
  • Crowded fields. We are trying to optimize the cadence and decide what we will cover in the Galactic plane, but we don't know how well our software will do here. Baseline is currently the safe option of "we will do what we can", but this makes it hard to plan DM team time. Crowded field codes exist, can we say we will do a similar job to them? Replicating one of these codes is probably not overly challenging, but it's a question of prioritization of effort. Do we need a study on this? We say we will do image differencing on crowded fields, is that dependent on having a daophot-ish algorithm? Right now can't change the DPDD without incurring a change in scope, requires higher approval.
  • Standard color - we are using a galaxy model to measure standard colors, then we need to store the structural parameters used for it. Unless we define an existing set of structural parameters in this measurement. Requires clarification.
  • Extendedness - Jim suggests having three, morphological, colors, and a combination of both. Color might be more of L3 territory.
  • Photo-z - None of DM is working on this. We are supposed to run someone else's provided algorithm, but it's not clear that we are signed up for tuning, calibrating, etc, so this might not very good, and isn't under our own control. 
  • There isn't published throughput number per field or per object in the DPDD. We don't have any per-field tables in the DPDD. 
  • Are we publishing fluxes based on an assumed flat SED, or based on our best-guess SED? Robert says the latter, Mario and Zeljko say the documents prescribe the former. Need to update LSE-180.
  • DPDD says both "We do not plan to keep best-seeing coadds" and "We will keep 7 deep coadds". Zeljko argues that the best-seeing coadds are the best visually, and should be kept over the deep but un-pretty coadds. Jim says we can drop the multiband coadd, since it can be easily reconstructed. 
  • Page 58 claims we will rewrite Level 3 code into our framework if we want to include it in new Level 2 runs. Requires approval from above to remove it. 
  • Page 51 says how we will subdivide the sky into tracts and patches. This is an implementation detail and should go into a lower-level document. Dealing with overlapping area is challenging and should also be addressed (but elsewhere). Need to add a PRIMARY flag.
  • Should we add a section to DPDD saying that we will detect low surface brightness objects, e.g., via binning. Don't want to be too specific about implementation.

Review of Tables - DIA Source

  • Do we need to deblend in difference imaging? Possibly in the plane. Can we handle negative fluxes in the deblender?
  • Why do we record midPointTai for each source? Because the shutter has finite speed.
  • Why do we store pixel xy? Isn't that a diagnostic output instead of a science output? If we are including diagnostics, then there may be a wider class of columns we should be including.
  • Is ra/dec a SDSS centroid or a PSF centroid? Long discussion of centroiding followed. RHL says adding additional centroids is unlikely to be necessary, Mario votes in favor of adding the centroids until we show they are unnecessary.
  • Add aperture magnitudes.
  • SNR – this seems like a low value diagnostic field. 
  • Trailed sources - We need to check if we need an additional centroid. Add, then remove if shown we don't need it.
  • fpFlux - Discussion of if we need to store additional information to make this measurement useful in crowded fields. Background gradient? But argued that it shouldn't bias the centroid to omit that information.
  • diffFlux - Useful for finding sources varying very rapidly in brightness. Is this useful for rejecting artifacts that only appear in one image? Maybe we want to measure trailed sources on the two snaps individually?
  • fpSky - Add that this is on the visit, not the difference.
  • Change E1, E2, to Ixx, Iyy, Ixy, drop mSum
  • Make available PSF Ixx, Iyy, Ixy

Possible additions:

  • Do we want to fit comet models?
  • Injecting false sources? Vague comment right now, but we should make it more clear. Also should add sky sources.
  • Need to specify how users are supposed to obtain light curves, should they get these from L1, or should the only depend on L2? Had a series of ideas on how to generate this, but we should write down a science-user level description of how they access this data.
  • Do we want to associate DIAObjects with Objects? We already do. What about DRP reprocessing of DIA Source tables?
  • Rename fpSky, since it should include flux from neighboring objects (like a host galaxy) rather than pure "sky". Also it seems more useful to make fpSky be measured on the template, not science exposure.
  •