Data Management
Space shortcuts
Page tree
Skip to end of metadata
Go to start of metadata

DP0 general information:

  • SLAC channel: #ops-data-previews
  • Meetings: Very optional biweekly meetings 8am Tuesday 
  • Deadline: Pipeline for DP0.2 freeze June 30th 2021.  (Patches for correcting performance of already extant components will be accepted for a couple months after) 


Accounting update for Summer:

  • Last call for story points on March/April/May: PREOPS-450. PREOPS-450 - Getting issue details... STATUS
    • Scheduled AP folks:  Ken: 32, Ian: 7.5, JohnP: 15, Chris: 15 :thankyou:
  • Epic for June/July/August: f21a-a&p-preops   PREOPS-477 - Getting issue details... STATUS
    • Scheduled:  Assume same as PREOPS-450

Data

Processing DR6-WFD: 
Only Variable Sources in WFD are SNe: https://arxiv.org/pdf/2010.05926.pdf
Good for SNe, and validating that everything else is flat


/repo/dc2 
https://community.lsst.org/t/shared-gen3-data-repositories-ready-for-some-use/4845 This includes all of the DESC DC2 DR6 WFD raws and processing that will be used in DP0.1 (it’s a clone of the IDF repo - or rather the converse). That includes the original calibs used for the DESC processing. The dataset processed approximately monthly by DM is a subset of this, and there are two TAGGED collections that contain the raws for these important subsets:

2.2i/raw/DP0: raws for DP0 
2.2i/raw/test-med-1 (DM monthly processing subset: tract 3828/3829
The full DESC-run DR6 WFD processing can be found in the 2.2i/runs/DP0.1 collection.

Pending reruns as of:
w_2021_14 HSC rerun will have best seeing templates
w_2021_16 DC2 rerun will contain difference images and diffim lightcurves 



What's left as of April 26 2020:


Difference imaging:

  • Are false positive rates on DC2 reprocessing as expected?
  • Rerun a tract using DR6. What’s a suitable template? Check with Bruno on what they did
  • Weird diasource centroids off footprint and off chip (DRP will look at this in pair coding again)
  • Ability to handle science images with seeing better than the templates. Decide whether to
    • deconvolve anyway
    • swap science and template
    • preconvolution


DRP-mode Association

  • Associate DiaSources into DiaObjects (At the very minimum do something (DONE) like: https://github.com/LSSTDESC/dia_pipe/blob/master/python/lsst/dia/pipe/multimatch_association.py)  
  • and then rename Objects to PreliminaryObjects and  associate DiaObjects + PreliminaryObjects  = Objects?  (DONE)
  • DOES NOT include variability stats for each of the objects (just centroids to seed forced photometry)
    • CBM: One comment, centroiding a DiaObject is, as I referred on the call, a DiaObject summary statistic so we will be expecting to compute at least a few summary stats on the DiaSource history according to the above statement.
  • Some summary stats.


Template Generation:

  •  Use CoaddPsfs from all warps in getTemplate task
  • Get data from multiple-tracts for complete templates. 

Forced Photometry:

  • Compare the calexp lightcurves vs. the diffim lightcurves. Does this behave as expected? Is dependence on seeing as expected? 
  • Ensure that whatever we do for association eventually gives us all diaObjects forced phot on coadds too. 

Standard Data Model:

  • ForcedSource SDM specifications (including psFlux and psDiffFlux) 
  • Visit Table  
  • CcdVisit Table  
  • New as of June 8:

June:

  • Instructions for how to build a pipeline and run it on bps. (Meredith)
  • Datasets to run:
    • 1.5 year depth is run by Monika 3828, 3829 DRP pipeline
    • 6 year depth is up to us to run 4431 DRP pipeline
  • Implementation of RFC-775 (all)
  • Look into slot_shapeFlag (Ken)
  • QA on w24 DC2 rerun (Eric)
  • QA on w24 (adapt Meredith's notebooks to use parquet tables) 


General Plumbing:


Minimum Viable Product version would be Bob Armstrong’s pipelines: https://github.com/LSSTDESC/dia_pipe


  • No labels