AP Pipeline Meeting, 2021-07-19

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan
• Krzysztof Findeisen
• Unknown User (cmorrison)
• Kenneth Herner
• Unknown User (gkovacs)
• Meredith Rawls
• Hayden

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • not much of broad interest
• Using the Rubin pipelines for moving objects (talk by Hayden Smotherman (UW)):
  • synthetic tracking: add flux around arbitrary trajectories through an image
  • KBMOD: do this on GPUs
  • either mask or difference images
  • convolve to make likelihoods, then coadd along trajectories (linear, right now)
  • initially were using ~20 images, now are up to 90-100 image stacks from DECam (DECAM VR band–broad filter around PS1 R)
  • using SP, butler gen3, startling with DECam raws to make calibs (bias, flats; no darks), run crosstalk, gaia astrometry and PS1 photometric calibratoin
    • assuming darks are negligible
  • A&L diffim with decorrelation afterburner
  • have explored separating templates from data to avoid subtracting off the signal, which loses sensitivity (10-20%)
    • using temporally separated data can create issues due to differences in template seeing
  • mask features seen in > 70% of images w/ rejected/detected
  • mask any bad/edge/no_data/suspect/unmaskednan
  • recovering objects from subthreshold individual stamps, as desired
  • 10^12 test trajectories; for ~100 candidates then do refined trajectory estimation
  • artifacts:
    • dipoles
    • sources on mask borders
    • edges
    • ringing deconvolution artifacts
    • bright stars that pass through the trajectory: can these be trusted?
      • can you throw out just the contaminated frames?  no, they are in ~all the frames
      • test with injected fakes onto a similar star?  yes, interested in injections as a next step. 
        • insertFakes.py in pipe_tasks is the starting point
        • ap_pipe has createApRandomFakes
        • https://community.lsst.org/t/synthetic-source-injection-workshop-2021-06-03/5198
  • Q&A:
    • how did you ingest the gen3 calibs?
      • nothing fancy, worked with Meredith to do this
    • what is the seeing selection (if any) on the template?
      • unsure–feeding all of the data to the template gen.  total image span is a few hours within the night over a few nights, so not a lot of spread in the
    • artifacts?
      • train ResNET50 CNN
      • "lots" of false positives (where real/bogus is on the whole trajectory + image stamps to coadd: don't train on individual stamps, only the coadd)
      • had to make synthetic reals since these KBOs are rare.  put on real variability, use range of trajectories, brightness
      • so no CNN → CNN drops number of candidates by 1-2 orders of magnitude.
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • nothing to report
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Meredith resurfacing from satellites
  • Ken: trying to rerun the 6-year depth with Jim's updated repo–didn't help much
    • he thinks this includes the no-work flag already
    • bubble up to #dm-science-pipelines?
  • also trying to work on the convolve science images
• Image differencing algorithms  (Unknown User (gkovacs), Ian Sullivan ):
  • Gabor: working on adding WCS to score image, CI on another
  • Ian: got configs sorted out for convolving science, hasn't had a chance to look in detail but the overall number seems to have decreased.  Ken to test at scale.
• association/SDM/fakes/astrometry (Unknown User (cmorrison)):
  • sent ticket moving pipelines for SS association to Ken for review.  Will ping Siegfried to see if he can help
  • running COSMOS on new metrics to provide numbers of fakes
• real-bogus (Eric Bellm, John Parejko ):
  • chatting with Nima about workflow
• alert distribution (Spencer Nelson):
  
  •  out today
• Preops DP0.2 preparation (all):
  •  sprint demo Wednesday?
• Review CI (https://chronograf-demo.lsst.codes/):
  • CI was out for a long while due to pandas; we've pinned back to an old version for now
  • no substantial changes
• Review outstanding action items
• AOB
  • Ian is out next Monday
  • PCW session on science pipelines: Eric and Ian to coordinate presentation.  Include large fakes run from Meredith, preview at the QA meeting in two weeks.

Action Items