AP Pipeline Meeting, 2021-05-24

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan
• Unknown User (cmorrison)
• Unknown User (gkovacs)
• Krzysztof Findeisen
• Kenneth Herner
• Meredith Rawls

regrets:

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • replan review is underway
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • working on getting fakes running in a way that will be compatible with later pipelines rearrangement–still just starting
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Meredith:
    • still having some trouble with the last stage of DECam template building in gen3 shared repo
    • is showing DC2 preliminary processing, small amount of g-band data
      
      • there are some edge effects visible, and troubling overdensities
      • is there dithering?  we think so
      • is the DDF included in this data/DP0.2?  If not, are there are variables
  • Ken: still stuck on problem w/ change since last weekly.  otherwise busy with non-Rubin work
  • QA meeting:
    • fakes?  Krzysztof is working on it this week but out next week, so we will see
    • Meredith: nothing to show with HSC, DECam still debugging; just showed DC2 (on ap_pipe)
      • Ken thinks he can work on DC2 in next couple weeks
      • Meredith will try to show DECam data if she can get it working
    • Ian: can we try to get cutouts with some metadata displayed alongside?  (seeing, etc.)
      • don't worry about making it elegant in the short term, can print it
    • future topics:
      • Ian: diffim unit tests?  if results
      • probably already overbooked for next QA meeting, but we could tag Gabor in the future for some show and tell
      • also for next month: showing/discussing persisted likelihood images would be useful

• Difference imaging algorithms  (Unknown User (gkovacs), Ian Sullivan ):
  • Gabor: showing a notebook trying to understand why simple positive/negative pixel swap metric changed when image order is swapped in ZOGY.
    • problem was due to padding images with random noise (to avoid artificially biasing the variance low near the edges)
    • found a bug that the metric was not actually calculated at source locations, but at random positions
    • also sees grid-like (high-frequency) oscillation around zero–so the unit test failed
    • fix is easy, already ready on a branch–seed the random number generator with the image rather than the image order
• association/SDM/fakes/astrometry (Unknown User (cmorrison)):
  • busy with DP0.2
• real-bogus (Eric Bellm, John Parejko ):
  • no update
• Preops DP0.2 preparation (all):
  • Chris:
    • cleaning up simple association from DESC's dia_pipe, adding unit
    • still waiting on review to merge main pipeline
  • Meredith:
    • draft writeup at "what I did" stage, unblocking Ken
  • Ken to work on processing from here, with shadowing from John Parejko
• Review CI (https://chronograf-demo.lsst.codes/):
  •  nothing new.  Chris believed he had identified some larger-than-floating point differences when reading from raws in DM-29901 - Getting issue details... STATUS , which conflicts with Lauren's claim that HSC Gen2/Gen3 were bitwise identical coming out of SFM.  We want to confirm Chris's work and ticket middleware if necessary.
• Review outstanding action items
• AOB
  • Eric organizing SP weekly meetings in June, so topic suggestions welcome!  (I have some ideas)
  • several vacations this week and next
  • Meredith still wondering where to put pipelines...  DRP folks are putting stuff in obs_subaru that import from pipe_tasks/DRP.yaml. 

Action Items