AP Pipeline Meeting, 2022-11-28

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Krzysztof Findeisen 
• Erin Howard 
• Brianna Smart 
• John Parejko 
• Kenneth Herner 
• Nima Sedaghat 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • Commissioning schedule update went out last week
  • Sprint planning this week
  • Data Processing sprint in December: Dec 7 during pair coding-following week, w/ sprint closeout Dec 16
    • probably looking at most recent HSC reprocessing
    • aiming to build up some tooling around digging into a new dataset, especially looking at sources over time, etc.
    • catalog data, lightcurves & timeseries analysis, John P's cutout visualization
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ): 
  • no updates
  • John: improved cutout plots in review
  • John: DC2 now running in CI daily
• Precursor processing ( Kenneth Herner , Erin Howard ):
  • Erin: reprocessed HSC COSMOS & DECam on weekly 46, now comparing to weekly 39–some slight changes, but neither major failures or major improvements
    • Ian: do capture on the ticket any FATAL errors
  • Erin: still fighting calibs for Saha Bulge
  • Ken: DC2 tract from NCSA still not transferred
• Image differencing algorithms, DCR  ( Ian Sullivan , Bruno Sanchez ):
  • Ian: worked through preconvolution ticket with Bruno (DM-34909), in review with John.  found some bugs that are also present in other processing, requires some design discussion.  Return of the question about growing bad pixel masks during convolution.  Trim them off?  or just mask edge.  Not clear throwing away data is a good idea, even if it makes the processing a bit faster/cleaner
    • Eric: we still haven't dealt with the general question of propagating mask planes w/ convolution
    • Nima: in computer vision we mirror edge pixels to help suppress edge effects
      • similar to the idea of interpolating over saturated streaks to avoiding edges during convolution
• transinet (Nima Sedaghat, Eric Bellm):
  • Nima: DM-36743 worked: successfully applied DC2 model in ap_verify, generated output butler dataset
  • right now we don't get the RB scores into the output catalog–need to ticket the plumbing
  • Nima thinks there could be some shortcuts if we want to use something for the sprint
  • some open questions about optional execution
  • hasn't yet had a chance to QA the outputs
• alert distribution (Brianna Smart , Eric Bellm ):
  • after over 100 messages on Weds, alert distribution is standing up and simulator running.  not externally accessible yet but Bri has a lead
• Solar-system processing (Ari Heinze ):
  • orbit fitting ran successfully on simulated LSST data over Thanksgiving–analyzing results now
  • learning some Python & astroquery to help with ZTF Jupyter trojan project.  pybind11 next.  JP: developer guide has some useful stuff. KF: yes, we have our own wrapper, so check the dev guide
• Review CI (https://chronograf-demo.lsst.codes/):
• Review outstanding action items
• QA meeting in December:
  • first results from template variance bug fix, and/or mode=auto?
    • former a 10% effect in the HSC reprocessing
    • Ken can show something for these in the diffim test dataset
  • DC2 CI.  RB scoring–defer for now?  or maybe Eric & JP try to work on this at pair coding Wednesday
  • Eric on detection thresholds/fakes forced photometry?  ...
• AOB
  • JP wants to get the RFC for ap column renaming?
  • Do we need Docker licenses?  advice is to ask on #dm-square

Action Items