AP Pipeline Meeting, 2022-11-21

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Brianna Smart 
• Krzysztof Findeisen 
• Ari Heinze 
• John Parejko 
• Nima Sedaghat 
• Erin Howard 

regrets

• Kenneth Herner (vacation)

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • All-hands meeting Dec 15 9am
  • sprint planning next week, watch for a doodle
  • Yusra down in Chile this week, will be on the summit next few days.  Hopes to give a virtual tool during pipelines meeting on Wednesday
  • Ian reminds us to check on our passport expiration now for the Chile meeting!
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ): 
  • Krzysztof: USDF PP now has run end-to-end!  feature parity with GCP.  question about source association in #dm-alert-production
  • JP: making Zooniverse cutouts, now with metadata.  hard to get typesetting right in matplotlib
    • Eric dreams of more rows: variance, mask planes, SNR, PSF, matching kernel...
    • how to handle our many flags?  colored symbols seem like a bad idea, maybe 3-letter abbreviations?
• Precursor processing ( Kenneth Herner , Erin Howard ):
  • Erin: data being processed!  both HiTS and COSMOS running, they're monitoring.  seems hopeful to finish before the computers shut down on Thanksgiving.   (running w47 on SLURM)
  • reminder to update run comments in AP Precursor Processing Runs
• Image differencing algorithms, DCR  ( Ian Sullivan , Bruno Sanchez ):
  • Ian: pair coded with Bruno last week (Bruno driving): put together Alard & Lupton with preconvolution pipeline, detection and measurement on the appropriate score image.  pipeline functions, has some unit tests–went to John P for review.  looked decent in limited tests on ci hits.  Found at least one more bug to fix in testing on larger dataset.  extra convolution created larger nan border that needs to be trimmed.
• transinet (Nima Sedaghat, Eric Bellm):
  • looking at transinet & dealt with some edge cases due to the tract boundary
  • next week working on ZTF data during a Caltech visit next week, will try to implement transient on ZTF and get labels, then work on domain adaptation activity
• alert distribution (Brianna Smart , Eric Bellm  ):
  • got alert stream simulator working!  didn't want to run on a branch.  but then killed all the brokers...  error message seems that schema registry is down
  • kowl also not connecting to right IP
• Solar-system processing (Ari Heinze ):
  • just ran a big test on the ATLAS data with no bugs he could find, ready to launch on simulated LSST data–not clear what runtime will be.
• Review CI (https://chronograf-demo.lsst.codes/):
  • several weeks without data due to squash outage
  • some shifting based on fix of template variance bug–nothing too unexpected
• Review outstanding action items
• QA meeting in December:
  • first results from template variance bug fix, and/or mode=auto?
  • Eric on detection thresholds/fakes forced photometry?
• AOB

Action Items