AP Pipeline Meeting, 2021-10-25

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Ian Sullivan 
• Unknown User (cmorrison) 
• Kenneth Herner 
• Krzysztof Findeisen 
• Spencer Nelson 
• Meredith Rawls 
• John Parejko 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • none
• Pipelines Infrastructure (Krzysztof Findeisen ):
  • looking into how to manage ap_verify datasets entirely in gen3
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Ken: made calibration products for Saha Bulge, tried pipelines but got no diffim–still debugging.  Hasn't given up on next week, but right now there's nothing in hand.
  • Meredith: made a plot of templates for gen3 HiTS–those look fine, so are usable if someone else wants to run.
    • DM-31769 is supposed to resolve the issues we've seen, but it hasn't landed yet
    • chatting with Colin about preops V&V next.
• Image differencing algorithms  ( Ian Sullivan ):
  • nothing
• association/SDM/fakes/astrometry (Unknown User (cmorrison)):
  • gave a review to Yusra about concatenating pandas dataframes. 
    • Eric: are they testing these?
  • finished two matcher optimization tickets, starting optimizing the next loop
• real-bogus (Eric Bellm, John Parejko ):
  • no updates
• alert distribution/performance tuning (Spencer Nelson):
  • went to squareops bootcamp, learned to use their Kubernetes cluster, starting Kafka deploy on the square cluster
• Preops DP0.2 preparation (all):
  • Ken: deep drilling field just waiting for pandas fix, will proceed when ready
• Review CI (https://chronograf-demo.lsst.codes/):
  • nothing noteworthy
• Review outstanding action items
  • we are going to drop the "add template building to CI" as we think it will make CI too big.  maybe consider a new CI job when we have incremental template building code...
• QA meeting in November:
  • Spencer performance metrics
  • Chris: short summary of integration of solar system association?
    • would be good in the future to get Ari to take a look at HSC/COSMOS outputs (+ trailed source fitting)
    • Chris: ap_verify_hits does run right now (JP: with some README updates), could get some ssobject #s, would run on one node in a day or two. 
  • Meredith: HiTS run w/ fakes? – stuck on DM-31769 & availability
  • Ken: saha bulge? – stuck on DM-31769 & availability
  • Meredith notes on SSP data processing hack days
• AOB
  • fill out sprint planning

Action Items