AP Pipeline Meeting, 2024-01-08

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm
• Erin Howard 
• Brianna Smart 
• John Parejko 
• Krzysztof Findeisen 
• Hsin-Fang Chiang 
• Ian Sullivan 
• Meredith Rawls 
• Bruno Sanchez 
• Nima Sedaghat 
• Kenneth Herner 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  
  • Happy New Year!
  • 2024-01-08 M1M3 Decision Point Meeting
  • AHM. Possibly 1 February. Planning on setting quarterly dates thereafter.
  • Rubin Zoom being rolled out - watch meeting invites and cancellations
  • DIA Science Unit members
  • submitting TRs for JTM
  • sprint planning schedule
• Auxtel processing status (all ):
  • on hiatus, on-sky date uncertain
• Pipelines Infrastructure (John Parejko, Krzysztof Findeisen ):
  • John: waiting on CalibrateImage reviews
  • Krzysztof: with Nima, got the RB models into /repo/main; not quite ready for use but coming soon.  will migrate into ap_verify datasets from /repo/main
• Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
  • Erin: trying some small test runs on Saha Bulge, which is creating a lot more WCS failures than before.  Moving backwards to try to find if there's a weekly that works.  Also, QG generation is now taking well over 90 minutes now.  John: was working on some known slowdowns with Jim, starting w47 or so.  Erin: slow even in w44.  KF: can middleware report how long generation takes?  Erin: there is.  Ian: Jim B noted butler slowdown end of the calendar year, which he's working on. Seems to be related to gaia refcat in /repo/main?  Hopes some improvement when duplicate gaia refcat is removed.  Ken: happy to look also at failures.
• Satellites (Meredith Rawls ):
  • mainly making AAS slides.  requested a session at the JTM to discuss streak details
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • Bruno: fighting with mask planes, trying to close out old tickets before returning to source injection
    • Open questions around how to make the fakes pipeline,
    • Shu & MWV trying to figure out
  •  DM-42353
    • Eric: statement was the diasources didn't change, but the count differs in the analysis notebook?
    • Krzysztof: metrics are from the source catalogs, and didn't change
    • Ian: we didn't lose anything, added sources.  these are coming from the APDB
  • Ian: John put in a bugfix to make sure the offimage flag was set for nan centroids, which helps with a downstream bug in preconvolution, rbClassifier.  Does not seem to have affected our metrics.
    • John: why are these happening at all?  Ian: seems like these are mostly sky sources near the edges, centroids walking off (even with the
  • Ian: have a ticket in review with Bruno to reject some sources in detectAndMeasure based on flags, and soon the centerAll flag–will cut the sources reaching the APDB.
• alert distribution (Brianna Smart , Eric Bellm ): 
  
  • Bri: still working on getting credentials into the pipelines, have to meet with Square Thurs.
  • Bri: also investigating large alerts; stamps are large but without science image it's a bit tough to diagnose
• ML (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • mainly working with Krzysztof on ingesting models at USDF
• Solar-system processing (Ari Heinze, Kurlander Jake , Mario Juric ):
  • out today (AAS)
• Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • nothing beyond the DM-42353 changes
• Review outstanding action items
• QA meeting February 5 (TBC--JTM):
  • streak masking (Meredith)
  • summarize all of the stack changes (flags & filtering) (all)
  • DIA Science Unit organization.
• AOB
  • John: need to sit down with Ian & Krzysztof about preloading–there are some challenges with butler dataset creation because we don't have all the relevant information.  KF: bring someone from middleware
  • no meeting next week: MLK day

Action Items