AP Pipeline Meeting, 2023-10-09

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm
• Ian Sullivan 
• Kurlander Jake 
• Krzysztof Findeisen 
• Erin Howard 
• John Parejko 
• Neven Caplar 
• Brianna Smart 
• Nima Sedaghat 
• Brianna Smart 
• Meredith Rawls 
• Bruno Sanchez 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • All Hands Friday: ComCam likely coming back into the plan, on-sky observing July?  So aim for functional completeness of our pipelines on that timescale
• Metrics Sprint wrap-up
  • closeout discussion at Science Pipelines Team Meeting Wednesday
  • slides: https://docs.google.com/presentation/d/12Kj0F9zRccQ-7HeTgxxlLwUYrrarS7GJrOVxiqIj0tE/edit?usp=sharing
• Auxtel processing status (all):
  • some problems last run, hopefully patched for tomorrow night
  • Neven: no blockers on analysis
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • Krzysztof returned from vacation
  • pandas v2.1 breaks ap_association
    • John: can we just use np.hstack?
    • Eric: worried about fiddling with pandas in ap_association–if we were to take it out it needs a top-to-bottom reworking
  • APDB versioning: how much do we care about backwards compatibility now (as opposed to in the future)? DM-41029, DM-41028
    • apdb versioning is coming
    • John wants to raise if there's no version
      • KF says as long as we can retroactively add versioning to older APDBs (such as the PPDB).  We can use AP Precursor Processing Runs to decide which to tag
    • some discussion of schema vs. pipeline versioning
• Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
  • Erin: not much from CM, they are adjusting documentation and working on HTCondor
  • processing runs: filled home directory with serialized alerts, completed a run with RB scores
• Satellites (Meredith Rawls ):
  • "they launched some more"
  • workshop in the Canaries last week, will be
  • ticket for streak detection got intermixed with sprint work–trying to clean that up before putting in review (w/ Bri)
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • Bruno: some time on metrics sprint, aiming to close it out
  • Ian: just the metrics sprint
• alert distribution (Brianna Smart , Eric Bellm ): 
  
  • Bri: RFC opened a bit more discussion than expected, so blocked on that to produce alerts
• ML (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • Nima: restored an old DC2 repo on GCP VM.  Noticed potential changes between old processed version.  Tried to figure out what weekly he'd used to run the stored version by comparing saved calexps to regenerated ones with several weeklies: large background gradient, and some astrometric shifts
    • Eric thinks this could be simply be different configurations in the original processing, Occam's Razor here
    • Nima: rerunning new model on newest
• Solar-system processing (Ari Heinze, Kurlander Jake  ):
  • they are prepping for AuxTel integration
• Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • No new changes
• Review outstanding action items
• QA meeting November 6:
  • metrics sprint report
  • streak masking (though Meredith may not be available to present due to ADASS)
• AOB
  • Eric out now-Friday (conference)
  • John and Erin going eclipse hunting, Ian out Friday-Monday

Action Items