AP Pipeline Meeting, 2023-08-14

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Brianna Smart 
• Krzysztof Findeisen 
• Bruno Sanchez 
• John Parejko 
• Kenneth Herner 
• Nima Sedaghat 
• Meredith Rawls 

regrets

• Erin Howard (La Serena)

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • fill out sprint planning
• Auxtel processing status (all):
  • Ian showing some images from PP from the last run.  Lots of structured noise, but diffim looks pretty clean
  • Bruno: doesn't look like any saturated stars, which helps
  • next AuxTel run starts tomorrow!
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • John:
    • has big ticket moving configs out of obs_package overrides–will start a broader conversation.
    • responding to Krzysztof's review of CalibrateImagesTask in AP
• Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
  • Ken: Long meeting at PCW from CM, nothing major for AP.  Quantum-backed butler is out and is now default as of last weekly
• Satellites (Meredith Rawls ):
  • Meredith: trying to get Claire and Adam Synder together (perhaps in pair coding) to get some test images through streak finding algorithms–they have developed two separate streak finding algs.
  • glints–lots of discussion at PCW in the context of a proposal for multi-resolution stamps
    
    • John P interested exposing tile compression to astropy CCDDdata
    • headers expensive too for small stamps; gzip whole cutouts? 
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • Bruno:
    • chatted at PCW about path forward with Science Unit; also presented in DIA session
    • source injection is highest priority. 
    • talked with Lee some and running some initial tests.  Current code takes a yaml pipeline and makes another that includes the source injection.  Not working just yet for AP
    • presented some efficiency calcs for DIA default settings, now exploring pre convolution–wants to try Ken's template sets
• alert distribution (Brianna Smart , Eric Bellm ):
  • Bri: alert stream broker is now up at the USDF.  Bri has access but it's not through Argo so access and permissions are limited.  Seems like deployment requires a lot of hands on the terminal
  • Bri got lots of suggestion to use flags rather than nightly topics for improved stability
  • brokers said they were eager to connect, somehow didn't know about the IDF?
    • Eric Bellm review broker communication channels 25 Sep 2023 
      
• transinet (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • Nima: chatted in Uber with Mario about SSObjects, realized that RB classifier as trained will reject trailed sources
  • Discussion of future handling of rbClassifier_data (all)
    • Ian summarizing: current git LFS holding of model requiring a number of hacks to keep working for ap_verify and others.  We should discuss how to go forward, what we've learned.
    • Krzysztof:  main issue is that it's temporary: not registered in eups, doesn't follow our git lfs convensions, etc. RFC-909 assumed git LFS implementation is temporary
      
      • main question is how far we are from having a butler-backed storage of the models
    • Ian: proposing we try to optimize the model for speed first, so we're not repeatedly ingesting models into the butler or changing the storage format. 
      • thinking of speed of classification rather than loading speed
    • KF agrees that if we have to change the file format it will cause more annoyance
    • Nima, other: doesn't see the connection between speed of inference and model storage format
    • Nima: still at a very exploratory phase of model development.  we will definitely have faster models, but doesn't think it's necessary to wait for a switch to butler-based storage–doesn't think the model storage will change substantially: pytorch code + a weights blob (basically a zip archive?)
    • Nima: not all the draft models have to be broadly shared
    • KF: logically RB task belongs in ap_pipe, but we can't put it there until there is a default model available from the default config
    • JP: seems nicer to have model churn in one repo rather than all of the ap_verify datasets?  KF: don't think this is any different than calibs and templates, and CCB said we had to get rid of rbClassifier data
    • sounds like we can go ahead and schedule some work on this
      • keep the status quo for current rbclassifier treatment for the next month.
• Solar-system processing (Ari Heinze ):
  • out today
• Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • no changes of note
• Review outstanding action items
• QA meeting September 11:
  • CalibrateImagesTask in AP (John)
  • Eric on detection thresholds/fakes forced photometry?
  • streak masking/long trail rejection (Bri & Meredith)
• AOB

Action Items