2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020



Topics for discussion:

  • Project updates (Eric Bellm , Ian Sullivan ):
    • fill out sprint planning
  • Auxtel processing status (all):
    • Ian showing some images from PP from the last run.  Lots of structured noise, but diffim looks pretty clean
    • Bruno: doesn't look like any saturated stars, which helps
    • next AuxTel run starts tomorrow!
  • Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
    • John:
      • has big ticket moving configs out of obs_package overrides–will start a broader conversation.
      • responding to Krzysztof's review of CalibrateImagesTask in AP
  • Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
    • Ken: Long meeting at PCW from CM, nothing major for AP.  Quantum-backed butler is out and is now default as of last weekly
  • Satellites (Meredith Rawls ):
    • Meredith: trying to get Claire and Adam Synder together (perhaps in pair coding) to get some test images through streak finding algorithms–they have developed two separate streak finding algs.
    • glints–lots of discussion at PCW in the context of a proposal for multi-resolution stamps
      • John P interested exposing tile compression to astropy CCDDdata
      • headers expensive too for small stamps; gzip whole cutouts? 
  • Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
    • Bruno:
      • chatted at PCW about path forward with Science Unit; also presented in DIA session
      • source injection is highest priority. 
      • talked with Lee some and running some initial tests.  Current code takes a yaml pipeline and makes another that includes the source injection.  Not working just yet for AP
      • presented some efficiency calcs for DIA default settings, now exploring pre convolution–wants to try Ken's template sets
  • alert distribution (Brianna Smart , Eric Bellm ):
    • Bri: alert stream broker is now up at the USDF.  Bri has access but it's not through Argo so access and permissions are limited.  Seems like deployment requires a lot of hands on the terminal
    • Bri got lots of suggestion to use flags rather than nightly topics for improved stability
    • brokers said they were eager to connect, somehow didn't know about the IDF?
      • Eric Bellm review broker communication channels  
  • transinet (Nima Sedaghat Harshit Rai , Eric Bellm ):
    • Nima: chatted in Uber with Mario about SSObjects, realized that RB classifier as trained will reject trailed sources
    • Discussion of future handling of rbClassifier_data (all)
      • Ian summarizing: current git LFS holding of model requiring a number of hacks to keep working for ap_verify and others.  We should discuss how to go forward, what we've learned.
      • Krzysztof:  main issue is that it's temporary: not registered in eups, doesn't follow our git lfs convensions, etc. RFC-909 assumed git LFS implementation is temporary
        • main question is how far we are from having a butler-backed storage of the models
      • Ian: proposing we try to optimize the model for speed first, so we're not repeatedly ingesting models into the butler or changing the storage format. 
        • thinking of speed of classification rather than loading speed
      • KF agrees that if we have to change the file format it will cause more annoyance
      • Nima, other: doesn't see the connection between speed of inference and model storage format
      • Nima: still at a very exploratory phase of model development.  we will definitely have faster models, but doesn't think it's necessary to wait for a switch to butler-based storage–doesn't think the model storage will change substantially: pytorch code + a weights blob (basically a zip archive?)
      • Nima: not all the draft models have to be broadly shared
      • KF: logically RB task belongs in ap_pipe, but we can't put it there until there is a default model available from the default config
      • JP: seems nicer to have model churn in one repo rather than all of the ap_verify datasets?  KF: don't think this is any different than calibs and templates, and CCB said we had to get rid of rbClassifier data
      • sounds like we can go ahead and schedule some work on this
        • keep the status quo for current rbclassifier treatment for the next month.
  • Solar-system processing (Ari Heinze ):
    • out today
  • Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
    • no changes of note
  • Review outstanding action items
  • QA meeting September 11:
    • CalibrateImagesTask in AP (John)
    • Eric on detection thresholds/fakes forced photometry?
    • streak masking/long trail rejection (Bri & Meredith)
  • AOB

Action Items

DescriptionDue dateAssigneeTask appears on
  • Eric Bellm write up retention policy and sizing for APDB  
17 Jul 2023Eric BellmAP Pipeline Meeting, 2023-06-26
  • Eric Bellm dig into alert serialization test in ap_association and ticket ways to improve it  
11 Sep 2023Eric BellmAP Pipeline Meeting, 2023-08-21
  • Eric Bellm review broker communication channels  
25 Sep 2023Eric BellmAP Pipeline Meeting, 2023-08-14