AP Pipeline Meeting, 2023-07-24

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Ian Sullivan 
• Krzysztof Findeisen 
• Erin Howard 
• Kenneth Herner 
• Meredith Rawls 
• John Parejko 
• Nima Sedaghat 

regrets

• Eric Bellm 

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • DMTN-199 was released to the public.
  • Jenkins authentication is broken, and everyone with admin privileges is on vacation until next. Ping Tim Jenness if you need to run a ticket branch on Jenkins.
• Auxtel processing status (all):
  • Another Auxtel run next week, running ApPipe again
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • KF merged DM-39412 that adds metadata to AP CI runs
    • Now we can look up on Chronograph the CI run ID for any datapoint. Link to download logs.
• Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
  • • KH: note that diffim sprint runs do not have spuriousness columns in the APDB (run prior to switch to reliability). Columns are None
    • EH: Noticing some interesting behavior. Restarted a run, so did not have to re-generate the quantum graph, but the run is still hanging with no progress.
      • Similar behavior with the Saha Bulge dataset over the weekend, one quantum stuck hanging all weekend
      • Meeting with Orion later to discuss Panda and CM tools
    • EH: leaving for Chile on Friday!
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • BS: merged DM-39467 to increase the density of fake source injections.
    • Preparing a larger run with many more detectors than was used for the DMTN
  • BS working with DM-32889 to propagate the injected fakes mask plane into the diffim.
  • BS started putting together slides for PCW
  • IS investigating potential improvements in preconvolution by replacing masked saturated/edge pixels
• alert distribution (Brianna Smart , Eric Bellm ):
  • BS: everything's broken! The setup for Vault secrets have changed, and the distribution system can't feed in new alerts. Need to contact Square.
• transinet (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • NS: DM-39833 sought good examples of cutouts to use for human classification, but dynamic range easily saturated.
    • Would be useful to have ground truth list of sources in the DC2 ap_verify dataset. Found sources that look legit, but are not in the truth list.
      • IS: could you be using the pipeline with fakes?
• Meredith
  • MR: Working on adding trailed detections to AP pipeline
• Solar-system processing (Ari Heinze ):
  • Absent
• Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • Expect to see an increase in timing in at least DC2 with the new Rubinenv 7 with python 11, followed by a decrease recently. No corresponding changes seen above the noise.
  • There is a dip in SubtractImages timing that could correspond to the changes, but appears to be in the wrong direction.
  • KF will add an annotation for the environment change.
• Review outstanding action items
• QA meeting September 11:
  • Eric on detection thresholds/fakes forced photometry?
  • streak masking/long trail rejection (Bri & Meredith)
  • CalibrateImagesTask in AP (John)
• AOB
  • Eric will be on vacation next two weeks, Ian will run the meeting

Action Items