AP Pipeline Meeting, 2023-08-21

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Erin Howard 
• Ari Heinze 
• Bruno Sanchez 
• Krzysztof Findeisen 
• Nima Sedaghat 
• Brianna Smart 
• Meredith Rawls 
• Kenneth Herner 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • JSR this week
• Auxtel processing status (all):
  • last run decent from a PP perspective but weather wasn't great
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • DM-33453 - Getting issue details... STATUS  
    • consensus is to go ahead with the change, no easy way to raise a warning
    • doesn't need a new RFC because it's implementing RFC-775
  • doPackageAlerts and its successors
    • Eric: doPackageAlerts serves as an integration test
    • Krzysztof: ap_verify wants to assume that ap_pipe is taken as-is, so creates maintenance
    • KF: why don't unit tests fill the need?  Eric: worried about Avro
    • KF: there is a roundtrip to Avro in https://github.com/lsst/ap_association/blob/main/tests/test_packageAlerts.py
    • Eric: need to double check that actually handles the schemas in a realistic way; alert_packet doesn't appear to be used?   If we can satisfy ourselves that our unit tests are comprehensive we can retire this doPackageAlerts unit test
      
      • Eric Bellm dig into alert serialization test in ap_association and ticket ways to improve it 11 Sep 2023 
        
  • DIASource centroid algorithms
    • Claire thinks DIASource centroid is from the dipole fit; Eric/Ian not sure
    • John is looking into this
  • John P: Mask planes on diffims: should we rename the masks that come from the template to e.g. TEMPLATE_BAD  when propagating them?
    • Meredith: we only propagate NO_DATA, BAD, SATURATED at the moment from the template
    • Eric: mask bit capacity an issue?  JP: no, we've got 32, presently using ~12
    • Eric thinks it would be better to adjust how masks are propagated into the template (so BAD actually means template pixels there are BAD, etc.).  IS: compareWarp only sets BAD/SATURATED if too many inputs have those problems to correct
    • so we'll keep them
• Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
  • Ken: run comparison tool is in review.  also testing quantum-backed butler with diffim sprint dataset–no obvious failures
  • Erin: back from Chile!
• Satellites (Meredith Rawls ):
  • Meredith: talked to Adam, Claire, John, others–have basic implementation of streak masking in single-frame, have to figure out some implementation issues with John's new CalibrateImagesTask
    • Claire's version runs on the detection plane
  • Meredith: now has a huge list of DECam images with streaks
  • KF: is this a plugin?  MR: directly run, at present
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • Bruno:
    • starting new processing runs that use the middle-seeing templates
    • playing with source_injection, but struggling a bit with local tests on ci_datasets, and with the auto-rewrite pipeline
• alert distribution (Brianna Smart , Eric Bellm ): 
  
  • Bri: have access to the USDF Argo, doing experiments on topic creation and alert production
• TransiNet (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • Nima: trying to copy out some of DC2 repo to GCP; some hiccups but some progress
    • (some discussion of whether entire DC2 is at USDF–run 2.2i is, plus user collections)
    • some discussion: we're run AP-mode on tract 4431 only
  • Nima: has ticket (DM-39821) for design discussion–on ticket or live? 
• Solar-system processing (Ari Heinze ):
  • Ari: returning from vacation.  Blocked in python wrapping because DIASources don't have all the columns he needs.
    • Eric will try to sync up with Ari later this week
  • otherwise will work on ISO
• Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • no changes
• Review outstanding action items
• QA meeting September 11:
  • CalibrateImagesTask in AP (John)
  • APDB run comparison (Ken)
  • streak masking/long trail rejection?  If visuals ready (Bri & Meredith)
  • Eric on detection thresholds/fakes forced photometry?
• AOB
  • John P: numSciSources meaning will change after calibrateImagesTask merges
    • Eric: let's disconnect it, can rewire it later to use the correct catalog in an afterburner

Action Items