2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020



Topics for discussion:

  • Project updates (Eric Bellm , Ian Sullivan ):
    • JSR this week
  • Auxtel processing status (all):
    • last run decent from a PP perspective but weather wasn't great
  • Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
    • DM-33453 - Getting issue details... STATUS  
      • consensus is to go ahead with the change, no easy way to raise a warning
      • doesn't need a new RFC because it's implementing RFC-775
    • doPackageAlerts and its successors
      • Eric: doPackageAlerts serves as an integration test
      • Krzysztof: ap_verify wants to assume that ap_pipe is taken as-is, so creates maintenance
      • KF: why don't unit tests fill the need?  Eric: worried about Avro
      • KF: there is a roundtrip to Avro in https://github.com/lsst/ap_association/blob/main/tests/test_packageAlerts.py
      • Eric: need to double check that actually handles the schemas in a realistic way; alert_packet doesn't appear to be used?   If we can satisfy ourselves that our unit tests are comprehensive we can retire this doPackageAlerts unit test
        • Eric Bellm dig into alert serialization test in ap_association and ticket ways to improve it  
    • DIASource centroid algorithms
      • Claire thinks DIASource centroid is from the dipole fit; Eric/Ian not sure
      • John is looking into this
    • John P: Mask planes on diffims: should we rename the masks that come from the template to e.g. TEMPLATE_BAD  when propagating them?
      • Meredith: we only propagate NO_DATA, BAD, SATURATED at the moment from the template
      • Eric: mask bit capacity an issue?  JP: no, we've got 32, presently using ~12
      • Eric thinks it would be better to adjust how masks are propagated into the template (so BAD actually means template pixels there are BAD, etc.).  IS: compareWarp only sets BAD/SATURATED if too many inputs have those problems to correct
      • so we'll keep them
  • Precursor processing & Campaign Management ( Kenneth Herner , Erin Howard):
    • Ken: run comparison tool is in review.  also testing quantum-backed butler with diffim sprint dataset–no obvious failures
    • Erin: back from Chile!
  • Satellites (Meredith Rawls ):
    • Meredith: talked to Adam, Claire, John, others–have basic implementation of streak masking in single-frame, have to figure out some implementation issues with John's new CalibrateImagesTask
      • Claire's version runs on the detection plane
    • Meredith: now has a huge list of DECam images with streaks
    • KF: is this a plugin?  MR: directly run, at present
  • Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
    • Bruno:
      • starting new processing runs that use the middle-seeing templates
      • playing with source_injection, but struggling a bit with local tests on ci_datasets, and with the auto-rewrite pipeline
  • alert distribution (Brianna Smart , Eric Bellm ): 
    • Bri: have access to the USDF Argo, doing experiments on topic creation and alert production
  • TransiNet (Nima Sedaghat Harshit Rai , Eric Bellm ):
    • Nima: trying to copy out some of DC2 repo to GCP; some hiccups but some progress
      • (some discussion of whether entire DC2 is at USDF–run 2.2i is, plus user collections)
      • some discussion: we're run AP-mode on tract 4431 only
    • Nima: has ticket (DM-39821) for design discussion–on ticket or live? 
  • Solar-system processing (Ari Heinze ):
    • Ari: returning from vacation.  Blocked in python wrapping because DIASources don't have all the columns he needs.
      • Eric will try to sync up with Ari later this week
    • otherwise will work on ISO
  • Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
    • no changes
  • Review outstanding action items
  • QA meeting September 11:
    • CalibrateImagesTask in AP (John)
    • APDB run comparison (Ken)
    • streak masking/long trail rejection?  If visuals ready (Bri & Meredith)
    • Eric on detection thresholds/fakes forced photometry?
  • AOB
    • John P: numSciSources meaning will change after calibrateImagesTask merges
      • Eric: let's disconnect it, can rewire it later to use the correct catalog in an afterburner

Action Items

DescriptionDue dateAssigneeTask appears on
  • Eric Bellm write up retention policy and sizing for APDB  
17 Jul 2023Eric BellmAP Pipeline Meeting, 2023-06-26
  • Eric Bellm dig into alert serialization test in ap_association and ticket ways to improve it  
11 Sep 2023Eric BellmAP Pipeline Meeting, 2023-08-21