2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020



Topics for discussion:

  • Project updates (Eric Bellm , Ian Sullivan ):
    • international logistics
    • DIA sprint
      • B305 is reserved 9-noon TuThFriMonTue; 9-11 Wednesday
      • sprint demo Weds 22
      • bright star mask rising in priority
      • Fred M. identified some improvements to the deblending being used in diffim–will deblend fewer things but do them more correctly.  Future algorithmic development would be needed to work on dipoles
  • Auxtel processing status (dashboard):
    • new data May 28
  • Priorities from now-ComCam (Eric Bellm ):
  • Pipelines Infrastructure (John Parejko, Krzysztof Findeisen ):
    • John: working with Jim on getTemplateTask
      • thinks he has a version that's 2-3x faster
      • exposed some deeper questions about how warping messes up the variance plane–seems to increase variance by larger factors than expected
    • Krzysztof: need to figure out who will set up preload task, which is now blocking
  • Precursor processing & Campaign Management ( Erin Howard, Kenneth Herner ):
    • Erin: made data for diffim sprint, now returned to HSC reprocessing: trying to understand new errors, create bug reports if needed
      • also going to rerun OR3 reprocessing on "real" embargo rack
  • Satellites (Meredith Rawls , Brianna Smart ):
    • Eric: security review scheduled July 29-30.   Need to get secret satellite filtering going ASAP.  Will try to set up a meeting with Bri, Ari, etc. to scope out  ...
    • Meredith requests to sit in
  • Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan, Neven Caplar, Audrey Budlong ):
    • Bruno: working with Lee on the fakes.  Lee has been trying to make the pipeline modification script work for AP.  Lee wants to move injection prior to calibration to help with contracts, but we all think we need to keep the injection known in physical units
      • John: doesn't think INJECTED planes are yet ignored during calibration
      • Bruno: with this change, could build injection pipeline automatically with scons
      • Bruno: has been working on matching with Trys.  source_injection built all their tools for tract, patch, though, so requires .  Good working with Trys on analysis_tools completeness plots.  But need to aggregate to get sufficient data for completeness ahead of metrics.  Matching could also be done in the afterburner?   Which would help with dynamically creating the pipeline
      • KF: remember that ap_verify doesn't do whole focal planes in our CI datasets, just combines 3-4 detectors
      • John: is injection in electrons or in physical flux.  We agree that we need to know that physical fluxes, and the injection should happen after calibration
        • KF, all: we should change the contracts to reflect what the pipelines need to do
    • Bruno: almost done updating CI datasets with injection catalogs.
    • Ian: working on sdm_schema enforcement on pandas to avoid Cassandra problems, takes the apdb url
      • Meredith: DRP uses transformDiaSource as well...
  • alert distribution (Brianna Smart , Eric Bellm ): 
    • Bri: blocked on new test alerts with brokers because /repo/dc2 doesn't have group day_obs.  Otherwise planning to let brokers to try to test throughput through 40k alerts vs. various partition configs
    • discussion of alert rates–we're still very high on number of
  • ML (Eric Bellm,Harshit Rai):
    • Harshit:
      • modified plotImageSubtractionCutouts to save as numpy arrays, now cleaning up bugs. 
      • Now testing model from Tatiana
      • Next will work on assigning truth table values from DC2
  • Solar-system processing (Ari Heinze, Kurlander Jake , Mario Juric ):
    • running heliolincc with more bad detections: up to 3k a visit tolerable
    • thinking about point vs. trail–very sensitive to NEO completeness
  • Review CI (https://usdf-rsp-dev.slac.stanford.edu/chronograf/):
  • Review outstanding action items
  • QA meeting (June):
  • AOB
    • no meeting next week (US holiday)

Action Items

DescriptionDue dateAssigneeTask appears on
  • Eric Bellm write up retention policy and sizing for APDB  
17 Jul 2023Eric BellmAP Pipeline Meeting, 2023-06-26
  • Eric Bellm dig into alert serialization test in ap_association and ticket ways to improve it  
11 Sep 2023Eric BellmAP Pipeline Meeting, 2023-08-21