AP Pipeline Meeting, 2022-05-16

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• John Parejko 
• Krzysztof Findeisen 
• Nima Sedaghat 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • two new AP team members!
    • Bruno Sanchez (at IN2P3)
    • Brianna Smart (at UW, Aug 1)
• Pipeline Performance sprint (Eric Bellm , John Parejko , Meredith Rawls ):
  • kicked off today
  • Umbrella ticket: DM-34623 - Getting issue details... STATUS
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  • Krzysztof: almost got a refactor of the ap_pipe pipeline ready to merge.  Next working on standardization of annotations–Dashboards will have a link to the confluence.
  • "ap_verify in Prompt Processing": there is a successor to Squash planned for operations ("Sasquatch", still influxDB/Chronograf).  Let's discuss what it would take to push metrics from PP.
    • KF: not true that metrics handling is baked into ap_verify!  In gen3, metrics are handled within the pipelines themselves.   Currently pipelines with metrics live in ap_verify, but no reason we couldn't compute metrics in prompt processing (at cost of additional latency etc.)
    • uploading to squash might be harder: square worried about
    • in prompt processing, could defer some of this to a central worker/service
      • ap_pipe could be minimally instrumented (e.g., timing, very cheap metrics), with the rest as afterburners
        • KF: right now metricTask has to re-read all of the inputs–middleware had been thinking about caching of input datasets
        • EB: timing metrics should be fast?  KF: still have to read in the metadata, but it should (?!) be fast.
• Precursor processing (Meredith Rawls, Kenneth Herner ):
  • Ken: ready to test refactored diffim when it's available.
  • Meredith: testing Krzysztof's refactor, then ready for HiTS rerun (+ performance sprint)
• Image differencing algorithms, DCR  ( Ian Sullivan ):
  • replicated diffim in standard configuration to machine precision w/ refactored diffim
  • a few differences in other places, but due to bugs in original ip_diffim
  • running on HiTS, hit some bugs due to metadata name changes
    • JP: we now don't have imageDifferenceTask.  so timingImageDifference is looking for metadata that doesn't exist. 
    • KF: so we'll need to create some new metrics.
    • JP: thinks there are some name changes (anything with ImageDifference).  Old image difference will continue to work with existing configs, but we'll need to
    • JP: had to manually exclude any diffim metrics due to failures.  KF: thinks it's a bug, likely in verify–should be None rather than fail if Task doesn't run.  Has DM-30869 - Getting issue details... STATUS to use new optional inputs facility
    • still trying to run full ap_pipe, failing somewhere in dia_pipe with some kind of filterName issue.  Hopes to try to debug this later this week if others are available.
  • filterLabel warnings: KF: we need to update the templates to newer versions.
  • also, butler uses alphabetical task names to order tasks it thinks can happen at the same time, which messed up QG generation (since getTemplateTask only needs a portion of the calexp, not the whole data product)
    
    • getTemplateTask fails, retrieveTemplateTask works
    • DM-34811.  see also DM-32725 - Getting issue details... STATUS
• transinet (Nima Sedaghat, Eric Bellm, John Parejko ):
  • trained transinet now finding SNe in science & templates from DC2
  • was actually hard to verify injected SNe by eye at all!
  • JP reminds us that DC2 sims injected to very faint magnitudes–so delta flux is not enough to be sure that it's detectable
  • Nima: wants to understand how to apply calibration to the two images
    • JP: sent the command
    • NS: need to understand how to inject sources on the fly.  JP: it's just nJy after calibration.  We don't know what delta flux is from DC2
  • started to design interface for applying classification to DIASources, working with JP on interfaces
• alert distribution (Eric Bellm ):
  • no update.  Frossie reminds me that the service is bitrotting–square has changed phalanx configurations
• Review CI (https://chronograf-demo.lsst.codes/)
• Review outstanding action items
• QA meeting in June:
  
  • Performance sprint update
  • Report on diffim refactor
• AOB
  • none

Action Items