AP Pipeline Meeting, 2023-05-15

2pm Pacific Time

Zoom: https://washington.zoom.us/j/98846655020

attending:

• Eric Bellm 
• Ian Sullivan 
• Erin Howard 
• Brianna Smart 
• Krzysztof Findeisen 
• Harshit Rai 
• John Parejko 
• Meredith Rawls 
• Kenneth Herner 
• Bruno Sanchez 
• Ari Heinze 

regrets

Topics for discussion:

• Project updates (Eric Bellm , Ian Sullivan ):
  • reminder that summit will be offline beginning of June, SLAC shutdown last week of June (following meeting-free week)
  • DRP doing a large RC2 run at USDF, may create resource issues over next few weeks
    • Erin thinks DM-38910 - Getting issue details... STATUS may not make it for the bootcamp.
    • see #ops-cm-team
  • SITCOM bootcamp next week
• Pipelines Infrastructure (Krzysztof Findeisen , John Parejko ):
  
  • Krzysztof Findeisen : Square is not going to fix squash, and we should migrate to Sasquatch.  KF is developing a way to migrate ap_verify metrics into sasquatch.  Challenge is that the data model is a bit more rigid than in squash–we lose the ability to add/ratio in the way we currently do in our Chronograph dashboard.
    
    • in InfluxDB, data are measurements with timestamps, which may have one or more fields.  Chronograph makes it very hard to do things between different measurements.  these are called Sasquatch metrics, analysis_tools Actions.
    • Sasquatch is based on a newer version of Influx.
    • metric uploads need to have consistent (or backwards compatible) schemas
    • Ian: what if we punt everything but the derived metrics?  KF: most of the metrics we use on the dashboard are derived, and
      • DRP don't have any timing, or any derived quantities
    • Eric: maybe we just get a dashboard up with the metrics that work, and implement replacements/improvements from there
      • Ian agrees, think of June sprint for metrics to build/spread knowledge.  Triage for now & start building history
      • don't attempt to recover "derived" quantity
  • John Parejko :  not sure how to resolve Colin's question in RFC-924 about deprecation.  Added "decl" duplicate columns, but don't have visibility into downstream user code
• Precursor processing ( Kenneth Herner , Erin Howard , Meredith Rawls ):
  • Erin Howard : issue last week that shut down and deleted all runs, which was pretty rough–lots of lost effort.  Leaky resource at USDF, problem understood.  (Ian: it was blocking Auxtel run due to storage running out).  On restart, lack of resources means processing is very slow
  • Kenneth Herner : progress understanding the firefly issue
  • Meredith Rawls : have been looking at AuxTel diffim–they exist!  but they are rather messy.  Shallow, lots of NO_DATA gaps (but flagged correctly).  report coming later this week
• Image differencing algorithms, DCR  ( Bruno Sanchez , Ian Sullivan ):
  • Bruno Sanchez finished analysis comparing the three runs, wrapping up the report, did a review for Jim Bosch for the preconvolution technote
  • Ian Sullivan : need to rebase several open ip_diffim tickets, including ImageDifferenceTask. Source detection will no longer detect on bad pixels (BAD, EDGE, NO_DATA), removes 30-50% of junk
    • JP: how does this compare to CenterAll flag proposal?  JP wants to chat
    • Nima: is this configurable?  Ian: which mask planes are rejected is configurable
• alert distribution (Brianna Smart , Eric Bellm ):
  • Brianna Smart : Dan Speck continuing to run throughput tests and break things. 
    • got local version of kafdrop working for the broker  but too slow–needs deployed web version of kafdrop set up at USDF in phalanx
• transinet (Nima Sedaghat Harshit Rai , Eric Bellm ):
  • RB scoring for DC2 now in CI
  • some initial Zooniverse run and documentation at https://www.zooniverse.org/projects/ebellm/rubin-observatory-alerts--everyone welcome to label.  This has most artifacts weeded out (via RB) but does have sub-threshold events (so lots of "noise"-looking images)
  • Ari think best to use the same (linear) stretch for all three cutouts
• Solar-system processing (Ari Heinze ):
  • Ari Heinze : no news, but now running at USDF
• Review CI (https://chronograf-demo.lsst.codes/):
  • no chronograf
• Review outstanding action items
• QA meeting in June:
  • RB scoring
  • prompt processing status through two AuxTel runs
  • edge flags? (& vignetting)
  • off-chip centroids – JP thinks it would be good to dig into why first
  • Eric on detection thresholds/fakes forced photometry?
  • Discussion of AP QA needs in the future that Commissioning/LINCC might assist with
  • preconvolution – this still needs some digging in to understand the edge behavior
• AOB

Action Items