AP Pipeline Meeting, 2020-06-29

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Unknown User (cmorrison)
• Unknown User (gkovacs)
• John Swinbank
• Krzysztof Findeisen
• Meredith Rawls
• Unknown User (emorganson)

regrets:

• Spencer Nelson
• Ian Sullivan

Topics for discussion:

• sample alerts for brokers (Eric Bellm , Unknown User (cmorrison) , Spencer Nelson ):
  • Eric Bellm finishing push of change controlled documents (LDM-612 and 723) to RFC, will close README ticket once sample alerts are done.
  • Unknown User (cmorrison) : everything is hooked up for templates, writing warped exposures to disk through the butler, just needs to track down some probably naming bugs
• Gen 3 conversion (all):
  • Krzysztof Findeisen: still in progress, no blockers

• Precursor processing (Meredith Rawls ):
  • did a mini-test run on HSC, still failing but thinks she misdiagnosed why previously.  Thinks its failing to find templates around edges?  Needs to make some spatial plots to investigate.  Also hitting ap_db error with physical filter names (rather than abstract filters)–believes there is an existing ticket to address this, will track it down.
  • still needs to look at June rerun in more detail.
  • a few weeks out from the monthly QA extravaganza, will prep after vacation
• report from NCSA (Unknown User (emorganson)  + Unknown User (manninga) et al):
  • nothing relevant
• Alert Distribution (Spencer Nelson ):
• ap_association/SDM/fakes/alerts (Unknown User (cmorrison)):
  • (discussed above)
  • points out the SST needs to think harder about what purpose the alert cutouts need to fulfill–currently no real way to get calexp from other two cutouts in a self-consistent way
• Crowded field photometry (Ian Sullivan ):
  • vacation this week
• Image differencing (Unknown User (gkovacs) ):
  • will circulate the DMTN by end of the day for RHL discussion tomorrow
  • then Ian's review
• Review CI (https://chronograf-demo.lsst.codes/):
  • ratio of DIASources to direct image sources changed, as expected: DM-24855 excluded Sky Sources from the metric.  Already helpfully annotated by Krzysztof Findeisen
  • Memory usage dropped about DM-24325, which replaced afw exposures in cutouts with CCDData.  Unknown User (cmorrison) theorizes this is because we could drop the afw Memory Managers for more pythonic fits handling
    • Krzysztof Findeisen cautions us that the metric is only reporting peak  memory usage across all tasks.  Last time he profiled memory-by-task (pre-alert-gen), diffim was the largest memory user, so this suggests that alert gen is now the memory hog.
• Review outstanding action items
  • John Swinbank to informally ask KT about whether running ap_verify against arbitrary ticket branches is something we can easily ask for; if not, Eric Bellmwill propose a DMLT f2f session about Jenkins more broadly 13 Jul 2020 
• AOB
  • Chris M has a butler question–no help on this call, off to Slack

Action Items