AP Pipeline Meeting, 2020-05-11

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• John Swinbank
• Unknown User (cmorrison)
• Krzysztof Findeisen
• Spencer Nelson
• Unknown User (manninga)
• Unknown User (gkovacs)
• Meredith Rawls
• Ian Sullivan

Topics for discussion:

• Gen 3 conversion (all):
  • remaining tickets are mostly on Krzysztof.  DM-21915.  Problem is that we're using obs_test, which is not currently supported under Gen 3. 
    • options: convert datasets to use obs_lsst for testing–adds significant blocker.  Other option: no gen3 specific testing right now, but leads to later breakage.
    • other work is not blocked on Gen 3 (although Chris M reminds us that ; so further delay is not an issue, just a psychological sadness.
    • JDS suggests we just go ahead and do the test work now; Eric agrees–have Krzysztof report back again next week on how long it seems like it will take

• Precursor processing (Meredith Rawls ):
  • Starlink ~done (to Pub Board)
  • still working on the non-g-band HSC data–needs an extended block to keep it all in her head
• report from NCSA (Unknown User (emorganson)  + Unknown User (manninga) et al):
  • nothing to report
• Alert Distribution (Spencer Nelson ):
  • demoed the alert stream simulator tool for John and Eric Friday; Chris reviewed it, so it's merged.  A couple more minor tweaks (and better test datasets) to get reviewed, then it's on to other work
• ap_association/SDM/fakes/alerts (Unknown User (cmorrison)):
  • alert creation ticket in review, wrapping up today.  Still can't merge because it's blocked by the lsst.alert.packet merge (DM-24500–Spencer and John to push this ahead)
  • then on to fakes, and then add more contents to alerts
  • Eric heard today that Dave Berry (AST author) will be contracted to add support for an enhanced gWCS (STSci) to AST–this requires some modifications to the standard, etc.  In principle  this might mean we could distribute afw exposure FITS that astropy could read the WCS of–but the rest of Exposure would still have our weird blobs for PSFs etc.  John thinks probably we should go ahead with CCDData RFC for now
• DCR (Ian Sullivan ):
  • ran tests tracking down the
  • now on to unit tests for assembleCoadd
• Image differencing (Unknown User (gkovacs) ):
  • working on ZOGY bugfixes.
  • Ran across David Reiss's old "image mapreduce"–piecewise constant solve.  Gabor points out that these still need the FT of the full image–better to include spatial variation in main task
  • returned to long-stalled RFC-638 on mask propagation.  suggested two potential solutions in the RFC.  Eric suggests following up directly with Colin and Robert.
• Review CI (https://chronograf-demo.lsst.codes/):
  • Ratio of DIASources to Direct Image sources dropped–but DIASource metrics didn't change. This means the SFP processing changed.
    • specifically this is due to DM-23078 - Getting issue details... STATUS which added a bunch of fake sources, artificially inflating the denominator.
    • created DM-24855 - Getting issue details... STATUS to fix it
  • Runtime has oscillated back down with the updated conda environement--K-T has done some investigation, without much resolution so far.
• Review outstanding action items
• AOB
  • Reviews
  • Is there something useful we could be doing with sky sources?

Action Items