AP Pipeline Meeting, 2020-06-15

2pm Pacific Time

BlueJeans: https://bluejeans.com/426716450

attending:

• Eric Bellm
• Unknown User (cmorrison)
• Unknown User (gkovacs)
• Ian Sullivan
• John Swinbank
• Krzysztof Findeisen
• Meredith Rawls
• Unknown User (emorganson)

regrets:

Topics for discussion:

• sample alerts for brokers (Eric Bellm , Unknown User (cmorrison) , Spencer Nelson ):
  • informational update sent to brokers today, with implication that sample alerts are immanent.
  • Eric Bellm finished the 3.0 schema.  README is next–seems like a text version is preferred for now, I may make an lsstdm repo to hold it.
    • also will think about wrapping this up to look slightly nicer than an open directory...
  • Spencer Nelson has one ticket left to merge but the work is done
  • Unknown User (cmorrison) taken by surprise by the deadline–Eric failed to communicate this clearly. 
    • long-ish discussion of WCS origins for cutouts
• Gen 3 conversion (all):
  • Krzysztof Findeisen Gen 3 pipeline is in review, working on getting ap_verify to ingest datasets in Gen 3

• Precursor processing (Meredith Rawls ):
  • not much; got help fixing a breakage to ap_verify
• report from NCSA (Unknown User (emorganson)  + Unknown User (manninga) et al):
  • continuing to work on Kafka + Kubernetes
• Alert Distribution (Spencer Nelson ):
  • closed out lingering simulator alerts, now working on SciMMA work, expecting to do some support when brokers get alert stream
• ap_association/SDM/fakes/alerts (Unknown User (cmorrison)):
  • (discussed above)
• Crowded field photometry (Ian Sullivan ):
  • poking at crowded field data
  • also working on coaddition test suit, should be in review soon (found some bugs around the way)
• Image differencing (Unknown User (gkovacs) ):
  • working on tech note, won't be fully complete by tomorrow for discussion with RHL
  • thinks he identified the source of some ZOGY artifacts, will write up–doesn't really affect the detection (Scorr) image though
  • worrying about runtime for spatially-varying ZOGY–might push us towards Image MapReduce again (piecewise constant representation)
• Review CI (https://chronograf-demo.lsst.codes/):
  • DM-24438 - Getting issue details... STATUS is now merged
    • but it doesn't actually do what we wanted–only runs on ap_verify ticket branches only, rather than bringing in
    • sounds like we need to ask ARCH nicely if they can do something better–since changes elsewhere are breaking ap_verify (example Friday for ISR)
      • John Swinbank and Eric Bellm to talk to ARCH about what we should expect from Jenkins going forward
    • in the meantime we should at minimum sort out the Docker workflow to allow us to check for breakages in advance
    • Krzysztof Findeisen is not certain we should repurpose ap_verify for this purpose, is less concerned about scrambling after a breakage
    •  generically we expect devs to make sure their changes don't break things, so we want it to be easy to run ap_verify in CI on  ticket branchs
  • (did our metrics change after DRP's psfex fix in DM-25210  got merged?)
• Review outstanding action items
• AOB

Action Items