There have been requests for the QA plots produced by pipe_analysis to be more easily inspected and dynamic.  Specifically, the following features have been discussed:

  • Mouseover inspection of dataId info of points
  • Ability to easily change the magnitude range of the statistic-relevant range (including showing only the marginalized histogram of subsets of the data)
  • Dynamically coloring points by selectable criteria
  • Linking scatter and spatial plots such that highlighting is linked
  • Possible table display of selected points? 

Bokeh is a good way to implement this.  I am running a prototype server at localhost:5006 on lsst-dev, which you can pipe back to your own machine by `ssh -N -f -L localhost:5006:localhost:5006 <user>@lsst-dev.ncsa.illinois.edu` and then open localhost:5006 on your browser.  

Below is a rough plan for updating pipe_analysis to be able to serve a server like this:  

  1. Identify a minimum number of dynamical plots required to reproduce the crucial information content of the current static .png figures.
  2. Identify the minimum data configuration needed to support these plots (e.g., a number of dataframes collated from selections on source catalogs, etc.)
  3. Write a task/script in pipe_analysis that simply generates and persists this data, rather than producing plots.  The idea is that this can be run once to produce the necessary summary data, and that plot calls (whether static or dynamic) on top of this will not be time-consuming.  
  4. Implement a prototype bokeh server app to host these plots, with layout/content more or less what we are looking for. I note that I do not intend the precise implementation of this to be production-robust; rather, to be useful ASAP to developers, to explore ideas, and to serve as a prototype for someone else down the line to do it better.  And I do intend these updates to pipe_analysis to be able to produce/deliver the data that such a system would use.


In addition to the above improvements, more nuts & bolts things to do with pipe_analysis include:

  • Pull in any remaining improvements from Paul Price.
  • Make sure LSST coding conventions are followed
  • Move the package to the master branch of some LSST org repo (or just migrate it from lsst-dm to lsst when appropriate?)
  • Somehow write tests for this?  (How would this work, as it requires a rerun to read in order to run?)
  • No labels

3 Comments

  1. Hey Tim, SQuaRE and in particular Angelo Fausti, has build up a lot of experience with Bokeh over the last year with https://squash.lsst.codes, including scatter plots and histograms for datasets behind the verification metric measurements (not everything is deployed intro production yet). Feel free to reach out.

    1. Unknown User (tmorton)

      Great, thanks! A pointer to a person who has had lots of experience with Bokeh is super helpful; I will be sure to ping Angelo with questions as they come up.  

  2. Hi Tim, nice, let's keep in touch. I have some examples of plots using bokeh in this repo https://github.com/lsst-sqre/bokeh-plots and more specifically in the squash product https://github.com/lsst-sqre/qa-dashboard/tree/master/squash/dashboard/viz let me know if you have any question.