https://bluejeans.com/426716450

Agenda

  • WG report.
    • DMTN-085 is the report from this working group.
    • Basic structure of that document is in place, and will continue to be refined. Comments and suggestions welcome.
    • Expectation is that John Swinbank will handle introductory sections and overall structure, while other WG members fill in detailed discussion and conclusions. Some names have already been added to sections of the document, and more will follow.
  • Review outstanding WG points of tension:
    • Drill-down workflow:
      • Is the QA task purely an afterburner on a repository, or are there pipeline execution settings that will be varied to make sure the data necessary for QA is available?
        • If the pipeline settings need to vary, are they related to lsstDebug?
        • We have examples of pipeline tasks that produce metrics (like Jointcal) and send them to SQuaSH.
        • An afterburner may not be adequate for prompt processing — but that may be outside the scope of this WG.
        • Some degree of modification of pipelines may be required in order to produce all the data we might need (e.g. DMTN-057).
        • But we cannot, in practice, keep all intermediate data products.
        • Conclusion: Adding new metrics may involve persisting more data, but we don't expect to adjust it by configuration for a pipeline run.
      • Have we got a coherent story about generating static plots?
        • Yes!
        • The ability to look at pipe_analysis type plots with interactivity is useful.
        • Static plots have the advantage of being future proof.
        • If we are serializing static plots to disk, they should be accessible through the G3 Butler by data ID.
        • Converged on the answer that large scale display with predefined plots is necessary, but we don't have a strong recommendation one way or the other as to whether static plots must be defined.
        • If static plots exist, they need to be retrievable from the Butler, but it is not a requirement that all plots must be retrievable from the Butler.
        • Require that all (or most) plots that are currently being generated by pipe_analysis should be generated with minimal overhead — ie, not requiring many clicks of interactive “drill down”, not requiring an internet connection, able to run on a laptop.
    • We've not seriously discussed a SQuaSH-like dashboard (which is distinct from the drill-down dashboard).
      • I think that's because we're basically agreed that what we have now is good, and we can just write that... but does anybody disagree?
        • Apparently not. (smile)
        • But we do not that are some improvements we could make to the SQuaSH system in terms of visualization.
      • We do need more automatic regression alerts, though.
      • Should we take a position on DMTN-057, lsst.verify, or the way that metrics are defined?
        • The WG is unlikely to define pipelines interfaces.
        • However, we should take another pass through DMTN-057 and see if the WG conclusions are relevant to it.
        • And a WG recommendation has to be that somebody make a call on DMTN-057 soon.
      • We talked about whether SQuaSH needs to track metrics at a more fine-grained level than per-pipeline-run:
        • Per-CCD
          • But we agreed that we don't need to track this.
          • But we need some data-input-quality filter: this is not SQuaSH.
        • Per-(tract,patch) (sky region)
          • We like the idea of having SQuaSH track things at the per-data-ID level.
      • Time series:
        • We note that regression has to be tracked on a static dataset as the code evolves, not as the dataset evolves.
        • That might limit the use of a SQuaSH like system in commissioning (where data and code will change) or operations (where data will change, and code more slowly), but we won't worry about that in the scope of this WG.
    • What do we do about image viewing?
      • I see a couple of options here — 
        1. Somebody actually writes down some requirements, but I think they will be very hard to converge on or do make them comprehensive.
        2. We accept the reality that the future is Firefly, but identify specific ways in which Firefly could be improved to make it more palatable to developers. (I note that the feedback from the LSST@Europe session seems pretty positive.)
      • In general folks at LSST@Europe were pretty happy with Firefly.
        • There are some little things that people care about, but the Firefly team were quick to fix them.
        • Trey is optimistic that we can address issues if there's a tighter testing loop.
      • The use case that Firefly doesn't do well is acting as a desktop application.
        • We wouldn't want to mandate to always use Firefly because of this.
      • We don't think it practical to develop a comprehensive set of requirements for display
        • Partly because we don't think we can be usefully comprehensive on a finite timescale.
        • Partly because we think this would have to be agile rather than top-down.
      • The SUIT team is developing suggestions for improvements to afwDisplay to make it more appropriate to Firefly.
        • And which no-ops on other backends.
      • The WG could make a recommendation that afwDisplay should be update to make it possible to provide optional components.
      • Note to John Swinbank — we should add components for full-sky visualization, and large-scale-image visualization to DMTN-085, and Simon Krughoff will write some requirements in them.
    • Provenance!
      • Meeting 09:00 on Monday!
  • Timeline for this WG.
    • The group is due to finish by the end of June.
    • That means we have only two more formal meetings.
    • Suggested schedule:
      • By Friday 2018-06-15, John Swinbank will circulate a more complete version of DMTN-085 with names in all the sections;
      • Rough but basically complete draft of DMTN-085 by 2018-06-20 (next week);
        • Please write as much as you can in sections with your name in them. In some cases, that might be a detailed set of requirements; in others, it might just be some notes, or a suggestion that it'll take a new WG to properly study and come up with a plan for this area.
        • Please make additions using a branch-and-PR workflow.
      • Revised draft by 2018-06-27 (two weeks; final WG meeting).
    • That timetable is ambitious: we should expect to be spending most of our time in the next two weeks on WG activities.