The Header Service component of DM will have an initial design review on 2017-10-10 at 11 AM Project Time (PDT/MST), 1 PM Central Time.

Attendees:

Presenter: Felipe Menanteau

DM Systems Engineering: Kian-Tat Lim, Donald Petravick, Tim Jenness, Gregory Dubois-Felsmann

Outside experts: Dave Mills, Robert Gruendl, Russell Owen

BlueJeans:

Connecting directly from a room system?

1) Dial: 199.48.152.152 or bjn.vc
2) Enter Meeting ID: 281757062 -or- use the pairing code


Just want to dial in? (all numbers)
1) Direct-dial with my iPhone or
+1 408 740 7256+1 408 740 7256
+1 888 240 2560+1 888 240 2560 (US Toll Free)
+1 408 317 9253+1 408 317 9253 (Alternate Number)

2) Enter Meeting ID: 281757062

Objectives:

As described in LDM-294 section 9.3, the design review will check that the design:

  • meets the requirements and satisfies the use cases, and an implementation can be verified as doing so
  • conforms to the LSST DM architecture and has well-defined interfaces
  • is expected to be efficient in terms of labor cost, non-labor cost, and schedule
  • is expected to be reliable, maintainable, supportable, usable, and secure
  • conforms to good engineering practices

The primary document to be reviewed is DMTN-058.  This document includes motivation, requirements (with a proposed test plan), and design information, in addition to concerns and questions that still need to be answered.

Notes:


Requirements and use cases are still a bit unclear
  • Need to agree on requirements
Dave Mills and Don Petravick had original vision, handed over to Felipe
GPDF: Who are clients? Defines what the header content should be
Why is "catch-up" mode relevant?
  • It's a destination, not a source
What does Archiving mean? What content does it imply?
  • To be discussed later, depends on project
Header Service runs "inside" EFD cluster
  • May be on a separate machine but still part of cluster
Does design require that it be physically co-located with database servers?
  • No
No direct writing to EFD
Planning to write one file per CCD
Could add another event to indicate full focal plane is done
Controller = OCS
Which EFD for Test Stand?
  • Only for NCSA Complete L1 Test Stands, can also work at Tucson for integration
Header Service only needs OCS Middleware, but not EFD itself
Note that EFD is not a CSC
Configuration files
  • Code-based released configuration files
  • Configuration key/value pairs published on DDS
  • Also could be extended to capture "knobbing"
Need to have all other CSCs started after Header Service?
  • No, can have cache of published configuration in DDS
Forwarders get path to Header LFO from published event

Concerns

Expecting to gather telemetry at notification of pointing, beginning of integration, start of readout, end of readout.  Other times could be added.  Potential 300 ms delay from Camera would need to be accommodated.  Note that telescope position moves beginning at start of readout.
What is the purpose?
  • GPDF: Meant to serve needs of summit systems and minimal amount for archiving, not long-term consumption
  • Agreed, but should be configurable
Don't know the required key/value pairs
Need to establish a procedure to define content
  • Procedure should go through Systems Engineering
  • Definition of a metadata item should be in configuration, not code
  • This configuration is different from hard-coded items
Proposed an initial list — but is it minimal?
  • DM has a set of metadata defined that AP can use: VisitInfo structure, but also need orientation of sky and spider
  • Russell and Felipe should walk through VisitInfo to determine sources in real system
Assume no variants to metadata for different observing programs
But there might be variants for calibration data
If per-CCDs are 98%, should there be a master + deltas?
  • Related to atomic unit of archiving?
  • Don: Captures anything that should be captured
  • Approximate WCS would not come from Camera; may not want it
The Header Service is an optimization; all of its functions can be reconstructed later (but perhaps not in time for Alert Production)
Cannot write directly into LFA filesystem, need to pass the file to EFD
  • Need to discuss LFA interface later offline (depends on client)


Should Header Service compute some metadata?
  • Gives uniformity
  • Russell: Concern is that computation is incorrect or inadequate; harder to change in Header Service than DM Stack — e.g. many equations for airmass
  • GPDF: Header Service should not be accountable for long-term correctness of anything
  • RobertG: Major issue is 1 or 189 or 189*16 pieces
  • GPDF: Courtesy metadata is useful for people looking at raw archived files with non-LSST tooling; files should be minimally useful with common tools (not an operational requirement but a debuggability consideration); WCS is most important
  • Russell: Can ignore some headers and rewrite in DM code
One potential client is Camera Diagnostic Cluster
Today, Camera systems write one file per CCD, with an HDU per amplifier
Baseline for DM is the same
Need to convert lsstSim to match
When science users ask for a raw image, we provide additional metadata that could have been computed and created later; they can also ask for actual raw image at the time of capture
Even reading into DS9 for debugging may not need WCS; WCS could be computed and separately attached by another utility
Wavefront sensor + guider raft might have separate header specification? Depends on what wavefront sensing system needs, but prefer it to be the same

Requirements:

Tim: These requirements need to go into a SysML model
Tim: Does it need to be FITS?
  • Felipe prefers whatever JimP wants
  • Internally they are dictionaries
  • Russell: Why not a database?
  • File is simpler; an alternative could be the whole string as a message
  • GPDF: JimP's system is not the only source of materialization of files; wavefront sensing does not get files from JimP; but they may also use FITS
Have a separate requirement for each consumer to simplify testing

Failure modes

  • Failures upstream that prevent getting data
  • System crashes (but rest of EFD is running), then stop taking images
  • RHL/Russell: must keep taking images
  • GPDF: Could drop Alert Production for a few visits, but could have problems if Active Optics depends on these and if the downtime is long
  • Can do engineering activities even if EFD is down
  • Other critical systems cannot operate in degraded mode; Header Service could perhaps be part of a degraded mode
  • Whether we should have a header regenerator in the case of failure is unclear
Potential clients:
  • AP xtalk — Russell (walk through VisitInfo)
  • Archiving raw — Jim Parsons
  • Wavefront sensing — Sandrine/Te-Wei
  • Summit visualization — ?
  • Camera Diagnostic — Tony
Requirements should go into a spreadsheet with name, requirement, rationale/discussion
Other questions from Russell:
  • Missing data: missing data is missing period, even from EFD, not just missing from Header Service
  • Why reading directly from telemetry vs. the EFD: the EFD gets information from DDS; faster and more immediate to get from DDS; EFD is just another listener
  • Why doesn't Header Service ask for items it needs? OCS commands everything in our design

Outcome:

  • Design is reasonable, with requested configurability, given uncertainties in requirements, but some aspects may need to change as requirements evolve

Action Items:

  • Felipe Menanteau to contact potential clients above to discuss content and use cases
  • Felipe Menanteau to transform requirements into spreadsheet for loading into SysML model; requirements to be reviewed as part of incorporation into an LSE requirements document.
  • No labels