15 November, 2017 -- Meeting notes and actions

Attendees: Russell Owen, Eric Bellm, Simon Krughoff, John Parejko, Merlin Fisher-Levine, Robert Lupton

Meeting goal: Get a list of actions to put the obs packages on a better footing in the next few months.

Agenda:

• What docs do we need?
• What do we need to do in the short term for ingestion?
• Do we need to split calibration from code/configuration?
• How much standardization should we shoot for in the next few months?

Docs

The working group decided that we should deliver a document on how to start from a template package and implement all the important parts of an obs package.  The test particle will be ZTF, but should not be specific to ZTF.

• Merlin Fisher-Levine will take a reasonable obs package as a starting point for obs_aux_tel.
• Merlin Fisher-Levine will deliver a set of coarse notes on what was needed to RHL. DM-12677 - Getting issue details... STATUS
  
• Eric Bellm will deliver some test data (possibly engineering data) from ZTF to RHL DM-12678 - Getting issue details... STATUS
  
• Robert Lupton will produce a cookbook for how to build an obs package starting from the template obs package chosen by Merlin and informed by the data format from Eric. DM-12679 - Getting issue details... STATUS
  
• Eric Bellm will work with RHL to use the cookbook to create an obs_ztf package. DM-12681 - Getting issue details... STATUS
  

Ingestion

Robert points out that ingestion of calibration frames is different from ingestion of other calibration like data: defects, linearity, etc.  This should all be done in a common way at some point.

In the mean time, raw ingest is fine.  Calibration ingest for most scenarios is fine, as well.  We will not touch ingestion code in the short term, but will produce documentation on how to use the current ingestion tools.

Calibs and configuration

The group agreed unanimously that we need to figure out how to store calibration information: camera geom, defects, etc., separate from configuration information: dataset definitions, pipeline configurations, etc.  The group also agreed unanimously that the Butler Gen3 world will be so much different from the current situation that any work to do this split now would be wasted.  This will be deferred until we have the initial Butler Gen3 implementation.

Standardization

The working group decided that we should standardize on camera descriptions going forward.  To do this we need to review Robert Lupton's YAML camera description design.  To do that I think we need at least a cursory documentation on how to use it.

• Robert Lupton produce a short design document describing the basic building blocks of the YAML camera description DM-12682 - Getting issue details... STATUS
• Simon Krughoff Schedule a design review of the system DM-12684 - Getting issue details... STATUS
  

Part of the above review should address how we could avoid going through pex_config on the way to the camera constructors.

Another decision was taken to allow the old camera descriptions to remain for now.  John Parejko registered a concern that that may allow old camera code to leak into the stack.  The rest of the group agreed that any special code for camera construction was completely within the obs package where it is used so should not leak out.

Currently there are already standard ways to represent defects, linearity, crosstalk, and BF.  Those should be used in new obs packages.  Some older packages use more ad hoc ways of dealing with these, but we will hold off on doing anything about those until we have the SuperTask and Butler Gen3 first versions.

Given that the obs_auxtel implementation goes ok, we will generate a skeleton obs package for people to use as a template.

• Someone will produce a template obs package starting from one of the newer, simpler obs packages DM-12685 - Getting issue details... STATUS