Replicating the SSTAR Standard Report in MAF (Requirements)

The Science Metrics that Steve Ridgway developed are collected here.

The SSTAR Standard Report will be replicated in the MAF and the following enhancements are requested:

• Table 1:  add the "startup comment" from Config table either to the title or caption.  There is a JIRA item to add this to the Config table.
• Table 2:  add the NAME of the filter file - it is specified in the Config table and can be different than Filter.conf.
• Table 3:  
  • add the actual NAME of the LSST.conf file as it can be different than LSST.conf (can simply put this string in the table subheader).  There is a JIRA item to add this information to the Config table.
  • add the parameter idleDelay from LSST.conf to the table in this subsection.  There is a JIRA item to add this information to the Config table.
  • remove recalcSkyCount from the section for Scheduler.conf (parameter is not used and has been removed from future installations of conf files).
  • add MinDistance2Moon to the section for Scheduler.conf
• Table 4: Benchmarks - should support ANY user specified benchmarks that are requested for comparisons (not just stretch and design number of visits, single visit and coadded depths)
• Table 5 (and others of this form): It makes more intuitive sense to switch the position of the +3sigma and -3sigma columns to show the number of outliers. Ultimately, it will be instructive to be able to look more closely at these fields or datapoints to be sure that the scheduler selection is staying within specified bounds, so we will want a way to display these by altering the axis limits or otherwise print or identify these fields.
• Metrics that are for the WFD observing mode may make use of MORE THAN ONE proposal.  SSTAR identifies these by the proposal's name, but this is not as robust as needed.  We need a way to tag ALL proposals set up to fulfill the WFD science goal.
• Metrics showing sky brightness and single visit (5sigma) depth use the 5sigma_modified and skybrightness_modified values in the output table which are the post-calculated values.  The simulator selects fields based on vSkyBright so it would be worth seeing those distributions to be sure algorithms are behaving as expected.  Values of  filtSkyBright & 5sigma are the vSkyBright adjusted to the given filter and calculated single visit depth; values of perry_skybrightness & 5sigma_ps are vSkyBright transformed into sky and depth using a form of the ETC model; the plotted values should use 5sigma_modified & skybrightness modified which are the perry_skybrightness & 5sigma_ps values except for twilight observations for which we insert filtSkyBright & 5sigma.
• All Histograms showing curves for all filters and/or by filter should be presented in a manner that allows for examination of the outliers.  Side by side presentation of the table of numbers helps identify the number of datapoints outside the 3 sigma bounds, but what we really want to know is how many of the values are outside the specified bounds (usually set in the configuration files) and list the identifiers for these datapoints (fieldID and expMJD for instance).
• All sky maps show a color bar as the scale 
  • would be better presented as a single color in shades from light to dark instead of multi-color.
  • would be better for cross-comparison with other simulations if the scale range could be set to the "requested" value as set in the configuration files, or to the "benchmark" value (stretch, design, user)
  • would be better presented as a two color shading (like blue/orange) where the transition in color is the value of 1 in the case of a ratio or 0 in the case of a residual (difference) - this adds more content to the plot as the reader can instantly see areas above (e.g. blue) and below (e.g. orange) a target value.  A two color plot could also be implemented for values presented in percent with the transition occuring at >=100%.
  • Hammer Aitoff projections are most useful as the distortion over the sky is the least of all the projections and percent area is more easily judged by eye.
  • We typically display E to the right and W to the left with 0 degrees in the center but perhaps we need to discuss this as a group.
• Joint completeness (described below) can be presented both as the number of fields in a percentage bin (say 10% as in the SSTAR Standard Report) and as a cumulative value of number of fields having a minimum completeness in each filter >= bin_lower_bound.  The choice of binsize (currently 10%) is somewhat arbitrary and it would be useful to be able to set this to 25%, 20% 10% 5% 2% bins (for instance).  Also, in the same table, or new table, we like to know how many fields got exactly 100% of what was requested, then the numbers in bins below and the number in bins above.  This lets us judge the effectiveness of various setups in the simulation.
• Section 5.1 Filter map plotted for individual years would be more useful if we could "flip" through the one year images (like in a powerpoint or a movie) to see the detail in every year.
• Section 6.1 Slew activity numbers are currently presented as verbatim text from an output file, but would be more readable if they were presented in a table along with the key values at the top of this section.
• Section 6.2 Inter-visit time numbers are plotted logarithmically but I find the current plot style hard to read, so I would recommend making the plot similar to other histograms in the report.  Also a missing piece of information here is the number of cable wraps which will be an indicator of overall efficiency of a survey.
• All figures and tables should include some caption or documentation that repeats back the assumptions and settings for that which is being displayed.
• All figures and tables need to use fonts that are readable and be of publishable quality, as many times we have discovered plots finding their way in to project presentations.  The pgplot font used for axes and labels is unsatisfactory and needs to be improved.
• More detailed documentation will be needed to describe to our potential users (and to remind power-users) what each of the fields in the distributed tables are (with units) and how the various post-processing fields are calculated.  For example, we designed the SSTAR Standard Report with this in mind and included some short explanatory material - there have been pros and cons about putting this in the Standard Report with some argument for placing this material in a "User's Handbook".

Here are the recommendations from the SSTAR Validation document (Dec 2013):

1. Add the capability to generate a configuration table (Appendix B) based on the type of observing mode.  Each observing mode type has its own set of parameters and currently, only one common set of parameters is assembled into the configuration specification tables for each observing mode.  This leaves out some parameters entirely for some observing mode types, and results in blank fields or "?" for other parameters that do not exist for some observing mode types.  This would enable complete reporting and eliminate confusion. For SSTAR v4.0 there are two types of observing classes but there is some degeneracy in parameter names when it comes to specifying sequences, so this will need to be accounted for.
2. Allow for backwards compatibility of the setting for the telescope seeing in the Engineering Specifications section for all simulation versions.  In OpSim v2.3, the seeing was handled by one parameter called SystemSeeingFactor, and concurrent versions of SSTAR reported this parameter and its value.  In OpSim v2.3.2 this parameter was changed to telSeeing and this was reflected in the SSTAR without backwards compatibility for simulations that used SystemSeeingFactor.  Likewise, in v2.5, the combined seeing was reflected by using the combination of three parameters, telSeeing, opticalDesSeeing and cameraSeeing.  The current version of SSTAR v3.9.2 reports these parameters and their values, however, Standard Reports from v3.9.2 run on OpSim v2.4 and earlier codebases will not report the SystemSeeingFactor or telSeeing properly.  This is a backwards compatibility request and I did not hear if there was a decision on whether the MAF will support analysis of OpSim v2.x simulations, or how it will adapt to a changing OpSim v3+ schema.
3. Alter the way fields-based statistics are computed by starting with the fields requested, and not just fields that were visited. All statistics based on the number of fields observed are computed using only the observations that are present.   This is an historical issue as the SSTAR originally extracted information from the *output*.sql table, which includes only the fields from the ObsHistory table, and information from the Field table was not accessible.  For a ten year run, it is probable that all fields requested will get at least one visit and these computations will be accurate.  However, for shorter runs, it would be important to know about the fields that asked for visits but that received none.  This is not possible to report give the current algorithms.  The fix involves cross-matching the requested userRegions from the configuration files with the Fields table, much like the OpSim does to evaluate and suggest targets, then accumulate statistics based on that list. (I think the binner strategy addresses this issue)
4. AstronomicalSky.conf parameters should be written to the database (in the Config table as in OpSim v2.6.1) to ensure a complete record of all settable parameters are preserved for each simulation.This item has been added as a JIRA issue but these fields still appear in the report.
5. DDcosmology.conf is a sequence of sequences and the sky map of the number of visits per field by filter (Figure 29) is not plotting properly all sequences. This is a side effect of the way sequences are specified in the observing mode configuration file that is not handled properly in SSTAR, so this may or may not be an issue in MAF.

Below is the text from the OpSim Technical Review's supporting document (Analysis Tools: Plan for the Metrics Analysis Framework (Doc 15319 and on the Review Hub).

4.1.1 Phase 1 Implementation

The following list describes metrics that are currently planned for rapid implementation. This is
intended to include all functions that have proven consistently useful from the legacy post-processing
tools. Metrics will be updated and supplemented based on user-experience and other input from the
LSST and science communities.

 Number of visits at a particular RA/Dec, per filter, evaluated over a spatial grid across the sky.
This will be calculated for all visits and for visits split by proposal ID. The results will be
presented as a sky map, a histogram as a function of area, and simple summary numbers of the
median, mean + rms, and +/- 3σ over the entire region visited.

 ‘Joint completeness’ - calculation of amount of area with more than Nfraction of Nvisits in each
filter, where Nvisits is the number of visits set by the ‘baseline’ value for each filter, and
Nfraction ranges from 0 to 100% in steps of 10%. The visits to be considered for this metric can
be restricted to only visits tagged as part of the WFD proposal, or visits evaluated over a spatial
grid matching the WFD footprint and meeting requirements in seeing and single visit m5 values.

 Median, minimum and maximum single visit m5 depth at a particular RA/Dec, per filter,
evaluated over a spatial grid across the sky. This will be calculated for all visits and for visits split
by proposal ID. The results will be presented as a sky map, a histogram as a function of area, and
simple summary numbers of the median, mean + rms, and +/3σ over the entire region visited.

 Single visit m5 depth per filter, evaluated for all visits (regardless of RA/Dec position). This will
be calculated for all visits, as well as only visits tagged for the WFD proposal or visits within the
WFD footprint and meeting requirements in seeing and single visit m5 depth. The results will be
presented as a histogram as a function of number of visits and simple summary numbers of the
median, mean + rms over all visits.

 Coadded m5 depth at a particular RA/Dec, per filter, evaluated over a spatial grid across the sky.
This will be calculated for all visits and for visits restricted to those meeting requirements in Analysis Tools: Plan for the MAF Doc-15319 Latest Revision Date 01/27/2013
14
seeing. The results will be presented as a sky map, a histogram as function of area, and simple
summary numbers of median, mean + rms, and +/- 3σ over the sky.

 ‘Filter map’ - a plot of visit day vs visit hour (relative to local midnight), color-coded by visit filter.
This plot will be produced for each year of the survey (separately) as well as for the entire
survey.

 Median, minimum and maximum sky brightness at a particular RA/Dec, per filter, evaluated over
a spatial grid across the sky. This will be calculated for all visits. The results will be presented as a
sky map, a histogram as a function of area, and simple summary numbers of the median, mean
+ rms, and +/3σ over the entire region visited.

 Sky brightness per filter, evaluated for all visits (regardless of RA/Dec position). This will be
calculated for all visits and as well as only for visits tagged with the WFD proposal ID. The results
will be presented as a histogram as a function of number of visits and simple summary numbers
of the median, mean + rms over all visits.

 Median, minimum and maximum seeing at a particular RA/Dec, per filter, evaluated over a
spatial grid across the sky. This will be calculated for all visits. The results will be presented as a
sky map, a histogram as a function of area, and simple summary numbers of the median, mean
+ rms, and +/3σ over the entire region visited.

 Seeing per filter, evaluated for all visits (regardless of RA/Dec position). This will be calculated
for all visits and as well as only for visits tagged with the WFD proposal ID. The results will be
presented as a histogram as a function of number of visits and simple summary numbers of the
median, mean + rms over all visits.

 Median, minimum and maximum airmass at a particular RA/Dec, per filter, evaluated over a
spatial grid across the sky. This will be calculated for all visits. The results will be presented as a
sky map, a histogram as a function of area, and simple summary numbers of the median, mean
+ rms, and +/3σ over the entire region visited.

 Airmass per filter, evaluated for all visits (regardless of RA/Dec position). This will be calculated
for all visits and as well as only for visits tagged with the WFD proposal ID. The results will be
presented as a histogram as a function of number of visits and simple summary numbers of the
median, mean + rms over all visits.

 Median, minimum and maximum normalized airmass (airmass divided by the minimum possible
airmass for that RA/Dec) at a particular RA/Dec, per filter, evaluated over a spatial grid across
the sky. This will be calculated for all visits. The results will be presented as a sky map, a
histogram as a function of area, and simple summary numbers of the median, mean + rms, and
+/3σ over the entire region visited.

 Normalized airmass per filter, evaluated for all visits (regardless of RA/Dec position). This will be
calculated for all visits and as well as only for visits tagged with the WFD proposal ID. The results
will be presented as a histogram as a function of number of visits and simple summary numbers
of the median, mean + rms over all visits.