View Source

Here's a little report on what we've done to replicate SSTAR with MAF

Elements of SSTAR replicated in MAF:

SSTAR	MAF Black text tells where the corresponding plot can be found Blue text signifies an expanded feature not in SSTAR Red points out things not replicated
Table 3: configuration parameters and settings	Opsim Configuration: Configuration Summary and Configuraiton Details
Table 4: WFD Benchmarks	MAF lists any used benchmark values in axis lables.
Table 5: Median, mean, and RMS of number of visits per field and per proposal	These are replicated in the "Nvisits" group. Values are computed for all proposals, per proposal, and for all WFD proposals.
Figure 1: 6-panel, Nvisits for WFD	multi color: 2: Nvisits : WFD
Figure 2: Nvisits scaled by the spec value	multi color: 2: Nvisits : WFD, ratio
Figure 3: Number of visit histogram	multi color: 2: Nvisits : WFD. Looks like we don't reproduce the legend percentiles anywhere?
Figure 4: Number of visits not broken down by filter, WFD	All Results: Nvisits, regular hist OpsimFieldSlicer propID XXX histOnly
Figure 5: cumulative version of Fig 3	multi color: Nvisits cumulative (OpsimFieldSlicer)
Figure 6: cumulative version of Fig 4	All Results: Nvisits, cumulative OpsimFieldSlicer propID 448 histOnly
Table 6: Completeness for WFD observing mode	Group: 4: Completeness; Subgroup: WFD; Slicer: OpsimFieldSlicer. Table bins expanded to include P==0, P==1, and P>1. Expanded to include sky maps of completeness
Table 7: Cumulative version of table 6	not included
Figure 7: Visits for all modes	2: Nvisits : All Props
Fig 8: Visits for all modes scaled by spec value	2: Nvisits : All Props, ratio
Table 8: Completeness for all observing modes	Group: 4: Completeness; Subgroup: All Props; Slicer: OpsimFieldSlicer
Table 9: Cumulative version of Table 8	not included
Figure 9: Median single visit depth	mulit color: Single Visit Depth : All Props
Figure 10: single visit depths histogram	multi color: Single Visit Depth : WFD
Table 10: Single visit depth per proposal and all proposals	summary stats: Group: Single Visit Depth; Subgroup: All Props; Slicer: UniSlicer Group: Single Visit Depth; Subgroup: Per Prop; Slicer: UniSlicer Group: Single Visit Depth; Subgroup: WFD; Slicer: HealpixSlicer
Fig 11: Coadded Depth	multi color: 3: CoaddDepth : All Props (note these can be run with the HealpixSlicer to resolve field overlap)
Figure 12: Coadded Depth, WFD	multi color: 3: CoaddDepth : WFD
Fig 13: Histogram of Fig 11	multi color: 3: CoaddDepth : All Props
Fig 14: Histogram of Fig 12	multi color: 3: CoaddDepth : WFD
Table 11: Coadded depth mean, median, RMS, etc per filter	Summary Stats: Group: 3: CoaddDepth; Subgroup: All Props; Slicer: HealpixSlicer Group: 3: CoaddDepth; Subgroup: WFD; Slicer: HealpixSlicer
Fig 15: Filter map	multi color: Hourglass : NULL
Fig 16: Median Sky Brightness, all proposals	multi color: Sky Brightness : All Props
Fig 17: Histogram of WFD sky brightnesses	multi color: Sky Brightness : WFD
Table 12: Sky Brightness per proposal, per filter	summary stats: Group: Sky Brightness; Subgroup: All Props; Slicer: UniSlicer Group: Sky Brightness; Subgroup: Per Prop; Slicer: UniSlicer Group: Sky Brightness; Subgroup: WFD; Slicer: UniSlicer
Fig 18: Seeing maps	multi color: Seeing : All Props
Fig 19, Histogram of WFD seeing	multi color: Seeing : WFD
Table 13: Seeing per proposal, per filter	summary stats: Group: Seeing; Subgroup: All Props; Slicer: UniSlicer Group: Seeing; Subgroup: Per Prop; Slicer: UniSlicer Group: Seeing; Subgroup: WFD; Slicer: UniSlicer
Fig 20: Median Airmass, all observations	multi color: Airmass : All Props (extended to include maximum airmass and median normalized airmass)
Fig 21: Airmass, WFD only	multi color: Airmass : WFD (extended to include maximum airmass and median normalized airmass)
Fig 22: Airmass histogram for WFD	multi color: Airmass Histogram (OneDSlicer)
Table 14: Airmass table per filter per proposal	Summary Stats: Group: Airmass; Subgroup: All Props; Slicer: UniSlicer Group: Airmass; Subgroup: Per Prop; Slicer: UniSlicer\| Group: Airmass; Subgroup: WFD; Slicer: UniSlicer
Section 6.1 Slew Activity, basic stats (number of nights, ave exposure time, etc)	Summary stats: Group: Slew Summary; Subgroup: NULL; Slicer: UniSlicer (note, number of nights is now number of nights where observations were made, not length of survey)
Section 6.1: Stats for telAlt, telAz, rotTelPos	Summary stats: Group: Slew Angle Stats; (note, values now in radians. Needs verification, I suspect SSTAR was taking a mean of absolute values rather than a true mean?)
Seciton 6.1: Slew Activity	Summary stats: Group: Slew (note values now in radians. Not all values replicated. Values do not seem to match due to undocumented normalization factor)
Section 6.1: Slew maximum speeds	Group: Slew Speed (in radians. Not clear what "% of slews" should be.)
Figure 23: Distribution of slew times and distances	All Results: Group: Technical; Subgroup: Slew Slew Distance Histogram OneDSlicer Slew Time Histogram OneDSlicer
Section A: Number of visits per proposal	multi color: 2: Nvisits : DDcosmology1 2: Nvisits : GalacticPlane etc,
Section B: Configuration files for proposals	Opsim Configuration: OpSim Configuration Details

Reviewing the Requirements:

The SSTAR Standard Report will be replicated in the MAF and the following enhancements are requested:

Table 1: add the "startup comment" from Config table either to the title or caption. There is a JIRA item to add this to the Config table. The OpsimComment is on the select run page
Table 2: add the NAME of the filter file - it is specified in the Config table and can be different than Filter.conf. Listed in the Opsim Configuration tab
Table 3:
- add the actual NAME of the LSST.conf file as it can be different than LSST.conf (can simply put this string in the table subheader). There is a JIRA item to add this information to the Config table. (LJ - this is actually in the opsim config page, just further down on the Config Details panel).
- add the parameter idleDelay from LSST.conf to the table in this subsection. There is a JIRA item to add this information to the Config table.
- remove recalcSkyCount from the section for Scheduler.conf (parameter is not used and has been removed from future installations of conf files).
- add MinDistance2Moon to the section for Scheduler.conf – These are all now listed in the Opsim Configuration page
Table 4: Benchmarks - should support ANY user specified benchmarks that are requested for comparisons (not just stretch and design number of visits, single visit and coadded depths) Benchmark values are specified in the maf config file and can be set to any value.
Table 5 (and others of this form): It makes more intuitive sense to switch the position of the +3sigma and -3sigma columns to show the number of outliers. Ultimately, it will be instructive to be able to look more closely at these fields or datapoints to be sure that the scheduler selection is staying within specified bounds, so we will want a way to display these by altering the axis limits or otherwise print or identify these fields. OK, the order is different in MAF than sstar. Not sure why one is more intuitive...
Metrics that are for the WFD observing mode may make use of MORE THAN ONE proposal. SSTAR identifies these by the proposal's name, but this is not as robust as needed. We need a way to tag ALL proposals set up to fulfill the WFD science goal. MAF builds a query for all proposals that are tagged as WFD (LJ - we need to update this identification in MAF to the new standard, which is that the info is just kept in the config table rather than the proposal table)
Metrics showing sky brightness and single visit (5sigma) depth use the 5sigma_modified and skybrightness_modified values in the output table which are the post-calculated values. The simulator selects fields based on vSkyBright so it would be worth seeing those distributions to be sure algorithms are behaving as expected. Values of filtSkyBright & 5sigma are the vSkyBright adjusted to the given filter and calculated single visit depth; values of perry_skybrightness & 5sigma_ps are vSkyBright transformed into sky and depth using a form of the ETC model; the plotted values should use 5sigma_modified & skybrightness modified which are the perry_skybrightness & 5sigma_ps values except for twilight observations for which we insert filtSkyBright & 5sigma. It was decided to drop the perry and modified skybrightnesses.
All Histograms showing curves for all filters and/or by filter should be presented in a manner that allows for examination of the outliers. Side by side presentation of the table of numbers helps identify the number of datapoints outside the 3 sigma bounds, but what we really want to know is how many of the values are outside the specified bounds (usually set in the configuration files) and list the identifiers for these datapoints (fieldID and expMJD for instance). Histogram ranges can be explicitly specified. If no ranges are set, the historgrams will show the full range of values including any outliers.
All sky maps show a color bar as the scale
- would be better presented as a single color in shades from light to dark instead of multi-color. The colorbar can be changed to any matplotlib color table.
- would be better for cross-comparison with other simulations if the scale range could be set to the "requested" value as set in the configuration files, or to the "benchmark" value (stretch, design, user) The colorbar ranges can be set in the config file (and then the same config run on different runs) There is a current JIRA issue to make it easier to replot figures with new ranges.
- would be better presented as a two color shading (like blue/orange) where the transition in color is the value of 1 in the case of a ratio or 0 in the case of a residual (difference) - this adds more content to the plot as the reader can instantly see areas above (e.g. blue) and below (e.g. orange) a target value. A two color plot could also be implemented for values presented in percent with the transition occuring at >=100%. Any matplotlib color table can be specified. Should we make a ticket to change the default color table for all the sstar plots?
- Hammer Aitoff projections are most useful as the distortion over the sky is the least of all the projections and percent area is more easily judged by eye. we default to aitoff projection for teh OpsimFieldSlicer and Mollview for Healpixslicer. Hopefully healpy does a better job in the future and we can switch that to aitoff as well.
- We typically display E to the right and W to the left with 0 degrees in the center but perhaps we need to discuss this as a group. We are using the community standard plotting directions
Joint completeness (described below) can be presented both as the number of fields in a percentage bin (say 10% as in the SSTAR Standard Report) and as a cumulative value of number of fields having a minimum completeness in each filter >= bin_lower_bound. The choice of binsize (currently 10%) is somewhat arbitrary and it would be useful to be able to set this to 25%, 20% 10% 5% 2% bins (for instance). Also, in the same table, or new table, we like to know how many fields got exactly 100% of what was requested, then the numbers in bins below and the number in bins above. This lets us judge the effectiveness of various setups in the simulation. We now have bins for exactly 100% and exactly zero. The total number of bins is a kwarg.
Section 5.1 Filter map plotted for individual years would be more useful if we could "flip" through the one year images (like in a powerpoint or a movie) to see the detail in every year. Filter maps are made for each year.
Section 6.1 Slew activity numbers are currently presented as verbatim text from an output file, but would be more readable if they were presented in a table along with the key values at the top of this section. Slew stats are now in tables. We have not managed to match the values in sstar.
Section 6.2 Inter-visit time numbers are plotted logarithmically but I find the current plot style hard to read, so I would recommend making the plot similar to other histograms in the report. Also a missing piece of information here is the number of cable wraps which will be an indicator of overall efficiency of a survey. It really doesn't show much on a linear scale. How do we get the number of cable wraps?
All figures and tables should include some caption or documentation that repeats back the assumptions and settings for that which is being displayed. All the metrics can have captions set in the config file
All figures and tables need to use fonts that are readable and be of publishable quality, as many times we have discovered plots finding their way in to project presentations. The pgplot font used for axes and labels is unsatisfactory and needs to be improved. We output both png and high res pdf files.
More detailed documentation will be needed to describe to our potential users (and to remind power-users) what each of the fields in the distributed tables are (with units) and how the various post-processing fields are calculated. For example, we designed the SSTAR Standard Report with this in mind and included some short explanatory material - there have been pros and cons about putting this in the Standard Report with some argument for placing this material in a "User's Handbook". Yes, we still need more documentation on Opsim output, this is not a MAF issue though.