...
- We held an extended discussion about the various issues that are connected to this.
- ACTION Srini will create a confluence page with a proposal for how to move forward, capturing these issues, especially ones Simon brought up
- We will hold a follow-up discussion Oct 15 (we should make sure to notify other interested parties)
- Issues that were brought up that should be considered in the final concept / design for the Simulator
- The simulator design ideally should be database agnostic
- The simulator design should mirror the scheduler which may NOT be database agnostic
- Currently the Simulator runs using MySQL DB and gets better performance (~30 days with no lookahead) than the branch with SQLite (SQLite is 20% worse without lookahead than MySQL). This may become an issue for when lookahead is ready (takes longer to produce and analyze runs) and if we do a very large number of runs to optimize groups of parameters this will be a factor. There may be some more things that can be done to tune SQLite to improve performance.
- Currently we can still use the SSTAR on the MySQL db until we have a workable replacement. But MAF can be configured to read the MySQL DB directly - this would restrict analysis to taking place on ops1 and ops2 (not on ones laptop)
- SessionID's (single simulated surveys) contained in one file (.sql or .sqlite) are needed for distribution to the Sci Collabs, and for OpSim to work independently of ops1 and ops2 (freeing computing resources in some cases). Tableau does not work with sqlite files so preserving single survey .sql is desired since it does not run on Linux.
- There is an advantage in analysis/validation to be able to examine more than one run at a time - this is trivial in MySQL, but not obvious how to do this in SQLite.
- A learning curve is necessary in being able to manipulate the SQLite files (very different than MySQL, although the actual "select *..." statements are the same)
- The Simulator is currently not deterministic.
- Too many "translation layers" or places where the foundation DB is converted to other forms is breeding ground for data errors and versioning issues as well as duplication of effort. (right now we manually control the uniqueness of naming a simulated survey for example) - so creating individual files (sql and sqlite) as well as adding information to tables after the simulation has completed are potential problems and should be minimized. We need to verify that currently, the schema in all these forms is identical; and ensure this going forward. Maybe address with write protection to whichever DB the Simulator is using. (Adding Dithering patterns and sky are two examples)
- Simplification is a core value
- We want to facilitate moving away from using SSTAR to using MAF
- The time to run MAF (any combination of drivers), and the tools which create these extra files, can be time consuming so the final implementation should optimize this
...