We would like to create a streamline process for executing a simulation, running standard MAF analyses on it, and locating output in an easily accessible repository.
There are four options for moving forward:
- continue as now: run opsim on mysql,, post process to yield a sqlite file, run maf on sqlite
- run opsim on mysql but directly generate sqlite file, run maf on sqlite
- run opsim on sqlite solely
- convert opsim to be ‘database agnostic’
We discussed and agreed on option 4.
This means we will have a separate database file for every sessionID (recorded in a tracking DB) whether it is MySQL or SQLite. OpSim will use the MySQL option internally, and will use the conversion tool to produce an SQLite file for distribution when needed, but we would like to phase this out to keep things simple and less prone to bugs if schemas get out of sync. There will be both internally and externally facing repositories (externally facing will use SQLite).
These are the tasks and time estimates to implement this plan:
- make schema changes in DB (make sure SQLite and MySQL match)
- needs wfd tag - implemented in simulator
- conversion tool changes to use wfd tag
- MAF will need a layer to interact with single MySQL files (??)
- create master script to run entire pipeline (simulator, standard maf output, relocate files to repository)
- production and testing needed for translation layer
- Set up internal repository (plan and/or implement external repository)