Please dont change the following page without talking to Srini Chandrasekharan. This page is being used as a specifications document

SPECIFICATIONS for Database Registration process

Workflow Considerations

We will need to consider how database registration (or creation of a product from the Simulator DB that is usable by MAF) will fit into the overall workflow. 

1.  Do we need to create a separate "pre-processing" codebase?  Should this be incorporated by MAF as part of "processing", or should it be incorporated by the Simulator as "post_processing" to the simulation?

2. If we keep the "summary_" table in the database will the database become too large at some point?  Do we need to start keeping simulated surveys in separate databases?

3. How does OpSim best transition from "output_" and SSTAR to "summary_" and MAF without inhibiting productivity (holding up current analysis).

   
   

Audiences / Use Cases

1. OpSim
   1. SSTAR performs "processing" that produces an extra table ("output_", to evolve to "summary_") that is written to a file as well as single-session files that are portable.  
   2. SSTAR acts on the  "output_" table as well as the database as a whole, keeping the "output_" table as an extra table in the database, to generate its output.
   3. If OpSim is to phase out SSTAR, then a new tool needs to be implemented to generate the "summary_" table (and other exported files as specified in Deliverable below) and MAF will have to completely replicate SSTAR.
2. MAF Team
   1. The starting input is an SQLlite file of the "output_" table (now "summary_"), assuming pre-processing is already completed.
3. Science Community user 
   1. To date the expectation is that the user will download a "summary_" file from theOpSim Data Access website, install MAF and do analysis. 
   2. Now that the Simulator is open source, if the user creates his/her own simulations, they will also need this pre-processing code before they can use MAF 

Specifications

• A new table will be created that is called "summary_machinename_sessionID"
• Columns will be copies of those from the ObsHistory table along with propConf from Proposal that corresponds with propID
• Specific columns:

• Files to be created
  • ASCII dat  format of the "summary_"  table (already implemented in SSTAR; needs changes above)
  • MySQL .sql format of all OpsimDB tables including "summary_" table for a sessionID (already implemented in SSTAR; needs changes above)
  • MySQL .sql  format of the "summary_" table (already implemented in SSTAR; needs changes above)
  • SQLlite format file containing all tables including "summary_" table

MEETING ON APRIL 10, 2014

Attendees: Peter, Simon, Kem, Cathy, Lynne, Srini, Steve

1. Deliverable of Opsim - What should it be?
   1. Both output table & the entire DB in sqlite format + output table in dat format
   2. csv discussion - Deferred
2. What happens to the additional columns ?
   1. dithra & dithdec etc will be added to preprocessing
   2. extra columns will goto preprocessing

Deliverable

Preprocessing layer which will take as input the OpSim DB and convert that to:

• Flat dat (output_ table) [already implemented]
• MySQL .sql (one for all tables including output tables for a sessionID, and one for the output table) - this is already being done phase out
• Add sqlite file containing (all tables including output_ table)
  
  

NOTES BEFORE THE MEETING

Meeting Objective

Discuss needs and wants associated with producing content (output) from the Simulator that is usable by both the team and the community, and that will be accessed by MAF.

Background

1. Generating "extra" columns
   1. 5sigma, perry_skybrightness,5sigma_ps, skybrightness_modified, 5sigma_modified, hexdithra, hexdithdec, vertex
   2. these quantities are not used or required by the Simulator, and philosophically no post-processing should add/change the OpsimDB database, so these are created by gen_output.py in SSTAR.
   3. the "output_machine_id" table is a copy of the ObsHistory table plus the fields in (a)
2. Sky Brightness models & fields
1. when a unified model is adopted by the project, it will be added to the Simulator and those fields will be recorded in the ObsHistory table.
2. the different ways of computing sky brightness have different advantages - skybrightness_modified and 5sigma_modified were developed in an effort represent the "best" value as a combination of the other two.  Here is the description from the SSTAR documentation:

Questions

1. Who are we creating the "output" for?  should OpSim/MAF be accessing the DB directly? what about community users?
2. What information is needed by OpSim (all tables), and the community (just ObsHistory?)  
3. Do we need one "materialized" table for simplicity or can all tables be distributed in one file?
4. What does SSTAR use?  (not output*)
5. While extensibility and inter-backend-operability is key for MAF what would be the ideal input format?
6. How do you include proposal information?  Are duplicate rows acceptable?