• Created by Unknown User (xiuqin), last modified on Sep 25, 2017

Summary of team discussion on September 6, 2017

  • Persist between sessions will be achieved by saving the data into a file, SQLite file format is one candidate.
  • Firefly DataGroup object will remain but only be a value object. The operations on the data will be achieved through DB operations and/or stored procedures.
  • Dealing with expressions:
    • current expressions in plotting: evaluate them first, a parser and translator needs to be written to convert them to SQL form
    • filter in table: Continue to support the simple column filter. complex ones will be in SQL form, with more attention on UI design 
  • One metaData table for each data table, with information:
    • column name, type, format, ...  Use IRSA DD table as reference
    • special database if not H2 (see 'Keep options open' bullet)
  • Keep options open:
    • FF server directly access other DB, i.e. LSST user database in Science Platform. 
    • FF server uses DAX API for LARGE table data operations, instead getting the data into FF H2 DB.
    • FF server access multiple database systems.  Meta data for those special data tables will keep those information.
  • Good practice 
    • cleanup the extra tables afterwards
  • Future benefits:
    • possible join of two tables
  • Performance check:
    • after the first round of implementation, performance comparison with the old system needs to be be done  for up to ~ 1 million row tables.
    • ?? billion row table is in a separate category
  • Todo:
    • DM-11814 - Getting issue details... STATUS has been created for the first round of implementation
      • one search processor update with new architecture, include ingesting returned results into H2
      • data type mapping from current IPAC table to H2
      • unique table ID generation, unique metaData table name/ID generation
    • second round:
      • other developers follow the guidance to work on more search processors
  • No labels