Page History

...

•  persistent support for super-transactions
  • table schema
  • DatabaseServices  
  • REST  services for managing super-transactions
•  database attribute to differentiate PUBLISHED databases from the ones which are being ingested
  • table schema
  • Configuration
  • REST  services for managing the database state (Configuration)
  • extended selectors in DatabaseServices  to differentiate operations with databases of different states. The default parameters of the selectors would always include database the PUBLISHED databases
•  test the above aded functions
  • first, merge DM-19226 into master, then rebase DM-19156 against master. This is neeed because of the schema change, whic
•   REST  services for Ingest
  •  Adding UNPUBLISHED databases
  •  Adding tables
    • in which format to store table schema for the ingested catalogs? CSS, MySQL INFORMATION_SCHEMA, ?
      •  added Configuration table to the Replication system to store column definitions
    •  creating prototype tables at workers
    •  add the director  table attribute to the Configuration as requested by a client. Also add the name of the objectId  column if the director  table attribute is provided when adding a table. Also add these attributes to class Configuration::DatabaseInfo.
      •  what about the overlap  radius? Should it be also stored in the Configuration?
    •  what about chunkId  and subChunkId  columns of the partitioned tables? Should the input schema be tested for a presence of those? Or should they be added automatically as the very last columns?  look at how it's done now.
  •  chunk allocator
    •  still needs to be fully tested. Probably after putting in place a simple loader server
  •  (super-)transaction management
    •  should the secondary index be also extended at this stage by harvesting triplets  from Object  (Qserv director  table)
  •  publishing databases
    •  build the secondary index.
      •  Should this be a separate REST call and a job? Note that it may be a lengthy operation.

      •  Should this be done when committing super-transactions? This may be much quicker compared to doing all at once.

      •  from the implementation prospective this would require adding a special kind of Controller and Worker requests to harvest triplets  from the Object  tables of the workers and loading them into the secondary index  table in Qserv czar's database.

    •  create database & tables entries in the Qserv master database

      •  should this be rather done when creating UNPUBLISHED  databases an adding tables?

    •  grant SELECT authorizations for the new databases to Qserv MySQL account(s) at all workers and the master(s)

    •  register the databases, tables and partitioning parameters at CSS

    •  enable databases at Qserv workers by adding an entry to table 'qservw_worker.Dbs'

      •  the implementation of this operation is ready. It just needs to be wired into the REST service

    •  remove MySQL partitions
      •  the implementation of this operation is ready. It just needs to be wired into the REST service

    •  ask Replication workers to reload their Configurations so that they recognized the new database as the published one. This step should be probably done after publishing the database.

•   loader server at workers
  •  extend Configuration to support loader services
    • extended attributes in WorkerInfo  
    • extended Configuration  schema
    • extended Configuration  API
  • Initially support TSV only
  •  support  both CSV and TSV files
  •  manage manage MySQL partitions
    •  the prototype tables created by the REST services have the default partition p0  as required by MySQL DDL. The loading service would have to ensure a new partition (based on numbers/values of the provided super-transactions) is added to the chunk-specific tables 
  • implement  implement a command-line tool for sending CSV/TSV files to the loader services
  •  what needs to be done to  Added a REST service for generating the empty chunks list? and placing it a usual location of the Qserv master  nodes
  •  add support for for the regular  (non-chunked) tables
    •  this can be done via an optional switch to the loading application. The protocol definition may also need to be extended.
•  add support for replicating semi-ingested tables when committing super-transactions 
•  add support for translating partitioned MySQL tables into the monolithic ones when publishing databases
•  Extend the Web Dashboard to display various aspects of the Ingest System (status of the UNPUBLISHED  databases, super-transactions, the number of chunks loaded, the amount of data loaded, etc.)

...