...
| Note |
|---|
Please visit sections Databases and database families and REST service for pulling configuration records on databases and database families for the explanation of the configuration REST service before reading the rest of this chapter. |
...
| parameter | description | How it can be used by the ingest workflows | ||
|---|---|---|---|---|
num_loader_processing_threads | This is the number of request processing threads in each worker's ingest service. | It affects ingests made via the proprietary binary protocol using the application qserv-replica-file. To achieve the maximum performance of the operations the workflows should aim at having each participated worker loaded with the number of ongoing requests equal to the number of threads reported. | ||
http_max_listen_conn | The maximum length of the queue of pending connections sent to the Ingest worker's HTTP-based service. | It affects ingest requests sent to the worker's REST services via the HTTP protocol as explained in chapter Ingesting files directly from workers. Both synchronous and asynchronous protocols are affected by the value of the parameter. It's recommended NOT to exceed the number of pending connections to the worker. Exceeding this parameter by sending more (than this number) requests to the worker may result in seeing the following error by a client:
| ||
num_http_loader_processing_threads | The number of request processing threads in each Replication worker's HTTP-based ingest service (SYNC). | It affects ingest requests sent to the worker's REST services via the HTTP protocol as explained in chapter Ingesting files directly from workers. The limit applies only to the synchronous protocol. To achieve the maximum performance of the operations the workflows should aim at having each participated worker loaded with the number of ongoing requests that is equal to or higher than the number of threads reported. | ||
num_async_loader_processing_threads | The number of request processing threads in each Replication worker's ASYNC ingest service. | It affects ingest requests sent to the worker's REST services via the HTTP protocol as explained in chapter Ingesting files directly from workers. The limit applies only to the asynchronous protocol. The ingest workflow is supposed to plan the loading effort to ensure as many loading threads are busy as possible. | ||
async_loader_auto_resume | The flag controlling the behavior of the Replication worker's ASYNC ingest service after a restart of the service. If the value of the parameter is not 0 the service will resume processing incomplete (queued or ongoing) requests. Note that requests that were in the final state of loading data into MySQL before the restart won't be resumed. These will be marked as failed. Setting a value of the parameter to 0 will result in failing all incomplete contribution requests existed before restart. Note that requests that failed in the last (loading) stage require aborting the corresponding super-transaction. | It affects ingest requests sent to the worker's REST services via the HTTP protocol as explained in chapter Ingesting files directly from workers. The flag applies only to the asynchronous protocol. In theory, there is nothing special to be done by the ingest workflows based on a value of the parameter. It's just recommended to enable (if it's not done) the option before ingesting large catalogs as it will allow reducing a need in aborting transactions after the failuresresubmitting most requests interrupted by the restarts.
|
On a typical Linux system, the default value of the parameter is set to 256. Though, the worker's host operating system and the ingest service run at the host may be configured with a higher limit.