1. Introduction

In the original implementation of Qserv, nearly any catalog ingest requirements were mostly driven by the needs of the development and integration tests. These requirements were successfully addressed by a simple tool qserv-data-loader.py backed by the Python-based service wmgr run at each worker node. This mechanism is still available since it works fairly well for scenarios it was developed for. Unfortunately, it quickly falls short in many areas where ingesting large quantities of data is needed, or where the high performance or reliability of the ingests is mandatory. It's also clear that the simple mechanism won't be able to address the primary target - ingesting the LSST's Data Release Products. The number and variety of these cases started growing over the last year as the Project (LSST) was approaching its critical stages, and more complex use case scenarios were showing up for Qserv. Eventually, a need in having a more sophisticated and versatile Ingest System emerged.

The current article documents the new system while retaining the main focus on the practical aspects of using the system. More documents on the requirements and the low-level technical details of its implementation (unless it's needed for the purposes of the document's goals) can be found elsewhere.

And the final comment is on the information flow in the document. It's recommended to read the document sequentially. Most ideas presented in the document are introduced in section An example of a simple workflow. The section is followed by a few more sections covering advanced topics. And The API Reference section at the very end of the document should be used to find complete descriptions of the REST services and tools mentioned in the document.