DISCUSSED: configuring Qserv containers

Three operating environments:
- Kubernetes + operator
    - Some skepticism due to "bumpiness" at NCSA
    - But reasons for optimism at USDF including Rancher/Anthos
- Docker compose single-node developer environment for rapid iteration (but still multiple-worker)
    - Why not use Kind or K3s or microK8s? Still not convenient enough for single developer
- "Igor" mode at NCSA: docker + bash
    - One-off, lowest-priority since will go away this Oct.
Configuration types:
- "boilerplate" details that need to be in place for qserv to function
    - Specific xrootd, mariadb settings that user should not change
    - Tied to specific releases of Qserv or constituent services
- Per-deployment configuration
    - Number of workers, storage classes, performance tweaks
    - Secrets typically fall here but need to be protected
- Dynamic configuration
    - Performance, logging configuration
    - May want to keep existing process for debugging, rather than restarting
    - K8s ConfigMap changes can be pushed into existing pods; need to notice those; but has to be for all pods, not just one
Configuration mechanisms:
- Environment variables
- Config files
- Command line parameters
- Databases
    - Working to eliminate this
Objectives:
- Keep the boilerplate within the containers
- Expose common configurations to users
    - K8s: "qserv" resource YAML
    - docker-compose: "qserv" CLI command-line options or YAML config file
    - docker+bash: config files mounted into containers
        - Compile-time in-memory defaults and configuration files or command line
        - Boilerplate and user-specific is mixed
        - Existing system is based on low-level service-specific config files
        - Replication system maintains its configuration in in-memory object derived from multiple sources (database, command line)
- Leverage K8s best practices
- Have sufficient flexibility for developers and operators to experiment, adjust, and debug
- Be consistent, predictable, and comprehensible
    - Previously, certain parts of the configuration were handled at different times
Proposal:
Unified "entrypoint" script instead of invoking raw binaries
- Describe the binary to run
- Pass user configuration arguments for that binary on command line, in environment, or in YAML files
- Knows the configuration needs for each binary; generates low-level configurations as needed
    - Via jinja templates for config files and low-level command line
In K8s, pass YAML files via ConfigMap
- Content is in the ConfigMap; ConfigMap is pointed to by filename
Avoids needing to keep operator in sync with Qserv versions
For developers:
- Pass through everything after -- to binary command line
    - Click doesn't have to be informed about all options available
- Provide ability to define template variables that are not yet known by script
    - Fritz thought of this only this morning
    - Don't need to do env vars, as there are other mechanisms for those
- Provide ability to override internal templates
Database schemas and bootstrap initialization
- Separate issues, including schema upgrade
- Similar plumbing for schema and configuration (both ConfigMap)
- May be useful to use initContainer, but not available in non-K8s environment
Complex templates sometimes needed
- May need to pass lists and generate domain names
- Jinja can do this
Should have most of this implemented in a week or so
Operator changes coming to enable adding workers and xrootd redirectors on the fly
- But still may need to have changes to replication system and xrootd to take advantage
- New non-DB replication system will use worker UUID identity