There are a number of shared resources in the Data Access Center (DAC) that need to be managed with quotas. The expectation is that every user will receive a "birthright" quota and that additional resources can be allocated via a resource allocation committee. Teams of users may also be allocated resources from the shared pool. In addition, it may be possible for a user or team to purchase resources (by adding to the shared pool and gaining a correspondingly increased quota).
Possible resources to be managed include:
| Resource Type | Potential Unit | Notes |
|---|---|---|
| Compute cycles | Core-hours | It remains to be decided what time period a compute quota would apply to. We could have a "for the remainder of operations" quota that you could use all at once or spread out over many years; we could have an annual quota that is refreshed, say, at each DR; we could have a monthly quota. Notebook, next-to-database, and batch usage seem likely to be charged to this quota. Qserv query usage, cutout generation, "virtual dataset" regeneration, and Alert mini-broker filter execution might be charged to this as well. Most Portal operations would not be charged, but there might be some for which it would be appropriate. Memory and local-to-node scratch disk will likely be handled by allocating a certain amount per core (the "machine/instance type" model) rather than through separate quotas. |
Persistent storage
| GB | |
| External bandwidth | GB | Ingress/egress from the DAC as a whole |
| Internal bandwidth | GB | e.g. Parquet files retrieved for next-to-database processing |
CPU and bandwidth, as "on-demand" resources, may require some level of fair-share priority allocation mechanism. This is typically available off-the-shelf for batch usage, and standard networking mechanisms may work well enough.
There may be some resources that are "too cheap to meter", managed by performance degradation (for a given user), or managed by social pressure.
Some services may also have rate limiting on a per-user or all-user or even IP address basis to prevent abuse.
In some cases (e.g. file storage), the underlying system can manage quotas for us; we need a management system that interfaces with the infrastructure-provided one for control, display, and alerting. In others, we will need to build quota checking and resource consumption logging into services.
Components needed:
- Quota database. This maintains the quota for each user or team for each resource. This may be fronted by a service.
- Quota checking. Some options include:
- A management system disables user access to services that consume resources for which the user's quota has been exceeded
- Each service consults a quota service to determine if sufficient resources are available to execute a task
- Resource consumption logging. This includes determining what resources attributable to a particular user were consumed by a task and logging this information.
- Resource log collection and aggregation. This updates quota values; the latency of this process controls the stringency of quota enforcement.
- Administrator management interface. This allows administrators to adjust quotas based on resource allocation committee decisions.
- User interface. This allows users to view their resource consumption and quotas.
- Alerting system. This informs users proactively when they are approaching their quota limits.