Docker CPU Limits and Throttling Behavior for Latency Sensitive Services

A hard CPU limit in Docker, set with the cpus option, caps how much CPU time a container can use in each scheduling period, and once that budget is spent the container is throttled until the next period. That throttling can add latency spikes even when average CPU usage looks low. For latency sensitive services, relative CPU shares or generous limits are often safer than tight quotas.

How Do Docker CPU Controls Work?

Docker uses Linux control groups to manage CPU. There are three main mechanisms, and they behave differently under load, which is why it matters to know which one you are using.

• CPU quota (cpus, cpu-quota): a hard cap. The container may use a fixed amount of CPU time per period, typically 100 milliseconds, across all of its threads.
• CPU shares (cpu-shares): a relative weight. It matters only when containers compete for CPU, and it does not limit a container on an idle host.
• Cpusets (cpuset-cpus): pin the container to specific cores, which limits where it runs but not how much it uses on those cores.

docker run --cpus=1.5 my-api
docker run --cpu-shares=512 my-worker
docker run --cpuset-cpus="0,1" my-db

Details such as how cgroup version affects these settings can differ between hosts, so check the Docker and kernel documentation for your environment.

Why Does a Quota Cause Latency Spikes?

The quota is enforced per period, not as a smooth average. A multi-threaded service that starts many threads at once can burn the whole period budget in a few milliseconds across several cores, then sit frozen until the period ends. Requests that arrive during that pause wait, which shows up as tail latency.

For example, a container limited to one CPU can run four threads for 25 milliseconds and exhaust its 100 millisecond budget. For the remaining 75 milliseconds, it is throttled. Average CPU utilization over a minute may look modest, yet individual requests experience delays that appear random.

Runtimes make this worse when they assume they have more cores than the quota allows. A language runtime that sizes its thread pool from the host core count, instead of the container limit, creates more parallelism than the quota can sustain. Check how your runtime detects container limits, since support differs between versions.

How Can You Detect Throttling?

Look at the cgroup CPU statistics. They record how many periods occurred, how many were throttled, and for how long. A rising throttled count is the clearest sign that a quota is hurting you, and it often explains latency that no application metric can.

cat /sys/fs/cgroup/cpu.stat
# look for nr_throttled and throttled_usec (cgroup v2)

Container monitoring tools can export these values. Alert on the ratio of throttled periods, and correlate it with request latency percentiles. If p99 latency rises together with throttling, the quota is too tight for the way the service uses CPU.

How Should You Size CPU for Latency Sensitive Services?

Match the setting to the workload. Services with strict latency goals usually do better with generous limits, or with no hard limit but proper shares, so bursts can use idle capacity. Batch jobs and noisy neighbors are the better candidates for strict caps.

• Set the runtime's worker and thread counts to match the CPU you actually get.
• Use shares to prioritize critical containers when the host is busy.
• Reserve headroom on the host so containers rarely compete at peak.
• Load test with the production limits in place, and watch p99 latency and throttling together.
• Pin cores only when you understand the scheduling and NUMA effects.

Orchestrators have their own terms, such as requests and limits, that map onto these mechanisms. Their documentation explains how scheduling and throttling work in that platform, so read it before copying values from a plain Docker setup.

Frequently Asked Questions

Does cpus=1 mean the container uses one core?

It means one core's worth of CPU time per period, spread over any cores. Several threads can run in parallel until the budget is gone.

Are CPU shares a limit?

No. They are relative weights that apply only under contention. An idle host lets a container use more than its share.

Why is my service slow when CPU usage is low?

It may be throttled. Check throttling counters, because averages hide short bursts that hit the quota.

Should I remove all CPU limits?

Not necessarily. Limits protect neighbors. Use them deliberately, with enough headroom for latency sensitive workloads.

Summary

Quotas are hard caps that can throttle bursts and raise tail latency, while shares and cpusets work differently. Monitor throttling, align runtime thread counts with the real limit, and size CPU generously for latency critical services. Confirm cgroup behavior in the documentation for your platform.