CPU Health
UMH Core reports whether an instance has the CPU it needs. The Management Console shows the result as a CPU status on the instance's detail page, and while that status is degraded UMH will not start another bridge there. High usage on its own does not make an instance degraded: it degrades when its headroom is gone, or when its work is measurably delayed, which UMH reads from throttling, CPU pressure and steal. (For the reasoning, see why average CPU utilization is the wrong signal.)
What UMH measures headroom against depends on whether the container has a CPU limit (a Docker --cpus or a Kubernetes CPU limit). With no limit, the machine is the ceiling: UMH averages how busy the machine is over 60 seconds and reports degraded when less than about one core is free. It also shows the container's own usage, so you can see how much of the machine total is UMH. With a limit, the limit is the ceiling: UMH measures headroom against those cores rather than the machine, and reports degraded once its usage passes 90% of the limit, the last 10% being held in reserve, or when the kernel throttles it.
What each CPU status means
CPU healthy
The instance has the CPU it needs. Usage is shown for context, for example "1.2 of 4 cores".
Nothing.
CPU healthy, limited visibility
The instance looks fine, but UMH cannot fully measure CPU health here: no CPU limit is set and the operating system is not reporting CPU-pressure statistics.
For full monitoring, set a CPU limit, or enable CPU pressure stats.
CPU limited
The instance hit its CPU limit and was paused until the next scheduling cycle, for example in 12% of cycles over the last minute. Work is being delayed.
Raise the CPU limit, or reduce the load on the instance.
CPU contention
Tasks inside the instance spent time waiting for a free CPU core, for example 23% of the last minute.
Reduce the load, or give the instance more CPU. Workloads sharing the server may be competing for it.
CPU taken by the server
Other virtual machines on the same physical server took CPU this instance needed.
On your virtualization platform, give this VM more guaranteed CPU, or move the other VMs off the server.
CPU running near full
There is no room left for the next burst of work, because either the machine is full or the instance is at its CPU limit. The status message says which.
Add CPU capacity, reduce load, or raise the CPU limit. If the host is full and this container is not the cause, reserve or pin cores for UMH, or reduce what else runs on the machine: a CPU limit caps UMH rather than protecting it.
If UMH cannot read a file it needs to measure the CPU, the CPU health is degraded and the message names the file, for example CPU not measured: open /sys/fs/cgroup/cpu.stat: no such file or directory. To check whether the container can read that file, run docker exec <container> cat <file>, for example docker exec umh-core cat /sys/fs/cgroup/cpu.stat.
Where to see the CPU health
The Management Console shows CPU on the instance's detail page: the status, the usage row, and a Technical Details section listing every signal UMH can measure on that machine.
Both the status and Technical Details carry the thresholds. A healthy instance states its remaining headroom, and each Technical Details line gives the reading next to the mark that would change it:
CPU healthy. This instance is using 0.3 of 2 cores (15% of its limit) and can use 1.5 more before it is marked degraded.
Technical Details:
Instance headroom 1.5 cores = 2 total - 0.3 used - 0.2 reserved (degrades below 0).
Instance usage not measured (throttling already shows whether this instance is hitting its CPU limit, so this rough estimate is not needed).
Throttling 2% (degrades above 5%).
Pressure 4% (degrades above 20%).
Steal not measured (this instance is not running in a virtual machine, so no other virtual machine can take its CPU).Only the headroom lines are conditional: a machine whose core count is readable also gets a Machine headroom line, and an instance with no CPU limit gets no Instance headroom line. Every other signal keeps its line whether or not this machine can measure it.
A signal that has already fired shows what would clear it instead, for example Throttling 12% (recovers below 3%). A signal this machine cannot measure names the fact that stops it being measured, rather than reading zero.
Thresholds
Each signal degrades at one value and recovers at another, held apart so the status does not flicker. Which direction is worse depends on the signal: throttling, pressure, steal and usage are worse the higher they read, headroom the lower. All readings are 60-second figures. A reading has to pass a threshold, not merely reach it, except where the table says "at".
Throttling
above 5% of scheduling periods
below 3%
a CPU limit is set
CPU pressure
above 20% (PSI avg60)
below 12%
the container can read cpu.pressure (see Enabling CPU pressure stats)
CPU steal
above 10%
below 6%
the machine is a virtual machine
Machine headroom
less than 1 core free
1.5 cores free
the machine's core count is readable
Limit headroom
usage past 90% of the limit
below 85% of the limit
a CPU limit is set
Instance usage
at 70%
below 60%
no CPU limit is set and the kernel publishes no PSI (last-resort fallback for machine headroom)
Steal uses the 95th percentile once 20 samples are in, and the mean before that, so a fresh instance is judgeable within seconds of starting. Bare metal reports no steal at all, so on a physical machine that signal reads "not measured" with its reason rather than 0%.
Enabling CPU pressure stats
UMH reads CPU pressure from the container's own /sys/fs/cgroup/cpu.pressure. That file exists only if the kernel booted with Pressure Stall Information (PSI) on and the machine runs cgroup v2. Most distributions ship both on. The Red Hat family is the exception: RHEL, Rocky, AlmaLinux and Oracle Linux ship PSI off, and version 8 of each also boots cgroup v1.
Check your own machine:
A line beginning some avg10= means there is nothing to do. No such file or directory means one or both are missing. Find out which, on the host:
cgroup2fs means only PSI is missing: set psi=1. tmpfs means the machine runs cgroup v1, where cpu.pressure does not exist at all.
On cgroup v1, UMH reads the container's usage, throttling and CPU limit from the v1 controller files, and machine headroom and steal from /proc/stat. CPU pressure is the only signal it cannot read, because cgroup v1 has no per-container pressure file.
Set the kernel parameters
Each operating system sets kernel parameters differently. Follow yours, then reboot:
RHEL, Rocky, AlmaLinux, Oracle Linux, Fedora
Configuring kernel command-line parameters: sudo grubby --update-kernel=ALL --args="psi=1"
Debian and Ubuntu
GRUB 2 setup: add psi=1 to GRUB_CMDLINE_LINUX in /etc/default/grub, then run sudo update-grub
update-grub is a Debian and Ubuntu wrapper. On another GRUB machine, edit GRUB_CMDLINE_LINUX the same way and run grub-mkconfig -o /boot/grub/grub.cfg.
Confirm UMH can read the file
Run the docker exec check again. It reads the same file UMH reads.
The Management Console shows the result on the instance's detail page. The Technical Details Pressure line changes from Pressure not available (not possible). to a percentage. Pressure not available (measuring). means UMH can read the file and is filling its 60-second window, so wait a minute and look again.
If the file is still missing, run cat /proc/pressure/cpu on the host. Output means PSI is on and cgroup v2 is what is missing. No such file or directory means the kernel ignored psi=1, which happens on a kernel built without PSI. If the kernel has no PSI, set a CPU limit on the container instead. UMH then judges the container against its limit. Pressure stays unavailable.
When a new bridge waits for CPU
While CPU is degraded, UMH will not start an additional bridge on the instance, because it would compete for CPU that is already short. The bridge waits, and its status reason names the resource that stopped it, here the CPU. For every reason a bridge waits and the full procedure, see Bridges Do Not Start. Bridges already running are left alone.
To start bridges anyway in an emergency, use the emergency setting Turn off bridge admission. It turns off every check, not only CPU:
While it is false, the instance shows as degraded.
This is separate from the bridge limit, the number of bridges a given core count can hold, which the Sizing Guide covers. The bridge limit is an upper bound, not a guarantee: because real CPU use varies per bridge, a bridge can wait on CPU health before you reach it.
Known limitation
A container pinned to a subset of the machine's CPUs (cpuset) with no CPU limit set can print a usage pair measured against different core counts, such as "using 6.0 of 2 cores", beside a healthy verdict. Set a CPU limit on such a container to get a correct reading.
Glossary
Throttling
The kernel caps a container to its CPU limit in short repeating periods, about 100 ms each. A container that needs more within a period is paused until the next one, so a workload whose average looks fine can still be paused during bursts.
CPU pressure
How much time tasks spent waiting for a free CPU core, from Linux Pressure Stall Information (PSI). High pressure means CPU is the bottleneck.
CPU steal
Time the hypervisor scheduled this machine's CPU elsewhere. High steal usually means the physical server is oversubscribed. A burstable cloud instance whose CPU credits have run out is capped by a different mechanism, but the guest counts that capped time as steal too.
Host contention
CPU used by software outside UMH on the same machine. UMH cannot see which processes those are, only that they are using CPU it needs. It is not the CPU contention status above: that one is measured inside the instance, as tasks of UMH's own waiting for a free core.
Last updated