> For the complete documentation index, see [llms.txt](https://docs.umh.app/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.umh.app/production/cpu-health.md). # CPU Health {% hint style="info" %} **Early access.** This reporting needs UMH Core v0.44.37 or later, started with `-e USE_FSMV2_CPU=true` added to the `docker run` command that starts the container. The flag is read at startup, so changing it means recreating the container. The CPU health check does not need Management Console credentials: it also runs on an instance started without `AUTH_TOKEN` and `API_URL`. Without the flag, an instance is marked degraded whenever its CPU usage stays above 70% of the cores it may use, whether or not work is being delayed. {% endhint %} UMH Core reports whether an instance has the CPU it needs. The Management Console shows the result as a CPU status on the instance's detail page, and while that status is degraded UMH will not start another bridge there. High usage on its own does not make an instance degraded: it degrades when its headroom is gone, or when its work is measurably delayed, which UMH reads from throttling, CPU pressure and steal. (For the reasoning, see [why average CPU utilization is the wrong signal](https://www.theocharis.dev/blog/why-we-should-get-rid-of-average-cpu-utilization/).) What UMH measures headroom against depends on whether the container has a CPU limit (a Docker `--cpus` or a Kubernetes CPU limit). With no limit, the machine is the ceiling: UMH averages how busy the machine is over 60 seconds and reports degraded when less than about one core is free. It also shows the container's own usage, so you can see how much of the machine total is UMH. With a limit, the limit is the ceiling: UMH measures headroom against those cores rather than the machine, and reports degraded once its usage passes 90% of the limit, the last 10% being held in reserve, or when the kernel throttles it. ## What each CPU status means | Status | What it means | What to do | | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **CPU healthy** | The instance has the CPU it needs. Usage is shown for context, for example "1.2 of 4 cores". | Nothing. | | **CPU healthy, limited visibility** | The instance looks fine, but UMH cannot fully measure CPU health here: no CPU limit is set and the operating system is not reporting CPU-pressure statistics. | For full monitoring, set a CPU limit, or [enable CPU pressure stats](#enabling-cpu-pressure-stats). | | **CPU limited** | The instance hit its CPU limit and was paused until the next scheduling cycle, for example in 12% of cycles over the last minute. Work is being delayed. | Raise the CPU limit, or reduce the load on the instance. | | **CPU contention** | Tasks inside the instance spent time waiting for a free CPU core, for example 23% of the last minute. | Reduce the load, or give the instance more CPU. Workloads sharing the server may be competing for it. | | **CPU taken by the server** | Other virtual machines on the same physical server took CPU this instance needed. | On your virtualization platform, give this VM more guaranteed CPU, or move the other VMs off the server. | | **CPU running near full** | There is no room left for the next burst of work, because either the machine is full or the instance is at its CPU limit. The status message says which. | Add CPU capacity, reduce load, or raise the CPU limit. If the host is full and this container is not the cause, reserve or pin cores for UMH, or reduce what else runs on the machine: a CPU limit caps UMH rather than protecting it. | If UMH cannot read a file it needs to measure the CPU, the CPU health is degraded and the message names the file, for example `CPU not measured: open /sys/fs/cgroup/cpu.stat: no such file or directory`. To check whether the container can read that file, run `docker exec cat `, for example `docker exec umh-core cat /sys/fs/cgroup/cpu.stat`. ## Where to see the CPU health The Management Console shows CPU on the instance's detail page: the status, the usage row, and a Technical Details section listing every signal UMH can measure on that machine. Both the status and Technical Details carry the thresholds. A healthy instance states its remaining headroom, and each Technical Details line gives the reading next to the mark that would change it: ``` CPU healthy. This instance is using 0.3 of 2 cores (15% of its limit) and can use 1.5 more before it is marked degraded. Technical Details: Instance headroom 1.5 cores = 2 total - 0.3 used - 0.2 reserved (degrades below 0). Instance usage not measured (throttling already shows whether this instance is hitting its CPU limit, so this rough estimate is not needed). Throttling 2% (degrades above 5%). Pressure 4% (degrades above 20%). Steal not measured (this instance is not running in a virtual machine, so no other virtual machine can take its CPU). ``` Only the headroom lines are conditional: a machine whose core count is readable also gets a Machine headroom line, and an instance with no CPU limit gets no Instance headroom line. Every other signal keeps its line whether or not this machine can measure it. A signal that has already fired shows what would clear it instead, for example `Throttling 12% (recovers below 3%)`. A signal this machine cannot measure names the fact that stops it being measured, rather than reading zero. ## Thresholds Each signal degrades at one value and recovers at another, held apart so the status does not flicker. Which direction is worse depends on the signal: throttling, pressure, steal and usage are worse the higher they read, headroom the lower. All readings are 60-second figures. A reading has to pass a threshold, not merely reach it, except where the table says "at". | Signal | Degrades | Recovers | Measured only when | | -------------------- | ------------------------------ | ---------------------- | ------------------------------------------------------------------------------------------------------- | | **Throttling** | above 5% of scheduling periods | below 3% | a CPU limit is set | | **CPU pressure** | above 20% (PSI `avg60`) | below 12% | the container can read `cpu.pressure` (see [Enabling CPU pressure stats](#enabling-cpu-pressure-stats)) | | **CPU steal** | above 10% | below 6% | the machine is a virtual machine | | **Machine headroom** | less than 1 core free | 1.5 cores free | the machine's core count is readable | | **Limit headroom** | usage past 90% of the limit | below 85% of the limit | a CPU limit is set | | **Instance usage** | at 70% | below 60% | no CPU limit is set and the kernel publishes no PSI (last-resort fallback for machine headroom) | Steal uses the 95th percentile once 20 samples are in, and the mean before that, so a fresh instance is judgeable within seconds of starting. Bare metal reports no steal at all, so on a physical machine that signal reads "not measured" with its reason rather than 0%. ## Enabling CPU pressure stats UMH reads CPU pressure from the container's own `/sys/fs/cgroup/cpu.pressure`. That file exists only if the kernel booted with Pressure Stall Information (PSI) on and the machine runs cgroup v2. Most distributions ship both on. The Red Hat family is the exception: RHEL, Rocky, AlmaLinux and Oracle Linux ship PSI off, and version 8 of each also boots cgroup v1. Check your own machine: ```bash docker exec umh-core cat /sys/fs/cgroup/cpu.pressure ``` A line beginning `some avg10=` means there is nothing to do. `No such file or directory` means one or both are missing. Find out which, on the host: ```bash stat -fc %T /sys/fs/cgroup ``` `cgroup2fs` means only PSI is missing: set `psi=1`. `tmpfs` means the machine runs cgroup v1, where `cpu.pressure` does not exist at all. On cgroup v1, UMH reads the container's usage, throttling and CPU limit from the v1 controller files, and machine headroom and steal from `/proc/stat`. CPU pressure is the only signal it cannot read, because cgroup v1 has no per-container pressure file. ### Set the kernel parameters Each operating system sets kernel parameters differently. Follow yours, then reboot: | Operating system | Instructions | | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | RHEL, Rocky, AlmaLinux, Oracle Linux, Fedora | [Configuring kernel command-line parameters](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_monitoring_and_updating_the_kernel/configuring-kernel-command-line-parameters_managing-monitoring-and-updating-the-kernel): `sudo grubby --update-kernel=ALL --args="psi=1"` | | Debian and Ubuntu | [GRUB 2 setup](https://help.ubuntu.com/community/Grub2/Setup): add `psi=1` to `GRUB_CMDLINE_LINUX` in `/etc/default/grub`, then run `sudo update-grub` | `update-grub` is a Debian and Ubuntu wrapper. On another GRUB machine, edit `GRUB_CMDLINE_LINUX` the same way and run `grub-mkconfig -o /boot/grub/grub.cfg`. ### Confirm UMH can read the file Run the `docker exec` check again. It reads the same file UMH reads. The Management Console shows the result on the instance's detail page. The Technical Details `Pressure` line changes from `Pressure not available (not possible).` to a percentage. `Pressure not available (measuring).` means UMH can read the file and is filling its 60-second window, so wait a minute and look again. If the file is still missing, run `cat /proc/pressure/cpu` on the host. Output means PSI is on and cgroup v2 is what is missing. `No such file or directory` means the kernel ignored `psi=1`, which happens on a kernel built without PSI. If the kernel has no PSI, set a CPU limit on the container instead. UMH then judges the container against its limit. Pressure stays unavailable. ## When a new bridge waits for CPU While CPU is degraded, UMH will not start an additional bridge on the instance, because it would compete for CPU that is already short. The bridge waits, and its status reason names the resource that stopped it, here the CPU. For every reason a bridge waits and the full procedure, see [Bridges Do Not Start](/usage/data-flows/bridges/bridges-do-not-start.md). Bridges already running are left alone. To start bridges anyway in an emergency, use the emergency setting *Turn off bridge admission*. It turns off every check, not only CPU: ```yaml agent: enableResourceLimitBlocking: false ``` While it is `false`, the instance shows as degraded. This is separate from the bridge limit, the number of bridges a given core count can hold, which the [Sizing Guide](/production/sizing-guide.md) covers. The bridge limit is an upper bound, not a guarantee: because real CPU use varies per bridge, a bridge can wait on CPU health before you reach it. ## Known limitation A container pinned to a subset of the machine's CPUs (`cpuset`) with no CPU limit set can print a usage pair measured against different core counts, such as "using 6.0 of 2 cores", beside a healthy verdict. Set a CPU limit on such a container to get a correct reading. ## Glossary | Term | Meaning | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **Throttling** | The kernel caps a container to its CPU limit in short repeating periods, about 100 ms each. A container that needs more within a period is paused until the next one, so a workload whose average looks fine can still be paused during bursts. | | **CPU pressure** | How much time tasks spent waiting for a free CPU core, from Linux Pressure Stall Information (PSI). High pressure means CPU is the bottleneck. | | **CPU steal** | Time the hypervisor scheduled this machine's CPU elsewhere. High steal usually means the physical server is oversubscribed. A burstable cloud instance whose CPU credits have run out is capped by a different mechanism, but the guest counts that capped time as steal too. | | **Host contention** | CPU used by software outside UMH on the same machine. UMH cannot see which processes those are, only that they are using CPU it needs. It is not the **CPU contention** status above: that one is measured inside the instance, as tasks of UMH's own waiting for a free core. | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.umh.app/production/cpu-health.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.