Optimizing cAdvisor for Lower CPU Usage

Introduction

I have been running couple of services in my homelab via docker containers. For monitoring purposes I use Grafana and Prometheus. I have been using cAdvisor ↗️ to monitor my docker containers.

Ever since I got the orange pi server, I started to see a noticable difference in the CPU usage being reported by cAdvisor container, it was always hovering around 15-20% CPU usage (which is way higher than any of the containers that I was running in idle state).

I started to investigate this and found a way to reduce the CPU usage of cAdvisor. Let’s see how I did it.

Solution

I won’t go into the details of what is cAdvisor since I believe if you are here you already know what it is. So let’s jump into the solution.

Here’s what my (and probably your’s as well) docker-compose file looked like:

services:
  cadvisor:
    image: "gcr.io/cadvisor/cadvisor-arm64:v0.49.1"
    container_name: "cadvisor"
    ports:
      - "8080:8080"
    devices:
      - /dev/kmsg:/dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    restart: unless-stopped

The above configuration is the default configuration that is provided in the cAdvisor documentation ↗️. The problem is with the default configuration and how cAdvisor collects the metrics.

Here are the configurations that I changed from the default configuration:

• -housekeeping_interval from 1s to 10s
• -docker_only from false to true

Here’s how my updated docker-compose file looks like:

services:
  cadvisor:
    image: 'gcr.io/cadvisor/cadvisor-arm64:v0.49.1'
    container_name: 'cadvisor'
    command:
      - '-housekeeping_interval=10s'
      - '-docker_only=true'
    ports:
      - '8080:8080'
    devices:
      - /dev/kmsg:/dev/kmsg
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    restart: unless-stopped

The major improvement came from the -docker_only=true flag. The -docker_only flag tells cAdvisor to only monitor the docker containers and not any other types of container runtime like Kubernetes. If you are also running only docker containers then setting this flag to true will help in reducing the overhead of collecting unnecessary metrics.

Another improvement came from the -housekeeping_interval=10s flag. This flag tells cAdvisor to collect the metrics every 10 seconds instead of the default 1 second. You can tweak this number based on your requirements but after doing some trail and error I found that 10 seconds was the sweet spot for me.

With this two changes I was able to reduce the CPU usage of cAdvisor from 15-16% to 5-6% which is a huge improvement.

Conclusion

If you are seeing high CPU usage from cAdvisor then you can try the above changes to reduce the CPU usage. We explore the two flags -housekeeping_interval and -docker_only which helped in reducing the CPU usage of cAdvisor.

If you have any questions or suggestions, feel free to reach out to me on Twitter ↗️ / Reddit ↗️ or leave a comment below.

Happy monitoring!