Linux CGroup (Control Group)

在擁有多個獨立的namespace之後，如何有效的分配既有的資源，就成了另一個課題。這些都是交由Linux kernel的CGroup功能負責，限制、控制、分離Control Group的資源（包括：CPU, Memory, I/O等等）。CGroup主要的功能有

• 限制資源上限

• 優先權決定

• 計算資源使用情況

• 將process group掛起或恢復

查看/sys/fs/cgroup底下有哪些項目

root@vm:/sys/fs/cgroup# ls -all
總計 0
drwxr-xr-x 15 root root 380  7月  1 20:50 .
drwxr-xr-x  9 root root   0  7月  1 19:49 ..
dr-xr-xr-x  5 root root   0  7月  1 20:50 blkio
lrwxrwxrwx  1 root root  11  7月  1 20:50 cpu -> cpu,cpuacct
lrwxrwxrwx  1 root root  11  7月  1 20:50 cpuacct -> cpu,cpuacct
dr-xr-xr-x  5 root root   0  7月  1 20:50 cpu,cpuacct
dr-xr-xr-x  3 root root   0  7月  1 20:50 cpuset
dr-xr-xr-x  5 root root   0  7月  1 20:50 devices
dr-xr-xr-x  3 root root   0  7月  1 20:50 freezer
dr-xr-xr-x  3 root root   0  7月  1 20:50 hugetlb
dr-xr-xr-x  5 root root   0  7月  1 20:50 memory
lrwxrwxrwx  1 root root  16  7月  1 20:50 net_cls -> net_cls,net_prio
dr-xr-xr-x  3 root root   0  7月  1 20:50 net_cls,net_prio
lrwxrwxrwx  1 root root  16  7月  1 20:50 net_prio -> net_cls,net_prio
dr-xr-xr-x  3 root root   0  7月  1 20:50 perf_event
dr-xr-xr-x  5 root root   0  7月  1 20:50 pids
dr-xr-xr-x  2 root root   0  7月  1 20:50 rdma
dr-xr-xr-x  6 root root   0  7月  1 20:50 systemd
dr-xr-xr-x  5 root root   0  7月  1 20:50 unified

當Docker建立一個容器時，會在cgroup底下建立資料夾，我在本機擁有一個portainer容器，容器ID為64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1，那麼在cgroup底下我可以找到這個容器ID的資料夾，裡面列出對各項資源的限制及統計，包括memory, cpu等等

root@BigData10:/home/jennifer# docker ps
CONTAINER ID        IMAGE                            COMMAND                  CREATED             STATUS              PORTS                                                                                        NAMES
64577c13ad58        portainer/portainer              "/portainer"             13 months ago       Up 19 hours         0.0.0.0:9000->9000/tcp                                                                       portainer

root@vm:/sys/fs/cgroup# find iname 64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./perf_event/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./blkio/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./net_cls,net_prio/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./memory/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./cpuset/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./pids/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./cpu,cpuacct/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./freezer/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./devices/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./hugetlb/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1
./systemd/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1

tasks顯示了64577容器內的process有哪些

root@vm:/sys/fs/cgroup/memory/docker/64577c13ad5842d3183b14f7d082b38aa8ba463aa51d2d7f9afe0179720227b1# cat tasks
2501
2822
2823
2824
2825
2927
2962
2963
3720
3721
4422
4423
6541

利用top查詢目前各process佔用資源的情況

root@:/sys/fs# top
top - 16:52:19 up 20:02,  1 user,  load average: 0.13, 0.08, 0.02
Tasks: 212 total,   1 running, 163 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.2 us,  0.0 sy,  0.0 ni, 99.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 16411728 total,  8069872 free,  6456780 used,  1885076 buff/cache
KiB Swap:   999420 total,   999420 free,        0 used.  9613404 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2550 999       20   0 5922284 4.181g  22672 S   0.3 26.7   6:41.87 java
 3117 root      20   0 1534032 414212  28416 S   0.3  2.5   3:05.76 uwsgi
 3511 root      20   0 2441536 233720  35252 S   0.3  1.4   5:08.07 python

PID=2550佔用主機26.7%的記憶體，先查看看他是屬於哪一個容器的task，再用容器id回查容器狀態

root@vm:/sys/fs/cgroup/memory/docker# grep 2550 /sys/fs/cgroup/cpu/docker/*/tasks
/sys/fs/cgroup/cpu/docker/c4bf14a28bd088b462496a9aa4f73f0433257768b8d48f40a72dfdff649a52ab/tasks:2550

root@vm:/sys/fs/cgroup/memory/docker#  docker ps | grep c4b
c4bf14a28bd0        cassandra                        "docker-entrypoint.s…"   4 weeks ago         Up 20 hours         7001/tcp, 0.0.0.0:7000->7000/tcp, 0.0.0.0:9042->9042/tcp, 0.0.0.0:9160->9160/tcp, 7199/tcp   cassandra-1

也可利用docker stats查看各容器目前使用資源的情況

root@vm:/home/jennifer# docker stats cassandra-1 portainer
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
c4bf14a28bd0        cassandra-1         0.57%               4.303GiB / 15.65GiB   27.49%              341MB / 342MB       313MB / 4.22GB      128
64577c13ad58        portainer           0.00%               14.12MiB / 15.65GiB   0.09%               553kB / 5.82MB      48.8MB / 268MB      14