Description
current limit of 512MiB set to akash-inventory-operator might be too small
https://github.com/akash-network/helm-charts/blob/provider-8.0.3/charts/akash-inventory-operator/templates/deployment.yaml#L36
$ kubectl -n akash-services get pods
NAME READY STATUS RESTARTS AGE
akash-hostname-operator-6795445db-jf46g 1/1 Running 0 27m
akash-inventory-operator-75d7758b86-kqk6s 1/1 Running 4 (68s ago) 29m
akash-node-1-0 1/1 Running 0 48m
akash-provider-0 1/1 Running 0 26m
root@node3:~# dmesg -T -l alert -l crit -l emerg -l err
...
[Tue Feb 20 19:10:18 2024] Memory cgroup out of memory: Killed process 80061 (provider-servic) total-vm:5025880kB, anon-rss:517808kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1620kB oom_score_adj:999
[Tue Feb 20 19:17:12 2024] Memory cgroup out of memory: Killed process 125599 (provider-servic) total-vm:5173344kB, anon-rss:516256kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1636kB oom_score_adj:999
[Tue Feb 20 19:17:12 2024] Memory cgroup out of memory: Killed process 125701 (provider-servic) total-vm:5173344kB, anon-rss:516256kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1636kB oom_score_adj:999
[Tue Feb 20 19:24:38 2024] Memory cgroup out of memory: Killed process 166260 (provider-servic) total-vm:5322344kB, anon-rss:517112kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1660kB oom_score_adj:999
[Tue Feb 20 19:24:38 2024] Memory cgroup out of memory: Killed process 166287 (provider-servic) total-vm:5322344kB, anon-rss:517112kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1660kB oom_score_adj:999
Env
provider:
sg.lneq
$ kubectl -n akash-services get pods -o custom-columns='NAME:.metadata.name,IMAGE:.spec.containers[*].image'
NAME IMAGE
akash-hostname-operator-6795445db-jf46g ghcr.io/akash-network/provider:0.4.8
akash-inventory-operator-544c75d855-qs8lh ghcr.io/akash-network/provider:0.4.8
akash-node-1-0 ghcr.io/akash-network/node:0.30.0
akash-provider-0 ghcr.io/akash-network/provider:0.4.8
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node1 Ready control-plane 71m v1.28.6 10.74.43.129 <none> Ubuntu 22.04.4 LTS 6.5.0-18-generic containerd://1.7.11
node2 Ready control-plane 71m v1.28.6 10.74.43.133 <none> Ubuntu 22.04.4 LTS 6.5.0-18-generic containerd://1.7.11
node3 Ready <none> 69m v1.28.6 10.74.43.131 <none> Ubuntu 22.04.4 LTS 6.5.0-18-generic containerd://1.7.11
node4 Ready <none> 69m v1.28.6 10.8.68.129 <none> Ubuntu 22.04.4 LTS 6.5.0-18-generic containerd://1.7.11
$ kubectl -n rook-ceph exec -i $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- ceph status
cluster:
id: 69d6af8d-3dfa-47cd-8f6e-bcbc5320987f
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 42m)
mgr: b(active, since 39m), standbys: a
osd: 8 osds: 8 up (since 40m), 8 in (since 40m)
data:
pools: 2 pools, 257 pgs
objects: 7 objects, 577 KiB
usage: 4.9 GiB used, 28 TiB / 28 TiB avail
pgs: 257 active+clean
$ kubectl -n rook-ceph exec -i $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 27.94470 root default
-5 6.98618 host node1
2 nvme 3.49309 osd.2 up 1.00000 1.00000
5 nvme 3.49309 osd.5 up 1.00000 1.00000
-9 6.98618 host node2
6 nvme 3.49309 osd.6 up 1.00000 1.00000
7 nvme 3.49309 osd.7 up 1.00000 1.00000
-3 6.98618 host node3
1 nvme 3.49309 osd.1 up 1.00000 1.00000
4 nvme 3.49309 osd.4 up 1.00000 1.00000
-7 6.98618 host node4
0 nvme 3.49309 osd.0 up 1.00000 1.00000
3 nvme 3.49309 osd.3 up 1.00000 1.00000
Observation: 3 vs 4 nodes
Interestingly, comparing this provider sg.lneq to sg.lnlm, the latter doesn't experience this issue:
$ kubectl -n akash-services get pods
NAME READY STATUS RESTARTS AGE
akash-hostname-operator-6795445db-5xhq5 1/1 Running 0 5d7h
akash-inventory-operator-75d7758b86-gh2wj 1/1 Running 0 5d6h
akash-node-1-0 1/1 Running 0 5d7h
akash-provider-0 1/1 Running 0 21h
$ kubectl -n rook-ceph exec -i $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- ceph status
cluster:
id: 661a3fe0-5ff2-4575-a421-f812501f463c
health: HEALTH_OK
services:
mon: 3 daemons, quorum a,b,c (age 5d)
mgr: a(active, since 5d), standbys: b
osd: 6 osds: 6 up (since 5d), 6 in (since 5d)
data:
pools: 2 pools, 257 pgs
objects: 491 objects, 889 MiB
usage: 9.8 GiB used, 5.2 TiB / 5.2 TiB avail
pgs: 257 active+clean
io:
client: 341 B/s wr, 0 op/s rd, 0 op/s wr
$ kubectl -n rook-ceph exec -i $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') -- ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 5.23975 root default
-3 1.74658 host node1
0 nvme 0.87329 osd.0 up 1.00000 1.00000
2 nvme 0.87329 osd.2 up 1.00000 1.00000
-5 1.74658 host node2
1 nvme 0.87329 osd.1 up 1.00000 1.00000
4 nvme 0.87329 osd.4 up 1.00000 1.00000
-7 1.74658 host node3
3 nvme 0.87329 osd.3 up 1.00000 1.00000
5 nvme 0.87329 osd.5 up 1.00000 1.00000
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node1 Ready control-plane 5d8h v1.28.6 192.168.0.100 <none> Ubuntu 22.04.3 LTS 5.15.0-94-generic containerd://1.7.11
node2 Ready control-plane 5d8h v1.28.6 192.168.0.101 <none> Ubuntu 22.04.3 LTS 5.15.0-94-generic containerd://1.7.11
node3 Ready <none> 5d8h v1.28.6 192.168.0.102 <none> Ubuntu 22.04.3 LTS 5.15.0-94-generic containerd://1.7.11
Next steps
I've lifted the RAM limit to 1GiB to see if it helps.