Date
1 - 4 of 4
Attempting to install Strimzi/Kafka on my microk8s cluster
tfv01@...
Hi,
I have a three node Raspberry Pi cluster on which I'd like to be able to install and run Kafka. Strimzi seems to be a very easy environment to get that working and I read that as-of Dec 2021, it supports ARM-based processors. I was following along the Quick Start Guide (Strimzi Quick Start guide (0.28.0) and got to step 2.4 Creating a Cluster. In this step (as-in all steps before, I am prefacing all "kubectl...." commands with "microk8s kubectl..."). Step 2.4 outputs: "kafka.kafka.strimzi.io/my-cluster created" But the next command times-out because the cluster never becomes Ready: microk8s kubectl wait kafka/my-cluster --for=condition=Ready --timeout=300s -n my-kafka-project When I look at the Kubernetes Dashboard, I see the following Events under the "my-kafka-project" Namespace: Message Source no persistent volumes available for this claim and no storage class is set persistentvolume-controller
0/4 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims, 3 node(s) had taint {node.kubernetes.io/unreachable: }, that the pod didn't tolerate. default-scheduler
0/4 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims, 3 node(s) had taint {node.kubernetes.io/unreachable: }, that the pod didn't tolerate. default-scheduler
0/4 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims, 3 node(s) had taint {node.kubernetes.io/unreachable: }, that the pod didn't tolerate. default-scheduler
0/4 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims, 3 node(s) had taint {node.kubernetes.io/unreachable: }, that the pod didn't tolerate. default-scheduler
No matching pods found controllermanager
create Pod my-cluster-zookeeper-0 in StatefulSet my-cluster-zookeeper successful statefulset-controller
Thank you! -Tim |
|
Jakub Scholz
Hi Tim, That looks like some scheduling issue between your cluster and your storage. TBH, these issues are a bit hard to decode without your environment. The taint suggests that some of your nodes are unreachable, so that might be the cause of the issue? Thanks & Regards Jakub On Sun, May 8, 2022 at 12:56 AM <tfv01@...> wrote: Hi, |
|
tfv01@...
Thank you Jakub, appreciate your response!
I'm not exactly sure I understand how to check if each node is "reachable", but here is the output of "kubectl describe nodes", and I don't know if the fact that all three Pis are listed means that they are in-fact reachable. -Tim ubuntu@pi4-01:~$ microk8s kubectl describe nodes Name: pi4-01
Roles: <none>
Labels: beta.kubernetes.io/arch=arm64
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm64
kubernetes.io/hostname=pi4-01
kubernetes.io/os=linux
microk8s.io/cluster=true
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 05 Jul 2021 22:02:48 +0000
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: pi4-01
AcquireTime: <unset>
RenewTime: Sat, 07 May 2022 23:34:18 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure False Sat, 07 May 2022 23:29:57 +0000 Mon, 05 Jul 2021 22:02:46 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Sat, 07 May 2022 23:29:57 +0000 Mon, 05 Jul 2021 22:02:46 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Sat, 07 May 2022 23:29:57 +0000 Mon, 05 Jul 2021 22:02:46 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Sat, 07 May 2022 23:29:57 +0000 Sat, 07 May 2022 21:18:54 +0000 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 10.0.0.10
Hostname: pi4-01
Capacity:
cpu: 4
ephemeral-storage: 245747040Ki
memory: 7998736Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 244698464Ki
memory: 7896336Ki
pods: 110
System Info:
Machine ID: b4d5f79c8636436cbd291ba7ac7f14c9
System UUID: b4d5f79c8636436cbd291ba7ac7f14c9
Boot ID: 3e28585a-dceb-4793-8425-f9d80f92a939
Kernel Version: 5.4.0-1052-raspi
OS Image: Ubuntu 20.04.2 LTS
Operating System: linux
Architecture: arm64
Container Runtime Version: containerd://1.2.5
Kubelet Version: v1.18.20
Kube-Proxy Version: v1.18.20
Non-terminated Pods: (10 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default zookeeper-deployment-1-6c88c95964-496xq 0 (0%) 0 (0%) 0 (0%) 0 (0%) 11d
ingress nginx-ingress-microk8s-controller-jmkgb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 306d
kafka strimzi-cluster-operator-74f9f5d7c7-gs5df 200m (5%) 1 (25%) 384Mi (4%) 384Mi (4%) 4h3m
kube-system coredns-588fd544bf-x9jxf 100m (2%) 0 (0%) 70Mi (0%) 170Mi (2%) 306d
kube-system dashboard-metrics-scraper-db65b9c6f-cq2t8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 306d
kube-system heapster-v1.5.2-5bc67ff868-2tkbf 100m (2%) 100m (2%) 184720Ki (2%) 184720Ki (2%) 306d
kube-system kubernetes-dashboard-c5b698784-wnfqj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 306d
kube-system monitoring-influxdb-grafana-v4-6cc44d985f-qstsn 200m (5%) 200m (5%) 600Mi (7%) 600Mi (7%) 306d
metallb-system controller-55d8b88d7f-tkvzq 100m (2%) 100m (2%) 100Mi (1%) 100Mi (1%) 17d
metallb-system speaker-ms5z4 100m (2%) 100m (2%) 100Mi (1%) 100Mi (1%) 124d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 800m (20%) 1500m (37%)
memory 1468816Ki (18%) 1571216Ki (19%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>
Name: pi4-02
Roles: <none>
Labels: beta.kubernetes.io/arch=arm64
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm64
kubernetes.io/hostname=pi4-02
kubernetes.io/os=linux
microk8s.io/cluster=true
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 05 Jul 2021 23:02:47 +0000
Taints: node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: pi4-02
AcquireTime: <unset>
RenewTime: Fri, 25 Feb 2022 05:08:28 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Fri, 25 Feb 2022 05:07:09 +0000 Fri, 25 Feb 2022 05:12:44 +0000 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Fri, 25 Feb 2022 05:07:09 +0000 Fri, 25 Feb 2022 05:12:44 +0000 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Fri, 25 Feb 2022 05:07:09 +0000 Fri, 25 Feb 2022 05:12:44 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Fri, 25 Feb 2022 05:07:09 +0000 Fri, 25 Feb 2022 05:12:44 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 10.0.0.11
Hostname: pi4-02
Capacity:
cpu: 4
ephemeral-storage: 245617500Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
hugepages-32Mi: 0
hugepages-64Ki: 0
memory: 7997148Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 244568924Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
hugepages-32Mi: 0
hugepages-64Ki: 0
memory: 7894748Ki
pods: 110
System Info:
Machine ID: 37c8b0e6c19c4bc09f1aace54cd91d55
System UUID: 37c8b0e6c19c4bc09f1aace54cd91d55
Boot ID: 02cb905f-359b-4584-b2c6-d20953e07322
Kernel Version: 5.11.0-1023-raspi
OS Image: Ubuntu 21.04
Operating System: linux
Architecture: arm64
Container Runtime Version: containerd://1.2.5
Kubelet Version: v1.18.20
Kube-Proxy Version: v1.18.20
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default kafka-broker0-86669885c5-xshms 0 (0%) 0 (0%) 0 (0%) 0 (0%) 90d
default nginx 0 (0%) 0 (0%) 0 (0%) 0 (0%) 125d
default zookeeper-deploy-7f5bb9785f-2tqf8 0 (0%) 0 (0%) 0 (0%) 0 (0%) 90d
ingress nginx-ingress-microk8s-controller-wzbnf 0 (0%) 0 (0%) 0 (0%) 0 (0%) 306d
metallb-system speaker-86pmc 100m (2%) 100m (2%) 100Mi (1%) 100Mi (1%) 124d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 100m (2%) 100m (2%)
memory 100Mi (1%) 100Mi (1%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
hugepages-32Mi 0 (0%) 0 (0%)
hugepages-64Ki 0 (0%) 0 (0%)
Events: <none>
Name: pi4-03
Roles: <none>
Labels: beta.kubernetes.io/arch=arm64
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm64
kubernetes.io/hostname=pi4-03
kubernetes.io/os=linux
microk8s.io/cluster=true
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 05 Jul 2021 23:03:01 +0000
Taints: node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: pi4-03
AcquireTime: <unset>
RenewTime: Fri, 25 Feb 2022 05:08:24 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Fri, 25 Feb 2022 05:03:45 +0000 Fri, 25 Feb 2022 05:12:44 +0000 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Fri, 25 Feb 2022 05:03:45 +0000 Fri, 25 Feb 2022 05:12:44 +0000 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Fri, 25 Feb 2022 05:03:45 +0000 Fri, 25 Feb 2022 05:12:44 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Fri, 25 Feb 2022 05:03:45 +0000 Fri, 25 Feb 2022 05:12:44 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 10.0.0.12
Hostname: pi4-03
Capacity:
cpu: 4
ephemeral-storage: 245617500Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
hugepages-32Mi: 0
hugepages-64Ki: 0
memory: 7997148Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 244568924Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
hugepages-32Mi: 0
hugepages-64Ki: 0
memory: 7894748Ki
pods: 110
System Info:
Machine ID: 5f870ea041464a25a73f93fb37cec85c
System UUID: 5f870ea041464a25a73f93fb37cec85c
Boot ID: 15e11c9e-4674-45c7-ac7f-021e83a17eff
Kernel Version: 5.11.0-1023-raspi
OS Image: Ubuntu 21.04
Operating System: linux
Architecture: arm64
Container Runtime Version: containerd://1.2.5
Kubelet Version: v1.18.20
Kube-Proxy Version: v1.18.20
Non-terminated Pods: (5 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default kafka-broker0-86669885c5-9k6f6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 90d
default nginx-f89759699-8zm68 0 (0%) 0 (0%) 0 (0%) 0 (0%) 124d
default zookeeper-deploy-7f5bb9785f-4j8kt 0 (0%) 0 (0%) 0 (0%) 0 (0%) 90d
ingress nginx-ingress-microk8s-controller-p6c9q 0 (0%) 0 (0%) 0 (0%) 0 (0%) 306d
metallb-system speaker-h28vg 100m (2%) 100m (2%) 100Mi (1%) 100Mi (1%) 124d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 100m (2%) 100m (2%)
memory 100Mi (1%) 100Mi (1%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
hugepages-32Mi 0 (0%) 0 (0%)
hugepages-64Ki 0 (0%) 0 (0%)
Events: <none>
Name: ubuntu
Roles: <none>
Labels: beta.kubernetes.io/arch=arm64
beta.kubernetes.io/os=linux
kubernetes.io/arch=arm64
kubernetes.io/hostname=ubuntu
kubernetes.io/os=linux
microk8s.io/cluster=true
Annotations: node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Mon, 05 Jul 2021 14:20:11 +0000
Taints: node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: ubuntu
AcquireTime: <unset>
RenewTime: Mon, 05 Jul 2021 22:03:34 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
MemoryPressure Unknown Mon, 05 Jul 2021 22:02:48 +0000 Mon, 05 Jul 2021 22:04:56 +0000 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Mon, 05 Jul 2021 22:02:48 +0000 Mon, 05 Jul 2021 22:04:56 +0000 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Mon, 05 Jul 2021 22:02:48 +0000 Mon, 05 Jul 2021 22:04:56 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Mon, 05 Jul 2021 22:02:48 +0000 Mon, 05 Jul 2021 22:04:56 +0000 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 10.0.0.11
Hostname: ubuntu
Capacity:
cpu: 4
ephemeral-storage: 245617500Ki
memory: 7997276Ki
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 244568924Ki
memory: 7894876Ki
pods: 110
System Info:
Machine ID: 37c8b0e6c19c4bc09f1aace54cd91d55
System UUID: 37c8b0e6c19c4bc09f1aace54cd91d55
Boot ID: 4aac5c6b-db60-4468-85e9-b5ef13b1e9f4
Kernel Version: 5.11.0-1007-raspi
OS Image: Ubuntu 21.04
Operating System: linux
Architecture: arm64
Container Runtime Version: containerd://1.2.5
Kubelet Version: v1.18.19
Kube-Proxy Version: v1.18.19
Non-terminated Pods: (6 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
ingress nginx-ingress-microk8s-controller-bzx2x 0 (0%) 0 (0%) 0 (0%) 0 (0%) 306d
kube-system coredns-588fd544bf-ntgwl 100m (2%) 0 (0%) 70Mi (0%) 170Mi (2%) 306d
kube-system dashboard-metrics-scraper-db65b9c6f-rl8l2 0 (0%) 0 (0%) 0 (0%) 0 (0%) 306d
kube-system heapster-v1.5.2-5bc67ff868-hl5kp 100m (2%) 100m (2%) 184720Ki (2%) 184720Ki (2%) 306d
kube-system kubernetes-dashboard-c5b698784-55rhq 0 (0%) 0 (0%) 0 (0%) 0 (0%) 306d
kube-system monitoring-influxdb-grafana-v4-6cc44d985f-2p4gd 200m (5%) 200m (5%) 600Mi (7%) 600Mi (7%) 306d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 400m (10%) 300m (7%)
memory 870800Ki (11%) 973200Ki (12%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none> |
|
Jakub Scholz
The first node looks ok. But if you check the description, you can see that some of the nodes are marked with the taint: That is what the events are complaining about. I expect that the taint is set there for some reason - but I never used Microk8s, so not sure where to look for more info. Also, the Conditions in the descriptions are unknown. So my guess is that maybe the cluster cannot communicate with them? Maybe some Microk8s forum might give you better answers on what might be causing this. Jakub On Sun, May 8, 2022 at 1:38 AM <tfv01@...> wrote: Thank you Jakub, appreciate your response! |
|