Kafka broker pods are not coming up and unable to create statefulset
Nag Raj
Hi Team, apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: my-cluster annotations: metallb.universe.tf/allow-shared-ip: smart spec: loadBalancerIP: xxxxxxx kafka: version: 2.5.0 replicas: 3 listeners: plain: {} tls: {} external: type: loadbalancer tls: false config: offsets.topic.replication.factor: 3 transaction.state.log.replication.factor: 3 transaction.state.log.min.isr: 2 log.message.format.version: "2.5" storage: type: persistent-claim size: 10Gi class: sc-paas deleteClaim: false resources: requests: memory: 8Gi cpu: "1" limits: memory: 24Gi cpu: "3" zookeeper: replicas: 3 resources: requests: memory: 8Gi cpu: "1" limits: memory: 24Gi cpu: "3" storage: type: persistent-claim size: 10Gi class: sc-paas deleteClaim: false entityOperator: topicOperator: {} userOperator: {} 2020-09-18 13:08:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:10:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:12:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:14:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:16:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:18:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:20:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:22:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:23:40 INFO OperatorWatcher:40 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED 2020-09-18 13:23:40 INFO AbstractOperator:173 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated 2020-09-18 13:23:40 INFO OperatorWatcher:40 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED 2020-09-18 13:23:50 WARN AbstractOperator:247 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:24:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:24:29 WARN AbstractOperator:247 - Reconciliation #13(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:26:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:26:29 WARN AbstractOperator:247 - Reconciliation #14(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:28:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:28:29 WARN AbstractOperator:247 - Reconciliation #15(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:29:52 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable 2020-09-18 13:29:52 WARN KafkaAssemblyOperator:1866 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:29:52 INFO OperatorWatcher:40 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED 2020-09-18 13:29:52 ERROR AbstractOperator:175 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:29:52 WARN AbstractOperator:330 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:29:52 INFO AbstractOperator:173 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated 2020-09-18 13:30:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:30:29 WARN AbstractOperator:247 - Reconciliation #17(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:32:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:32:29 WARN AbstractOperator:247 - Reconciliation #18(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:33:46 INFO OperatorWatcher:40 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED 2020-09-18 13:33:56 WARN AbstractOperator:247 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:34:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:34:29 WARN AbstractOperator:247 - Reconciliation #20(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:34:44 INFO OperatorWatcher:40 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was DELETED 2020-09-18 13:34:53 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable 2020-09-18 13:34:53 WARN KafkaAssemblyOperator:1866 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:498 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Current Kafka resource not found 2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:244 - Failed to set status for my-cluster 2020-09-18 13:34:53 ERROR AbstractOperator:175 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:34:53 WARN AbstractOperator:330 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:34:53 INFO AbstractOperator:179 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be deleted 2020-09-18 13:34:53 INFO AbstractOperator:184 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Assembly my-cluster should be deleted by garbage collection 2020-09-18 13:34:53 INFO AbstractOperator:318 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): reconciled 2020-09-18 13:35:36 INFO OperatorWatcher:40 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED 2020-09-18 13:35:36 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP 2020-09-18 13:35:36 INFO AbstractOperator:173 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated 2020-09-18 13:35:36 INFO OperatorWatcher:40 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED 2020-09-18 13:35:46 WARN AbstractOperator:247 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:36:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:36:29 WARN AbstractOperator:247 - Reconciliation #24(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:38:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:38:29 WARN AbstractOperator:247 - Reconciliation #25(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:40:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:40:29 WARN AbstractOperator:247 - Reconciliation #26(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:41:22 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable 2020-09-18 13:41:22 WARN KafkaAssemblyOperator:1866 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:41:22 ERROR AbstractOperator:175 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:41:22 INFO OperatorWatcher:40 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED 2020-09-18 13:41:22 WARN AbstractOperator:330 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:41:22 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP 2020-09-18 13:41:22 INFO AbstractOperator:173 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated 2020-09-18 13:42:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:42:29 WARN AbstractOperator:247 - Reconciliation #28(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:44:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:44:29 WARN AbstractOperator:247 - Reconciliation #29(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:46:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:46:23 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable 2020-09-18 13:46:23 WARN KafkaAssemblyOperator:1866 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:46:23 ERROR AbstractOperator:175 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:46:23 WARN AbstractOperator:330 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:46:23 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP 2020-09-18 13:46:23 INFO AbstractOperator:173 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated 2020-09-18 13:48:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:48:29 WARN AbstractOperator:247 - Reconciliation #31(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:50:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:50:29 WARN AbstractOperator:247 - Reconciliation #32(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:51:25 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable 2020-09-18 13:51:25 WARN KafkaAssemblyOperator:1866 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:51:25 ERROR AbstractOperator:175 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:51:25 WARN AbstractOperator:330 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:52:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:52:19 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP 2020-09-18 13:52:19 INFO AbstractOperator:173 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated 2020-09-18 13:54:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:54:29 WARN AbstractOperator:247 - Reconciliation #34(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:56:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:56:29 WARN AbstractOperator:247 - Reconciliation #35(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 13:57:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable 2020-09-18 13:57:20 WARN KafkaAssemblyOperator:1866 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:57:20 ERROR AbstractOperator:175 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:57:20 WARN AbstractOperator:330 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 13:58:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 13:58:19 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP 2020-09-18 13:58:19 INFO AbstractOperator:173 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated 2020-09-18 14:00:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 14:00:29 WARN AbstractOperator:247 - Reconciliation #37(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 14:02:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test... 2020-09-18 14:02:29 WARN AbstractOperator:247 - Reconciliation #38(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms. 2020-09-18 14:03:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable 2020-09-18 14:03:20 WARN KafkaAssemblyOperator:1866 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 14:03:20 ERROR AbstractOperator:175 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. 2020-09-18 14:03:20 WARN AbstractOperator:330 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned. |
|
Jakub Scholz
I guess it might depend on your MetalLB configuration. I use it on my local Kube cluster without any special settings. But in your case, it looks like the LoadBalancers are not being provisioned. You can use `kubectl get service` to see if at least some were provisioned or not. In case you want the LoadBalancers with some specific IP address, you can also use this in the KAfka CR: external: type: loadbalancer tls: false overrides: bootstrap: loadBalancerIP: 172.29.3.10 brokers: - broker: 0 loadBalancerIP: 172.29.3.1 - broker: 1 loadBalancerIP: 172.29.3.2 - broker: 2 loadBalancerIP: 172.29.3.3 This should make sure that the loadbalancers get these assigned IP addresses if possible. But as I said - not needed on my MetalLB installation. But you can give it a try. If you want to use an annotation on the service (such as metallb.universe.tf/allow-shared-ip), you can use the perPodService and exteralBootstrapService options here: https://strimzi.io/docs/operators/latest/full/using.html#type-KafkaClusterTemplate-reference ... but I don't think Kafka will work with a shared IP address. Thanks & Regards Jakub On Fri, Sep 18, 2020 at 5:09 PM <tsnagraj.08@...> wrote:
|
|