Kafka broker pods are not coming up and unable to create statefulset
Hi Team,
Could you help me out with Listener configurations in kafka resource configurations. I am struggling with external listener configurations as i am using metallb. Is there any configuration changes required for metallb.
kafka-persistent.yaml:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
annotations:
metallb.universe.tf/allow-shared-ip: smart
spec:
loadBalancerIP: xxxxxxx
kafka:
version: 2.5.0
replicas: 3
listeners:
plain: {}
tls: {}
external:
type: loadbalancer
tls: false
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.5"
storage:
type: persistent-claim
size: 10Gi
class: sc-paas
deleteClaim: false
resources:
requests:
memory: 8Gi
cpu: "1"
limits:
memory: 24Gi
cpu: "3"
zookeeper:
replicas: 3
resources:
requests:
memory: 8Gi
cpu: "1"
limits:
memory: 24Gi
cpu: "3"
storage:
type: persistent-claim
size: 10Gi
class: sc-paas
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
Logs for operator:
2020-09-18 13:08:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:10:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:12:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:14:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:16:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:18:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:20:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:22:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:23:40 INFO OperatorWatcher:40 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED
2020-09-18 13:23:40 INFO AbstractOperator:173 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:23:40 INFO OperatorWatcher:40 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:23:50 WARN AbstractOperator:247 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:24:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:24:29 WARN AbstractOperator:247 - Reconciliation #13(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:26:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:26:29 WARN AbstractOperator:247 - Reconciliation #14(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:28:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:28:29 WARN AbstractOperator:247 - Reconciliation #15(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:29:52 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:29:52 WARN KafkaAssemblyOperator:1866 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:29:52 INFO OperatorWatcher:40 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:29:52 ERROR AbstractOperator:175 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:29:52 WARN AbstractOperator:330 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:29:52 INFO AbstractOperator:173 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:30:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:30:29 WARN AbstractOperator:247 - Reconciliation #17(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:32:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:32:29 WARN AbstractOperator:247 - Reconciliation #18(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:33:46 INFO OperatorWatcher:40 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:33:56 WARN AbstractOperator:247 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:34:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:34:29 WARN AbstractOperator:247 - Reconciliation #20(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:34:44 INFO OperatorWatcher:40 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was DELETED
2020-09-18 13:34:53 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:34:53 WARN KafkaAssemblyOperator:1866 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:498 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Current Kafka resource not found
2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:244 - Failed to set status for my-cluster
2020-09-18 13:34:53 ERROR AbstractOperator:175 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:34:53 WARN AbstractOperator:330 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:34:53 INFO AbstractOperator:179 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be deleted
2020-09-18 13:34:53 INFO AbstractOperator:184 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Assembly my-cluster should be deleted by garbage collection
2020-09-18 13:34:53 INFO AbstractOperator:318 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): reconciled
2020-09-18 13:35:36 INFO OperatorWatcher:40 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED
2020-09-18 13:35:36 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:35:36 INFO AbstractOperator:173 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:35:36 INFO OperatorWatcher:40 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:35:46 WARN AbstractOperator:247 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:36:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:36:29 WARN AbstractOperator:247 - Reconciliation #24(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:38:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:38:29 WARN AbstractOperator:247 - Reconciliation #25(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:40:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:40:29 WARN AbstractOperator:247 - Reconciliation #26(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:41:22 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:41:22 WARN KafkaAssemblyOperator:1866 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:41:22 ERROR AbstractOperator:175 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:41:22 INFO OperatorWatcher:40 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:41:22 WARN AbstractOperator:330 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:41:22 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:41:22 INFO AbstractOperator:173 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:42:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:42:29 WARN AbstractOperator:247 - Reconciliation #28(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:44:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:44:29 WARN AbstractOperator:247 - Reconciliation #29(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:46:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:46:23 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:46:23 WARN KafkaAssemblyOperator:1866 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:46:23 ERROR AbstractOperator:175 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:46:23 WARN AbstractOperator:330 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:46:23 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:46:23 INFO AbstractOperator:173 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:48:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:48:29 WARN AbstractOperator:247 - Reconciliation #31(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:50:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:50:29 WARN AbstractOperator:247 - Reconciliation #32(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:51:25 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:51:25 WARN KafkaAssemblyOperator:1866 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:51:25 ERROR AbstractOperator:175 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:51:25 WARN AbstractOperator:330 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:52:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:52:19 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:52:19 INFO AbstractOperator:173 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:54:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:54:29 WARN AbstractOperator:247 - Reconciliation #34(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:56:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:56:29 WARN AbstractOperator:247 - Reconciliation #35(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:57:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:57:20 WARN KafkaAssemblyOperator:1866 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:57:20 ERROR AbstractOperator:175 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:57:20 WARN AbstractOperator:330 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:58:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:58:19 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:58:19 INFO AbstractOperator:173 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 14:00:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 14:00:29 WARN AbstractOperator:247 - Reconciliation #37(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 14:02:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 14:02:29 WARN AbstractOperator:247 - Reconciliation #38(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 14:03:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 14:03:20 WARN KafkaAssemblyOperator:1866 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 14:03:20 ERROR AbstractOperator:175 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 14:03:20 WARN AbstractOperator:330 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
type: loadbalancer
tls: false
overrides:
bootstrap:
loadBalancerIP: 172.29.3.10
brokers:
- broker: 0
loadBalancerIP: 172.29.3.1
- broker: 1
loadBalancerIP: 172.29.3.2
- broker: 2
loadBalancerIP: 172.29.3.3
Hi Team,
Could you help me out with Listener configurations in kafka resource configurations. I am struggling with external listener configurations as i am using metallb. Is there any configuration changes required for metallb.
kafka-persistent.yaml:apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
annotations:
metallb.universe.tf/allow-shared-ip: smart
spec:
loadBalancerIP: xxxxxxx
kafka:
version: 2.5.0
replicas: 3
listeners:
plain: {}
tls: {}
external:
type: loadbalancer
tls: false
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.5"
storage:
type: persistent-claim
size: 10Gi
class: sc-paas
deleteClaim: false
resources:
requests:
memory: 8Gi
cpu: "1"
limits:
memory: 24Gi
cpu: "3"
zookeeper:
replicas: 3
resources:
requests:
memory: 8Gi
cpu: "1"
limits:
memory: 24Gi
cpu: "3"
storage:
type: persistent-claim
size: 10Gi
class: sc-paas
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
Logs for operator:2020-09-18 13:08:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:10:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:12:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:14:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:16:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:18:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:20:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:22:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:23:40 INFO OperatorWatcher:40 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED
2020-09-18 13:23:40 INFO AbstractOperator:173 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:23:40 INFO OperatorWatcher:40 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:23:50 WARN AbstractOperator:247 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:24:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:24:29 WARN AbstractOperator:247 - Reconciliation #13(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:26:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:26:29 WARN AbstractOperator:247 - Reconciliation #14(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:28:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:28:29 WARN AbstractOperator:247 - Reconciliation #15(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:29:52 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:29:52 WARN KafkaAssemblyOperator:1866 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:29:52 INFO OperatorWatcher:40 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:29:52 ERROR AbstractOperator:175 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:29:52 WARN AbstractOperator:330 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:29:52 INFO AbstractOperator:173 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:30:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:30:29 WARN AbstractOperator:247 - Reconciliation #17(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:32:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:32:29 WARN AbstractOperator:247 - Reconciliation #18(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:33:46 INFO OperatorWatcher:40 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:33:56 WARN AbstractOperator:247 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:34:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:34:29 WARN AbstractOperator:247 - Reconciliation #20(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:34:44 INFO OperatorWatcher:40 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was DELETED
2020-09-18 13:34:53 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:34:53 WARN KafkaAssemblyOperator:1866 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:498 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Current Kafka resource not found
2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:244 - Failed to set status for my-cluster
2020-09-18 13:34:53 ERROR AbstractOperator:175 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:34:53 WARN AbstractOperator:330 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:34:53 INFO AbstractOperator:179 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be deleted
2020-09-18 13:34:53 INFO AbstractOperator:184 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Assembly my-cluster should be deleted by garbage collection
2020-09-18 13:34:53 INFO AbstractOperator:318 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): reconciled
2020-09-18 13:35:36 INFO OperatorWatcher:40 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED
2020-09-18 13:35:36 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:35:36 INFO AbstractOperator:173 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:35:36 INFO OperatorWatcher:40 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:35:46 WARN AbstractOperator:247 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:36:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:36:29 WARN AbstractOperator:247 - Reconciliation #24(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:38:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:38:29 WARN AbstractOperator:247 - Reconciliation #25(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:40:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:40:29 WARN AbstractOperator:247 - Reconciliation #26(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:41:22 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:41:22 WARN KafkaAssemblyOperator:1866 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:41:22 ERROR AbstractOperator:175 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:41:22 INFO OperatorWatcher:40 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED
2020-09-18 13:41:22 WARN AbstractOperator:330 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:41:22 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:41:22 INFO AbstractOperator:173 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:42:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:42:29 WARN AbstractOperator:247 - Reconciliation #28(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:44:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:44:29 WARN AbstractOperator:247 - Reconciliation #29(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:46:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:46:23 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:46:23 WARN KafkaAssemblyOperator:1866 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:46:23 ERROR AbstractOperator:175 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:46:23 WARN AbstractOperator:330 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:46:23 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:46:23 INFO AbstractOperator:173 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:48:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:48:29 WARN AbstractOperator:247 - Reconciliation #31(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:50:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:50:29 WARN AbstractOperator:247 - Reconciliation #32(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:51:25 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:51:25 WARN KafkaAssemblyOperator:1866 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:51:25 ERROR AbstractOperator:175 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:51:25 WARN AbstractOperator:330 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:52:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:52:19 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:52:19 INFO AbstractOperator:173 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 13:54:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:54:29 WARN AbstractOperator:247 - Reconciliation #34(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:56:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:56:29 WARN AbstractOperator:247 - Reconciliation #35(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 13:57:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 13:57:20 WARN KafkaAssemblyOperator:1866 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:57:20 ERROR AbstractOperator:175 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:57:20 WARN AbstractOperator:330 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 13:58:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 13:58:19 WARN AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP
2020-09-18 13:58:19 INFO AbstractOperator:173 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated
2020-09-18 14:00:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 14:00:29 WARN AbstractOperator:247 - Reconciliation #37(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 14:02:19 INFO ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...
2020-09-18 14:02:29 WARN AbstractOperator:247 - Reconciliation #38(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.
2020-09-18 14:03:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable
2020-09-18 14:03:20 WARN KafkaAssemblyOperator:1866 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 14:03:20 ERROR AbstractOperator:175 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.
2020-09-18 14:03:20 WARN AbstractOperator:330 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile
io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.