Topics

Kafka broker pods are not coming up and unable to create statefulset


tsnagraj.08@...
 

Hi Team,

Could you help me out with Listener configurations in kafka resource configurations. I am struggling with external listener configurations as i am using metallb. Is there any configuration changes required for metallb.
kafka-persistent.yaml:

apiVersion: kafka.strimzi.io/v1beta1

kind: Kafka

metadata:

  name: my-cluster

  annotations:

    metallb.universe.tf/allow-shared-ip: smart

spec:

  loadBalancerIP: xxxxxxx

  kafka:

    version: 2.5.0

    replicas: 3

    listeners:

      plain:  {}

      tls: {}

      external:

        type: loadbalancer

        tls: false

    config:

      offsets.topic.replication.factor: 3

      transaction.state.log.replication.factor: 3

      transaction.state.log.min.isr: 2

      log.message.format.version: "2.5"

    storage:

      type: persistent-claim

      size: 10Gi

      class: sc-paas

      deleteClaim: false

    resources:

      requests:

        memory: 8Gi

        cpu: "1"

      limits:

        memory: 24Gi

        cpu: "3"

  zookeeper:

    replicas: 3

    resources:

      requests:

        memory: 8Gi

        cpu: "1"

      limits:

        memory: 24Gi

        cpu: "3"

    storage:

      type: persistent-claim

      size: 10Gi

      class: sc-paas

      deleteClaim: false

  entityOperator:

    topicOperator: {}

    userOperator: {}

Logs for operator:

2020-09-18 13:08:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:10:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:12:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:14:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:16:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:18:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:20:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:22:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:23:40 INFO  OperatorWatcher:40 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED

2020-09-18 13:23:40 INFO  AbstractOperator:173 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:23:40 INFO  OperatorWatcher:40 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:23:50 WARN  AbstractOperator:247 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:24:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:24:29 WARN  AbstractOperator:247 - Reconciliation #13(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:26:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:26:29 WARN  AbstractOperator:247 - Reconciliation #14(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:28:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:28:29 WARN  AbstractOperator:247 - Reconciliation #15(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:29:52 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:29:52 WARN  KafkaAssemblyOperator:1866 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:29:52 INFO  OperatorWatcher:40 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:29:52 ERROR AbstractOperator:175 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:29:52 WARN  AbstractOperator:330 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:29:52 INFO  AbstractOperator:173 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:30:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:30:29 WARN  AbstractOperator:247 - Reconciliation #17(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:32:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:32:29 WARN  AbstractOperator:247 - Reconciliation #18(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:33:46 INFO  OperatorWatcher:40 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:33:56 WARN  AbstractOperator:247 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:34:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:34:29 WARN  AbstractOperator:247 - Reconciliation #20(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:34:44 INFO  OperatorWatcher:40 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was DELETED

2020-09-18 13:34:53 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:34:53 WARN  KafkaAssemblyOperator:1866 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:498 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Current Kafka resource not found

2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:244 - Failed to set status for my-cluster

2020-09-18 13:34:53 ERROR AbstractOperator:175 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:34:53 WARN  AbstractOperator:330 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:34:53 INFO  AbstractOperator:179 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be deleted

2020-09-18 13:34:53 INFO  AbstractOperator:184 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Assembly my-cluster should be deleted by garbage collection

2020-09-18 13:34:53 INFO  AbstractOperator:318 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): reconciled

2020-09-18 13:35:36 INFO  OperatorWatcher:40 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED

2020-09-18 13:35:36 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:35:36 INFO  AbstractOperator:173 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:35:36 INFO  OperatorWatcher:40 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:35:46 WARN  AbstractOperator:247 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:36:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:36:29 WARN  AbstractOperator:247 - Reconciliation #24(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:38:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:38:29 WARN  AbstractOperator:247 - Reconciliation #25(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:40:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:40:29 WARN  AbstractOperator:247 - Reconciliation #26(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:41:22 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:41:22 WARN  KafkaAssemblyOperator:1866 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:41:22 ERROR AbstractOperator:175 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:41:22 INFO  OperatorWatcher:40 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:41:22 WARN  AbstractOperator:330 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:41:22 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:41:22 INFO  AbstractOperator:173 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:42:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:42:29 WARN  AbstractOperator:247 - Reconciliation #28(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:44:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:44:29 WARN  AbstractOperator:247 - Reconciliation #29(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:46:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:46:23 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:46:23 WARN  KafkaAssemblyOperator:1866 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:46:23 ERROR AbstractOperator:175 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:46:23 WARN  AbstractOperator:330 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:46:23 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:46:23 INFO  AbstractOperator:173 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:48:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:48:29 WARN  AbstractOperator:247 - Reconciliation #31(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:50:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:50:29 WARN  AbstractOperator:247 - Reconciliation #32(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:51:25 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:51:25 WARN  KafkaAssemblyOperator:1866 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:51:25 ERROR AbstractOperator:175 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:51:25 WARN  AbstractOperator:330 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:52:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:52:19 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:52:19 INFO  AbstractOperator:173 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:54:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:54:29 WARN  AbstractOperator:247 - Reconciliation #34(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:56:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:56:29 WARN  AbstractOperator:247 - Reconciliation #35(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:57:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:57:20 WARN  KafkaAssemblyOperator:1866 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:57:20 ERROR AbstractOperator:175 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:57:20 WARN  AbstractOperator:330 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:58:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:58:19 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:58:19 INFO  AbstractOperator:173 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 14:00:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 14:00:29 WARN  AbstractOperator:247 - Reconciliation #37(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 14:02:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 14:02:29 WARN  AbstractOperator:247 - Reconciliation #38(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 14:03:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 14:03:20 WARN  KafkaAssemblyOperator:1866 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 14:03:20 ERROR AbstractOperator:175 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 14:03:20 WARN  AbstractOperator:330 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.





Jakub Scholz
 

I guess it might depend on your MetalLB configuration. I use it on my local Kube cluster without any special settings. But in your case, it looks like the LoadBalancers are not being provisioned. You can use `kubectl get service` to see if at least some were provisioned or not. In case you want the LoadBalancers with some specific IP address, you can also use this in the KAfka CR:

  external:
    type: loadbalancer
    tls: false
    overrides:
      bootstrap:
        loadBalancerIP: 172.29.3.10
      brokers:
      - broker: 0
        loadBalancerIP: 172.29.3.1
      - broker: 1
        loadBalancerIP: 172.29.3.2
      - broker: 2
        loadBalancerIP: 172.29.3.3


This should make sure that the loadbalancers get these assigned IP addresses if possible. But as I said - not needed on my MetalLB installation. But you can give it a try.

If you want to use an annotation on the service (such as metallb.universe.tf/allow-shared-ip), you can use the perPodService and exteralBootstrapService options here: https://strimzi.io/docs/operators/latest/full/using.html#type-KafkaClusterTemplate-reference ... but I don't think Kafka will work with a shared IP address.

Thanks & Regards
Jakub

On Fri, Sep 18, 2020 at 5:09 PM <tsnagraj.08@...> wrote:

Hi Team,

Could you help me out with Listener configurations in kafka resource configurations. I am struggling with external listener configurations as i am using metallb. Is there any configuration changes required for metallb.
kafka-persistent.yaml:

apiVersion: kafka.strimzi.io/v1beta1

kind: Kafka

metadata:

  name: my-cluster

  annotations:

    metallb.universe.tf/allow-shared-ip: smart

spec:

  loadBalancerIP: xxxxxxx

  kafka:

    version: 2.5.0

    replicas: 3

    listeners:

      plain:  {}

      tls: {}

      external:

        type: loadbalancer

        tls: false

    config:

      offsets.topic.replication.factor: 3

      transaction.state.log.replication.factor: 3

      transaction.state.log.min.isr: 2

      log.message.format.version: "2.5"

    storage:

      type: persistent-claim

      size: 10Gi

      class: sc-paas

      deleteClaim: false

    resources:

      requests:

        memory: 8Gi

        cpu: "1"

      limits:

        memory: 24Gi

        cpu: "3"

  zookeeper:

    replicas: 3

    resources:

      requests:

        memory: 8Gi

        cpu: "1"

      limits:

        memory: 24Gi

        cpu: "3"

    storage:

      type: persistent-claim

      size: 10Gi

      class: sc-paas

      deleteClaim: false

  entityOperator:

    topicOperator: {}

    userOperator: {}

Logs for operator:

2020-09-18 13:08:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:10:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:12:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:14:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:16:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:18:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:20:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:22:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:23:40 INFO  OperatorWatcher:40 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED

2020-09-18 13:23:40 INFO  AbstractOperator:173 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:23:40 INFO  OperatorWatcher:40 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:23:50 WARN  AbstractOperator:247 - Reconciliation #12(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:24:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:24:29 WARN  AbstractOperator:247 - Reconciliation #13(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:26:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:26:29 WARN  AbstractOperator:247 - Reconciliation #14(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:28:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:28:29 WARN  AbstractOperator:247 - Reconciliation #15(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:29:52 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:29:52 WARN  KafkaAssemblyOperator:1866 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:29:52 INFO  OperatorWatcher:40 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:29:52 ERROR AbstractOperator:175 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:29:52 WARN  AbstractOperator:330 - Reconciliation #11(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:29:52 INFO  AbstractOperator:173 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:30:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:30:29 WARN  AbstractOperator:247 - Reconciliation #17(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:32:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:32:29 WARN  AbstractOperator:247 - Reconciliation #18(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:33:46 INFO  OperatorWatcher:40 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:33:56 WARN  AbstractOperator:247 - Reconciliation #19(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:34:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:34:29 WARN  AbstractOperator:247 - Reconciliation #20(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:34:44 INFO  OperatorWatcher:40 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was DELETED

2020-09-18 13:34:53 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:34:53 WARN  KafkaAssemblyOperator:1866 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:498 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Current Kafka resource not found

2020-09-18 13:34:53 ERROR KafkaAssemblyOperator:244 - Failed to set status for my-cluster

2020-09-18 13:34:53 ERROR AbstractOperator:175 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:34:53 WARN  AbstractOperator:330 - Reconciliation #16(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:34:53 INFO  AbstractOperator:179 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be deleted

2020-09-18 13:34:53 INFO  AbstractOperator:184 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): Assembly my-cluster should be deleted by garbage collection

2020-09-18 13:34:53 INFO  AbstractOperator:318 - Reconciliation #21(watch) Kafka(kafka-op-test/my-cluster): reconciled

2020-09-18 13:35:36 INFO  OperatorWatcher:40 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was ADDED

2020-09-18 13:35:36 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:35:36 INFO  AbstractOperator:173 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:35:36 INFO  OperatorWatcher:40 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:35:46 WARN  AbstractOperator:247 - Reconciliation #23(watch) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:36:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:36:29 WARN  AbstractOperator:247 - Reconciliation #24(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:38:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:38:29 WARN  AbstractOperator:247 - Reconciliation #25(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:40:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:40:29 WARN  AbstractOperator:247 - Reconciliation #26(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:41:22 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:41:22 WARN  KafkaAssemblyOperator:1866 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:41:22 ERROR AbstractOperator:175 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:41:22 INFO  OperatorWatcher:40 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster in namespace kafka-op-test was MODIFIED

2020-09-18 13:41:22 WARN  AbstractOperator:330 - Reconciliation #22(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:41:22 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:41:22 INFO  AbstractOperator:173 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:42:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:42:29 WARN  AbstractOperator:247 - Reconciliation #28(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:44:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:44:29 WARN  AbstractOperator:247 - Reconciliation #29(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:46:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:46:23 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:46:23 WARN  KafkaAssemblyOperator:1866 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:46:23 ERROR AbstractOperator:175 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:46:23 WARN  AbstractOperator:330 - Reconciliation #27(watch) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:46:23 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:46:23 INFO  AbstractOperator:173 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:48:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:48:29 WARN  AbstractOperator:247 - Reconciliation #31(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:50:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:50:29 WARN  AbstractOperator:247 - Reconciliation #32(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:51:25 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:51:25 WARN  KafkaAssemblyOperator:1866 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:51:25 ERROR AbstractOperator:175 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:51:25 WARN  AbstractOperator:330 - Reconciliation #30(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:52:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:52:19 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:52:19 INFO  AbstractOperator:173 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 13:54:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:54:29 WARN  AbstractOperator:247 - Reconciliation #34(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:56:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:56:29 WARN  AbstractOperator:247 - Reconciliation #35(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 13:57:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 13:57:20 WARN  KafkaAssemblyOperator:1866 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:57:20 ERROR AbstractOperator:175 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:57:20 WARN  AbstractOperator:330 - Reconciliation #33(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 13:58:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 13:58:19 WARN  AbstractOperator:106 - Kafka resource my-cluster in namespace kafka-op-test: Contains object at path spec with an unknown property: loadBalancerIP

2020-09-18 13:58:19 INFO  AbstractOperator:173 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Kafka my-cluster should be created or updated

2020-09-18 14:00:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 14:00:29 WARN  AbstractOperator:247 - Reconciliation #37(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 14:02:19 INFO  ClusterOperator:125 - Triggering periodic reconciliation for namespace kafka-op-test...

2020-09-18 14:02:29 WARN  AbstractOperator:247 - Reconciliation #38(timer) Kafka(kafka-op-test/my-cluster): Failed to acquire lock lock::kafka-op-test::Kafka::my-cluster within 10000ms.

2020-09-18 14:03:20 ERROR Util:120 - Exceeded timeout of 300000ms while waiting for Service resource my-cluster-kafka-external-bootstrap in namespace kafka-op-test to be addressable

2020-09-18 14:03:20 WARN  KafkaAssemblyOperator:1866 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 14:03:20 ERROR AbstractOperator:175 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): createOrUpdate failed

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.

2020-09-18 14:03:20 WARN  AbstractOperator:330 - Reconciliation #36(timer) Kafka(kafka-op-test/my-cluster): Failed to reconcile

io.vertx.core.impl.NoStackTraceThrowable: No loadbalancer address found in the Status section of Service my-cluster-kafka-external-bootstrap resource. Loadbalancer was probably not provisioned.