Date   

adding annotation to service account

amit.cahanovich@...
 

Hi, 
I work with strimzi (as kafka connect) on eks. 
I would like to add  to the service account aws role annotation to s3 (something like: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxx:role/s3-read-role).
Is there a trivial way to do it?
Thanks,
Amit


[ANNOUNCE] [RELEASE] Strimzi Kafka Operators 0.28.0

Jakub Scholz
 

Strimzi Kafka Operators 0.28.0 has been released. The main changes in this release include:
* Add support for Kafka 3.1.0; remove Kafka 2.8.0 and 2.8.1
* Add support for `StrimziPodSet` resources (disabled by default through the `UseStrimziPodSets` feature gate)
* Support custom authentication mechanisms in Kafka listeners
* Intra-broker disk balancing using Cruise Control
* Added the option `createBootstrapService` in the Kafka Spec to disable the creation of the bootstrap service for the Load Balancer Type Listener. It will save the cost of one load balancer resource, specially in the public cloud.
* Add support for disabling the FIPS mode in OpenJDK
* Fix renewing your own CA certificates

For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.28.0

We have also created a video about the changes in this release which you can watch on our YouTube channel: https://youtu.be/PZKbrDUU1zo

Important: This release supports only the API version v1beta2 and CRD version apiextensions.k8s.io/v1. If upgrading from Strimzi 0.22, migration to v1beta2 needs to be completed for all Strimzi CRDs and CRs before the upgrade to 0.28 is done! If upgrading from Strimzi version earlier than 0.22, you need to first install the CRDs from Strimzi 0.22 and complete the migration to v1beta2 for all Strimzi CRDs and CRs before the upgrade to 0.28 is done! For more details about the CRD upgrades, see the documentation.

Thanks to everyone who contributed to these releases!

Thanks & Regards
Strimzi team


RC1 of Strimzi Kafka Operators 0.28.0 is available for testing

Jakub Scholz
 

Release candidate 1 of Strimzi Kafka Operators 0.28.0 is now available for testing. The main changes in this release include:
* Add support for Kafka 3.1.0; remove Kafka 2.8.0 and 2.8.1
* Add support for `StrimziPodSet` resources (disabled by default through the `UseStrimziPodSets` feature gate)
* Support custom authentication mechanisms in Kafka listeners
* Intra-broker disk balancing using Cruise Control
* Added the option `createBootstrapService` in the Kafka Spec to disable the creation of the bootstrap service for the Load Balancer Type Listener. It will save the cost of one load balancer resource, specially in the public cloud.
* Add support for disabling the FIPS mode in OpenJDK
* Fix renewing your own CA certificates


Any feedback can be provided on the Strimzi mailing list, on the #strimzi Slack channel on CNCF Slack or as a GitHub issue.

Thanks & Regards
Strimzi team


[ANNOUNCE] [RELEASE] Strimzi Kafka Bridge 0.21.4

Jakub Scholz
 

New version 0.21.4 of Strimzi Kafka Bridge has been released. The main changes in this release are:
* Dependency updates (Configuration providers, Vert.x, Netty, Oauth client and more)
* Add support for disabling the FIPS mode in OpenJDK
* Add transactions `isolation.level` configuration parameter on consumer creation
* Support for s390x platform


Thanks to everyone who contributed to any of these releases!

Regards
Strimzi team


Re: Broker Hostname mismatch error seen with aiokafka APIs

Jakub Scholz
 

You are using the external loadbalancer listener with internal service name. That is causing the problem. You should either switch to one of the internal interfaces - e.g. the on on port 9093. Or you should use the proper loadbalancer bootstrap address which you can find in the status section of the Kafka custom resource (`kubectl get kafka -o yaml` should show it for you). In general, if your app runs inside the same Kubernetes, using the internal listener on port 9093 would be the right way to go => it should be cheaper and more performant than going through the loadbalancer. That should be used by apps outside your Kube cluster.

Jakub

On Fri, Feb 4, 2022 at 10:49 AM <udaykumartj@...> wrote:
Hi all,
I am getting the below error for the code (which uses aiokafka APIs) that i have pasted below (also provided kafka resource yaml snippet). Instead of kafka broker hostname, if i give IP address, it works. Not sure why kafka broker (dns) hostname is not working. Please help.

Unable connect to "strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094": [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local'. (_ssl.c:1129)

Note: If i use confluent_kafka APIs (Consumer, Producer), i dont see this issue.

Code snippet:
 context = create_ssl_context(
      cafile='/etc/vcerts/cluster/ca.crt',
      certfile='/etc/vcerts/client/user.crt',
      keyfile='/etc/vcerts/client/user.key',
  )

consumer = AIOKafkaConsumer(
      my_topic, bootstrap_servers='strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094',
      auto_offset_reset='latest',
      group_id=group_id,
      security_protocol="SSL",
      ssl_context=context)

  await consumer.start() <=== This line throws the above error.

Kafka spec:
kind: Kafka
metadata:
  name: strimzi-kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: tls
    template:
      clusterCaCert:
        metadata:
....
 


Broker Hostname mismatch error seen with aiokafka APIs

udaykumartj@...
 

Hi all,
I am getting the below error for the code (which uses aiokafka APIs) that i have pasted below (also provided kafka resource yaml snippet). Instead of kafka broker hostname, if i give IP address, it works. Not sure why kafka broker (dns) hostname is not working. Please help.

Unable connect to "strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094": [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local'. (_ssl.c:1129)

Note: If i use confluent_kafka APIs (Consumer, Producer), i dont see this issue.

Code snippet:
 context = create_ssl_context(
      cafile='/etc/vcerts/cluster/ca.crt',
      certfile='/etc/vcerts/client/user.crt',
      keyfile='/etc/vcerts/client/user.key',
  )

consumer = AIOKafkaConsumer(
      my_topic, bootstrap_servers='strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094',
      auto_offset_reset='latest',
      group_id=group_id,
      security_protocol="SSL",
      ssl_context=context)

  await consumer.start() <=== This line throws the above error.

Kafka spec:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: strimzi-kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: tls
    template:
      clusterCaCert:
        metadata:
....
 


Broker Hostname mismatch error seen with aiokafka APIs

udaykumartj@...
 

Hi all,
I am getting the below error for the code (which uses aiokafka APIs) that i have pasted below (also provided kafka resource yaml snippet). Instead of kafka broker hostname, if i give IP address, it works. Not sure why kafka broker (dns) hostname is not working. Please help.

Unable connect to "strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094": [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local'. (_ssl.c:1129)

Note: If i use confluent_kafka APIs (Consumer, Producer), i dont see this issue.

Code snippet:
 context = create_ssl_context(
      cafile='/etc/vcerts/cluster/ca.crt',
      certfile='/etc/vcerts/client/user.crt',
      keyfile='/etc/vcerts/client/user.key',
  )

consumer = AIOKafkaConsumer(
      my_topic, bootstrap_servers='strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094',
      auto_offset_reset='latest',
      group_id=group_id,
      security_protocol="SSL",
      ssl_context=context)

  await consumer.start() <=== This line throws the above error.

Kafka spec:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: strimzi-kafka
spec:
  kafka:
    version: 3.0.0
    replicas: 3
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
        authentication:
          type: tls
      - name: external
        port: 9094
        type: loadbalancer
        tls: true
        authentication:
          type: tls
    template:
      clusterCaCert:
        metadata:
....
 


[ANNOUNCE] [RELEASE] Mirror Maker 2 Extensions 1.2.0, EnvVar Configuration Provider 1.0.0 and Kubernetes Configuration Provider 1.0.0

Jakub Scholz
 

New versions of Mirror Maker 2 Extensions, EnvVar Configuration Provider and Kubernetes Configuration Provider have been released.

The biggest change involves the Mirror Maker 2 Extensions. The Strimzi Identity Replication Policy is now deprecated and all users of Kafka 3.0.0 and higher are recommended to update their configurations and use the Identity Replication Policy which is directly part of the Apache Kafka project. The 1.2.0 release of the Mirror Maker 2 Extensions is used for backwards compatibility and provides the same behaviour as the Kafka policy but using the original Strimzi class name. The Config Provider releases contain minor improvements and dependency updates. For more details, visit the release pages of each project:

* Mirror Maker 2 Extensions: https://github.com/strimzi/mirror-maker-2-extensions/releases/tag/1.2.0
* Kafka EnvVar Configuration Providerhttps://github.com/strimzi/kafka-env-var-config-provider/releases/tag/1.0.0
* Kafka Kubernetes Configuration Providerhttps://github.com/strimzi/kafka-kubernetes-config-provider/releases/tag/1.0.0

Thanks to everyone who contributed to any of these releases!

Regards
Strimzi team


Re: Adding annotations and limits of kafka connect created pods

amit.cahanovich@...
 

for reference,
The following is kafkaconnect that is integrated with datadog (named my-connect-cluster)

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  namespace: kafka
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  image: amitca71/strimzi-connect:0.27.1-kafka-2.8.1
  version: 2.8.1
  replicas: 3
  bootstrapServers: "${bootstrat_servers}"
  template:
    deployment:
      metadata:
        annotations:
          reloader.stakater.com/auto: "true"
    pod:
      metadata:
        annotations:
          ad.datadoghq.com/my-connect-cluster-connect.check_names: '["openmetrics"]'
          ad.datadoghq.com/my-connect-cluster-connect.init_configs: '[{}]'
          ad.datadoghq.com/my-connect-cluster-connect.instances: '[{"prometheus_url": "http://%%host%%:%%port%%/metrics","namespace": "strimzi_kafka_connect","metrics": ["kafka_connect_connector_task_batch_size_avg", "kafka_admin_client_node_incoming_byte_total", "kafka_connect_worker_connector_paused_task_count","kafka_connect_version_info"]}]'    
  


RC1 of Mirror Maker 2 Extensions 1.2.0, EnvVar Configuration Provider 1.0.0 and Kubernetes Configuration Provider 0.1.1

Jakub Scholz
 

As we are preparing for the next release of the Strimzi operators, we are doing different minor and patch releases of some of our smaller projects used as dependencies. Today, we released the Release Candidate 1 of several sub-projects:
* Mirror Maker 2 Extensions: https://github.com/strimzi/mirror-maker-2-extensions/releases/tag/1.2.0-rc1
* Kafka EnvVar Configuration Provider: https://github.com/strimzi/kafka-env-var-config-provider/releases/tag/1.0.0-rc1
* Kafka Kubernetes Configuration Provider: https://github.com/strimzi/kafka-kubernetes-config-provider/releases/tag/1.0.0-rc1

They contain some smaller bug fixes and improvements as well as updated dependencies. If you are interested, give them a try! Any feedback can be provided on the Strimzi mailing list, on the #strimzi Slack channel on CNCF Slack or as a GitHub issue.

Thanks & Regards
Strimzi team


Re: Adding annotations and limits of kafka connect created pods

Jakub Scholz
 

The YAML you shared has no indentation. If that is how it really looks, it is indeed invalid. If the indentation was list during copy pasting, nobody can say what the original indentation was and what would be the error.

Thanks & Regards
Jakub

On Mon, Jan 31, 2022 at 4:08 PM <amit.cahanovich@...> wrote:
Hi,
i have the following configuration:
kind: KafkaConnect
metadata:
name: my-connect-cluster
namespace: kafka
annotations:
spec:
template:
pod:
metadata:
annotations:
ad.datadoghq.com/kafka-connect-container-name.logs: '[{"type":"file", "source":"java","sourcecategory":"sourcecode", "service":"kafka-connect"}]'
ad.datadoghq.com/container.instances: '[{"prometheus_url": "http://%%host%%:9404/metrics","namespace": "kafka","metrics": ["kafka_connect_connector_task_batch_size_avg"],"type_overrides": {"kafka_connect_incoming_byte_total": "gauge"}}]'
image:


i keep on getting: error: error validating "kafka-connect-prometeus.yaml": error validating data: ValidationError(KafkaConnect): unknown field "template" in io.strimzi.kafka.v1beta2.KafkaConnect; if you choose to ignore these errors, turn validation off with --validate=false

any advise on it (when removing it. all work smoothly...)
Thanks,
Amit


Re: Adding annotations and limits of kafka connect created pods

amit.cahanovich@...
 

Hi,
i have the following configuration:
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster
namespace: kafka
annotations:
strimzi.io/use-connector-resources: "true"
spec:
template:
pod:
metadata:
annotations:
ad.datadoghq.com/kafka-connect-container-name.logs: '[{"type":"file", "source":"java","sourcecategory":"sourcecode", "service":"kafka-connect"}]'
ad.datadoghq.com/container.check_names: '["openmetrics"]'
ad.datadoghq.com/container.init_configs: '[{}]'
ad.datadoghq.com/container.instances: '[{"prometheus_url": "http://%%host%%:9404/metrics","namespace": "kafka","metrics": ["kafka_connect_connector_task_batch_size_avg"],"type_overrides": {"kafka_connect_incoming_byte_total": "gauge"}}]'
image:


i keep on getting: error: error validating "kafka-connect-prometeus.yaml": error validating data: ValidationError(KafkaConnect): unknown field "template" in io.strimzi.kafka.v1beta2.KafkaConnect; if you choose to ignore these errors, turn validation off with --validate=false

any advise on it (when removing it. all work smoothly...)
Thanks,
Amit


[ANNOUNCE] [RELEASE] Strimzi Test Container Operators 0.100.0

Maros Orsak
 

Strimzi Test Container 0.100.0 has been released. Main changes since 0.25.0 release include:
-   adds a variety of options how to configure broker to run Strimzi Kafka container:
  • with a specific Kafka version
  • with additional configuration
  • on a fixed port
  • with a custom server.properties file
  • with custom bootstrap servers
  • with custom Kafka image specified by System property
  • with KRaft (KIP-500) - supported from `Kafka 3.0.0`
  • allow specifying docker image by StrimziKafkaContainer constructor
-   adds Kafka 3.1.0 and solves the problem with KRaft (adding new listener CONTROLLER)
-   restriction KRaft for Kafka 2.8.1 (for using KRaft you need at least 3.0.0 Kafka version)
-   added parameterized test cases
-   adds support for ARM64 and s390x architectures
-   using pure [Kafka binaries](https://dlcdn.apache.org/kafka/) instead of using strimzi-kafka-operator built kafka images.
-   adds image for new Kafka 3.1.0
-   (Experimental) support for multi-node setup


Best regards,
--

Maroš Orsák

Quality Engineer - AMQ Streams

Red Hat

morsak@...   


[ANNOUCE] [Release Candidate] Strimzi test containers 0.100.0

Maros Orsak
 

Release candidate 3 of Strimzi test containers 0.100.0 is now available for testing.* The main changes from version 0.100-0-rc2 in this release include:
  • fix KRaft mode when Controller listens on the localhost instead of the container host
Images used by Strimzi test container (not change for images that's why rc2)

quay.io/strimzi-test-container/test-container:0.100.0-rc2-kafka-3.0.0
quay.io/strimzi-test-container/test-container:0.100.0-rc2-kafka-2.8.1

Maven artefacts
To test the Maven artefacts which are part of this release, use the staging repository by including the following in your pom.xml:
  <repositories>
    <repository>
      <id>staging</id>
      <url>https://oss.sonatype.org/content/repositories/iostrimzi-1156</url>
    </repository>
  </repositories>

Best regards,
--

Maroš Orsák

Quality Engineer - AMQ Streams

Red Hat

morsak@...   


[ANNOUCE] [Release Candidate] Strimzi test containers 0.100.0

Maros Orsak
 

Release candidate 2 of Strimzi test containers 0.100.0 is now available for testing.** The main changes from version 0.100-0-rc1 in this release include :
  •  adds a variety of options how to configure broker to run Strimzi Kafka container:
  • allow specifying docker image by StrimziKafkaContainer constructor
  •  adds Kafka 3.1.0 and solves the problem with KRaft (adding new listener CONTROLLER)
  • restriction KRaft for Kafka 2.8.1 (for using KRaft you need at least 3.0.0 Kafka version)
  • added parameterized test cases
Images used by Strimzi test container
 Maven artefacts

To test the Maven artefacts which are part of this release, use the staging repository by including the following in your pom.xml:

  <repositories>
    <repository>
      <id>staging</id>
      <url>https://oss.sonatype.org/content/repositories/iostrimzi-1155</url>
    </repository>
  </repositories>


--

Maroš Orsák

Quality Engineer - AMQ Streams

Red Hat

morsak@...   


[ANNOUNCE] [Release Candidate] Strimzi test containers 0.100.0 is out

Maros Orsak
 

Release candidate 1 of Strimzi test containers 0.100.0 is now available for testing. The main changes since version 0.25.0 are many options how to configure broker to run Strimzi Kafka container:
  • with a specific Kafka version 
  • with additional configuration
  • on a fixed port 
  • with KRaft (KIP-500)
  • with a custom server.properties file
  • with custom bootstrap servers
  • with custom Kafka image
  • adds support for ARM64 and s390x architectures
  • (Experimental) support for multi-node setup
Maven artefacts

To test the Maven artefacts which are part of this release, use the staging repository by including the following in your pom.xml
<repositories>
  <repository>
      <id>staging</id>
     <url>https://oss.sonatype.org/content/repositories/iostrimzi-1154</url>
  </repository>
</repositories>
Images used by Strimzi test container


Github links
--

Maroš Orsák

Quality Engineer - AMQ Streams

Red Hat

morsak@...   


[ANNOUNCE] [RELEASE] Strimzi Kafka Operators 0.27.1

Jakub Scholz
 

Strimzi Kafka Operators 0.27.1 has been released. The main changes in this release include:
* Fix Helm Chart issue when configuring additional environment variables
* Update Log4j2 to 2.17.1
* Update Fabric8 Kubernetes Client to 5.10.2

For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.27.1

Important: This release supports only the API version v1beta2 and CRD version apiextensions.k8s.io/v1. If upgrading from Strimzi 0.22, migration to v1beta2 needs to be completed for all Strimzi CRDs and CRs before the upgrade to 0.27 is done! If upgrading from Strimzi version earlier than 0.22, you need to first install the CRDs from Strimzi 0.22 and complete the migration to v1beta2 for all Strimzi CRDs and CRs before the upgrade to 0.27 is done! For more details about the CRD upgrades, see the documentation.

Thanks to everyone who contributed to this release!

Thanks & Regards
Strimzi team


[ANNOUNCE] [RELEASE] Strimzi Kafka Bridge 0.21.3

Paolo Patierno
 

Strimzi Kafka Bridge 0.21.3 is now availableIt will be also used in the upcoming 0.27.1 release of Strimzi Operators.
The main change is about the fix for the Log4j2 CVE-2021-44832.


Thanks to everyone who contributed to this release!

Thanks & Regards
Strimzi team


Paolo Patierno
Principal Software Engineer @ Red Hat
Microsoft MVP on Azure

Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience


[ANNOUNCE] [RELEASE] Strimzi Kafka Operators 0.27.0 and Strimzi Drain Cleaner 0.3.0

Jakub Scholz
 

Strimzi Kafka Operators 0.27.0 has been released. The main changes in this release include:
* (Experimental) support for AArch64 / ARM64 platform
* The ControlPlaneListener and ServiceAccountPatching feature gates are now in the beta phase and are enabled by default.
* Updated Log4j2 dependency to 2.17.0

For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.27.0

We have also created a video about the changes in this release which you can watch on our YouTube channel: https://youtu.be/cdAz997VC_0

Important: This release supports only the API version v1beta2 and CRD version apiextensions.k8s.io/v1. If upgrading from Strimzi 0.22, migration to v1beta2 needs to be completed for all Strimzi CRDs and CRs before the upgrade to 0.27 is done! If upgrading from Strimzi version earlier than 0.22, you need to first install the CRDs from Strimzi 0.22 and complete the migration to v1beta2 for all Strimzi CRDs and CRs before the upgrade to 0.27 is done! For more details about the CRD upgrades, see the documentation.

----

We also released Strimzi Drain Cleaner 0.3.0. The main changes in this release include support for both v1beta1 and v1 versions of the Eviction API and updates of dependencies.

For more details and installation files, go to https://github.com/strimzi/drain-cleaner/releases/tag/0.3.0

----

Thanks to everyone who contributed to these releases!

Thanks & Regards
Strimzi team


RC2 of Strimzi Kafka Operators 0.27.0

Jakub Scholz
 

Release candidate 2 of Strimzi Kafka Operators 0.27.0 is now available for testing. The changes since RC1 are:
* Fixed AArch64 version of the kaniko-executor container image
* Add Canary installation files to release archives


Any feedback can be provided on the Strimzi mailing list, on the #strimzi Slack channel on CNCF Slack or as a GitHub issue.

Thanks & Regards
Strimzi team

81 - 100 of 245