|
adding annotation to service account
amit.cahanovich@...
Hi,
I work with strimzi (as kafka connect) on eks. I would like to add to the service account aws role annotation to s3 (something like: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxx:role/s3-read-role). Is there a trivial way to do it? Thanks, Amit
|
|
|
|
[ANNOUNCE] [RELEASE] Strimzi Kafka Operators 0.28.0
Jakub Scholz
Strimzi Kafka Operators 0.28.0 has been released. The main changes in this release include: * Add support for Kafka 3.1.0; remove Kafka 2.8.0 and 2.8.1 * Add support for `StrimziPodSet` resources (disabled by default through the `UseStrimziPodSets` feature gate) * Support custom authentication mechanisms in Kafka listeners * Intra-broker disk balancing using Cruise Control * Added the option `createBootstrapService` in the Kafka Spec to disable the creation of the bootstrap service for the Load Balancer Type Listener. It will save the cost of one load balancer resource, specially in the public cloud. * Add support for disabling the FIPS mode in OpenJDK * Fix renewing your own CA certificates For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.28.0 We have also created a video about the changes in this release which you can watch on our YouTube channel: https://youtu.be/PZKbrDUU1zo Important: This release supports only the API version v1beta2 and CRD version apiextensions.k8s.io/v1. If upgrading from Strimzi 0.22, migration to v1beta2 needs to be completed for all Strimzi CRDs and CRs before the upgrade to 0.28 is done! If upgrading from Strimzi version earlier than 0.22, you need to first install the CRDs from Strimzi 0.22 and complete the migration to v1beta2 for all Strimzi CRDs and CRs before the upgrade to 0.28 is done! For more details about the CRD upgrades, see the documentation. Thanks to everyone who contributed to these releases! Thanks & Regards Strimzi team
|
|
|
|
RC1 of Strimzi Kafka Operators 0.28.0 is available for testing
Jakub Scholz
Release candidate 1 of Strimzi Kafka Operators 0.28.0 is now available for testing. The main changes in this release include: * Add support for Kafka 3.1.0; remove Kafka 2.8.0 and 2.8.1 * Add support for `StrimziPodSet` resources (disabled by default through the `UseStrimziPodSets` feature gate) * Support custom authentication mechanisms in Kafka listeners * Intra-broker disk balancing using Cruise Control * Added the option `createBootstrapService` in the Kafka Spec to disable the creation of the bootstrap service for the Load Balancer Type Listener. It will save the cost of one load balancer resource, specially in the public cloud. * Add support for disabling the FIPS mode in OpenJDK * Fix renewing your own CA certificates For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.28.0-rc1 Any feedback can be provided on the Strimzi mailing list, on the #strimzi Slack channel on CNCF Slack or as a GitHub issue. Thanks & Regards Strimzi team
|
|
|
|
[ANNOUNCE] [RELEASE] Strimzi Kafka Bridge 0.21.4
Jakub Scholz
New version 0.21.4 of Strimzi Kafka Bridge has been released. The main changes in this release are: * Dependency updates (Configuration providers, Vert.x, Netty, Oauth client and more) * Add support for disabling the FIPS mode in OpenJDK * Add transactions `isolation.level` configuration parameter on consumer creation * Support for s390x platform For more details, go to https://github.com/strimzi/strimzi-kafka-bridge/releases/tag/0.21.4 Thanks to everyone who contributed to any of these releases! Regards Strimzi team
|
|
|
|
Re: Broker Hostname mismatch error seen with aiokafka APIs
Jakub Scholz
You are using the external loadbalancer listener with internal service name. That is causing the problem. You should either switch to one of the internal interfaces - e.g. the on on port 9093. Or you should use the proper loadbalancer bootstrap address which you can find in the status section of the Kafka custom resource (`kubectl get kafka -o yaml` should show it for you). In general, if your app runs inside the same Kubernetes, using the internal listener on port 9093 would be the right way to go => it should be cheaper and more performant than going through the loadbalancer. That should be used by apps outside your Kube cluster. Jakub
On Fri, Feb 4, 2022 at 10:49 AM <udaykumartj@...> wrote:
|
|
|
|
Broker Hostname mismatch error seen with aiokafka APIs
udaykumartj@...
Hi all,
I am getting the below error for the code (which uses aiokafka APIs) that i have pasted below (also provided kafka resource yaml snippet). Instead of kafka broker hostname, if i give IP address, it works. Not sure why kafka broker (dns) hostname is not working. Please help. Unable connect to "strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094": [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local'. (_ssl.c:1129) Note: If i use confluent_kafka APIs (Consumer, Producer), i dont see this issue. Code snippet: context = create_ssl_context( cafile='/etc/vcerts/cluster/ca.crt',
certfile='/etc/vcerts/client/user.crt',
keyfile='/etc/vcerts/client/user.key',
)
consumer = AIOKafkaConsumer(
my_topic, bootstrap_servers='strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094',
auto_offset_reset='latest',
group_id=group_id,
security_protocol="SSL",
ssl_context=context)
await consumer.start() <=== This line throws the above error.
Kafka spec: apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: strimzi-kafka
spec:
kafka:
version: 3.0.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: tls
- name: external
port: 9094
type: loadbalancer
tls: true
authentication:
type: tls
template:
clusterCaCert:
metadata:
....
|
|
|
|
Broker Hostname mismatch error seen with aiokafka APIs
udaykumartj@...
Hi all,
I am getting the below error for the code (which uses aiokafka APIs) that i have pasted below (also provided kafka resource yaml snippet). Instead of kafka broker hostname, if i give IP address, it works. Not sure why kafka broker (dns) hostname is not working. Please help. Unable connect to "strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094": [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local'. (_ssl.c:1129) Note: If i use confluent_kafka APIs (Consumer, Producer), i dont see this issue. Code snippet: context = create_ssl_context( cafile='/etc/vcerts/cluster/ca.crt',
certfile='/etc/vcerts/client/user.crt',
keyfile='/etc/vcerts/client/user.key',
)
consumer = AIOKafkaConsumer(
my_topic, bootstrap_servers='strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094',
auto_offset_reset='latest',
group_id=group_id,
security_protocol="SSL",
ssl_context=context)
await consumer.start() <=== This line throws the above error.
Kafka spec: apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: strimzi-kafka
spec:
kafka:
version: 3.0.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: tls
- name: external
port: 9094
type: loadbalancer
tls: true
authentication:
type: tls
template:
clusterCaCert:
metadata:
....
|
|
|
|
[ANNOUNCE] [RELEASE] Mirror Maker 2 Extensions 1.2.0, EnvVar Configuration Provider 1.0.0 and Kubernetes Configuration Provider 1.0.0
Jakub Scholz
New versions of Mirror Maker 2 Extensions, EnvVar Configuration Provider and Kubernetes Configuration Provider have been released. The biggest change involves the Mirror Maker 2 Extensions. The Strimzi Identity Replication Policy is now deprecated and all users of Kafka 3.0.0 and higher are recommended to update their configurations and use the Identity Replication Policy which is directly part of the Apache Kafka project. The 1.2.0 release of the Mirror Maker 2 Extensions is used for backwards compatibility and provides the same behaviour as the Kafka policy but using the original Strimzi class name. The Config Provider releases contain minor improvements and dependency updates. For more details, visit the release pages of each project: * Mirror Maker 2 Extensions: https://github.com/strimzi/mirror-maker-2-extensions/releases/tag/1.2.0 * Kafka EnvVar Configuration Provider: https://github.com/strimzi/kafka-env-var-config-provider/releases/tag/1.0.0 * Kafka Kubernetes Configuration Provider: https://github.com/strimzi/kafka-kubernetes-config-provider/releases/tag/1.0.0 Thanks to everyone who contributed to any of these releases! Regards Strimzi team
|
|
|
|
Re: Adding annotations and limits of kafka connect created pods
amit.cahanovich@...
for reference,
The following is kafkaconnect that is integrated with datadog (named my-connect-cluster) apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster
namespace: kafka
annotations:
strimzi.io/use-connector-resources: "true"
spec:
image: amitca71/strimzi-connect:0.27.1-kafka-2.8.1
version: 2.8.1
replicas: 3
bootstrapServers: "${bootstrat_servers}"
template:
deployment:
metadata:
annotations:
reloader.stakater.com/auto: "true"
pod:
metadata:
annotations:
ad.datadoghq.com/my-connect-cluster-connect.check_names: '["openmetrics"]'
ad.datadoghq.com/my-connect-cluster-connect.init_configs: '[{}]'
ad.datadoghq.com/my-connect-cluster-connect.instances: '[{"prometheus_url": "http://%%host%%:%%port%%/metrics","namespace": "strimzi_kafka_connect","metrics": ["kafka_connect_connector_task_batch_size_avg", "kafka_admin_client_node_incoming_byte_total", "kafka_connect_worker_connector_paused_task_count","kafka_connect_version_info"]}]'
|
|
|
|
RC1 of Mirror Maker 2 Extensions 1.2.0, EnvVar Configuration Provider 1.0.0 and Kubernetes Configuration Provider 0.1.1
Jakub Scholz
As we are preparing for the next release of the Strimzi operators, we are doing different minor and patch releases of some of our smaller projects used as dependencies. Today, we released the Release Candidate 1 of several sub-projects: * Mirror Maker 2 Extensions: https://github.com/strimzi/mirror-maker-2-extensions/releases/tag/1.2.0-rc1 * Kafka EnvVar Configuration Provider: https://github.com/strimzi/kafka-env-var-config-provider/releases/tag/1.0.0-rc1 * Kafka Kubernetes Configuration Provider: https://github.com/strimzi/kafka-kubernetes-config-provider/releases/tag/1.0.0-rc1 They contain some smaller bug fixes and improvements as well as updated dependencies. If you are interested, give them a try! Any feedback can be provided on the Strimzi mailing list, on the #strimzi Slack channel on CNCF Slack or as a GitHub issue. Thanks & Regards Strimzi team
|
|
|
|
Re: Adding annotations and limits of kafka connect created pods
Jakub Scholz
The YAML you shared has no indentation. If that is how it really looks, it is indeed invalid. If the indentation was list during copy pasting, nobody can say what the original indentation was and what would be the error. Thanks & Regards Jakub
On Mon, Jan 31, 2022 at 4:08 PM <amit.cahanovich@...> wrote: Hi,
|
|
|
|
Re: Adding annotations and limits of kafka connect created pods
amit.cahanovich@...
Hi,
i have the following configuration: apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster
namespace: kafka
annotations:
strimzi.io/use-connector-resources: "true"
spec:
template:
pod:
metadata:
annotations:
ad.datadoghq.com/kafka-connect-container-name.logs: '[{"type":"file", "source":"java","sourcecategory":"sourcecode", "service":"kafka-connect"}]'
ad.datadoghq.com/container.check_names: '["openmetrics"]'
ad.datadoghq.com/container.init_configs: '[{}]'
ad.datadoghq.com/container.instances: '[{"prometheus_url": "http://%%host%%:9404/metrics","namespace": "kafka","metrics": ["kafka_connect_connector_task_batch_size_avg"],"type_overrides": {"kafka_connect_incoming_byte_total": "gauge"}}]'
image:
i keep on getting: error: error validating "kafka-connect-prometeus.yaml": error validating data: ValidationError(KafkaConnect): unknown field "template" in io.strimzi.kafka.v1beta2.KafkaConnect; if you choose to ignore these errors, turn validation off with --validate=false any advise on it (when removing it. all work smoothly...) Thanks, Amit
|
|
|
|
[ANNOUNCE] [RELEASE] Strimzi Test Container Operators 0.100.0
Maros Orsak
Strimzi Test Container 0.100.0 has been released. Main changes since 0.25.0 release include: - adds a variety of options how to configure broker to run Strimzi Kafka container:
- adds Kafka 3.1.0 and solves the problem with KRaft (adding new listener CONTROLLER) - restriction KRaft for Kafka 2.8.1 (for using KRaft you need at least 3.0.0 Kafka version) - added parameterized test cases - adds support for ARM64 and s390x architectures - using pure [Kafka binaries](https://dlcdn.apache.org/kafka/) instead of using strimzi-kafka-operator built kafka images. - adds image for new Kafka 3.1.0 - (Experimental) support for multi-node setup Github links Best regards,
|
|
|
|
[ANNOUCE] [Release Candidate] Strimzi test containers 0.100.0
Maros Orsak
Release candidate 3 of Strimzi test containers 0.100.0 is now available for testing.* The main changes from version 0.100-0-rc2 in this release include:
quay.io/strimzi-test-container/test-container:0.100.0-rc2-kafka-2.8.1 Maven artefacts To test the Maven artefacts which are part of this release, use the staging repository by including the following in your pom.xml: <repositories> <repository> <id>staging</id> <url>https://oss.sonatype.org/content/repositories/iostrimzi-1156</url> </repository> </repositories> Best regards,
|
|
|
|
[ANNOUCE] [Release Candidate] Strimzi test containers 0.100.0
Maros Orsak
Release candidate 2 of Strimzi test containers 0.100.0 is now available for testing.** The main changes from version 0.100-0-rc1 in this release include :
To test the Maven artefacts which are part of this release, use the staging repository by including the following in your pom.xml: <repositories> <repository> <id>staging</id> <url>https://oss.sonatype.org/content/repositories/iostrimzi-1155</url> </repository> </repositories> Github links
|
|
|
|
[ANNOUNCE] [Release Candidate] Strimzi test containers 0.100.0 is out
Maros Orsak
Release candidate 1 of Strimzi test containers 0.100.0 is now available for testing. The main changes since version 0.25.0 are many options how to configure broker to run Strimzi Kafka container:
Maven artefacts To
test the Maven artefacts which are part of this release, use the
staging repository by including the following in your pom.xml <repositories>
<repository>
<id>staging</id>
<url>https://oss.sonatype.org/content/repositories/iostrimzi-1154</url>
</repository>
</repositories>Images used by Strimzi test container Github links
|
|
|
|
[ANNOUNCE] [RELEASE] Strimzi Kafka Operators 0.27.1
Jakub Scholz
Strimzi Kafka Operators 0.27.1 has been released. The main changes in this release include: * Fix Helm Chart issue when configuring additional environment variables * Update Log4j2 to 2.17.1 * Update Fabric8 Kubernetes Client to 5.10.2 For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.27.1 Important: This release supports only the API version v1beta2 and CRD version apiextensions.k8s.io/v1. If upgrading from Strimzi 0.22, migration to v1beta2 needs to be completed for all Strimzi CRDs and CRs before the upgrade to 0.27 is done! If upgrading from Strimzi version earlier than 0.22, you need to first install the CRDs from Strimzi 0.22 and complete the migration to v1beta2 for all Strimzi CRDs and CRs before the upgrade to 0.27 is done! For more details about the CRD upgrades, see the documentation. Thanks to everyone who contributed to this release! Thanks & Regards Strimzi team
|
|
|
|
[ANNOUNCE] [RELEASE] Strimzi Kafka Bridge 0.21.3
Paolo Patierno
Strimzi Kafka Bridge 0.21.3 is now available. It
will be also used in the upcoming 0.27.1 release of Strimzi Operators.
The main change is about the fix for the Log4j2 CVE-2021-44832.
For more details, go to https://github.com/strimzi/strimzi-kafka-bridge/releases/tag/0.21.3
Thanks & Regards
Strimzi team
Paolo Patierno
Principal Software Engineer @ Red Hat
Microsoft MVP on Azure
|
|
|
|
[ANNOUNCE] [RELEASE] Strimzi Kafka Operators 0.27.0 and Strimzi Drain Cleaner 0.3.0
Jakub Scholz
Strimzi Kafka Operators 0.27.0 has been released. The main changes in this release include: * (Experimental) support for AArch64 / ARM64 platform * The ControlPlaneListener and ServiceAccountPatching feature gates are now in the beta phase and are enabled by default. * Updated Log4j2 dependency to 2.17.0 For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.27.0 We have also created a video about the changes in this release which you can watch on our YouTube channel: https://youtu.be/cdAz997VC_0 Important: This release supports only the API version v1beta2 and CRD version apiextensions.k8s.io/v1. If upgrading from Strimzi 0.22, migration to v1beta2 needs to be completed for all Strimzi CRDs and CRs before the upgrade to 0.27 is done! If upgrading from Strimzi version earlier than 0.22, you need to first install the CRDs from Strimzi 0.22 and complete the migration to v1beta2 for all Strimzi CRDs and CRs before the upgrade to 0.27 is done! For more details about the CRD upgrades, see the documentation. ---- We also released Strimzi Drain Cleaner 0.3.0. The main changes in this release include support for both v1beta1 and v1 versions of the Eviction API and updates of dependencies. For more details and installation files, go to https://github.com/strimzi/drain-cleaner/releases/tag/0.3.0 ---- Thanks to everyone who contributed to these releases! Thanks & Regards Strimzi team
|
|
|
|
RC2 of Strimzi Kafka Operators 0.27.0
Jakub Scholz
Release candidate 2 of Strimzi Kafka Operators 0.27.0 is now available for testing. The changes since RC1 are: * Fixed AArch64 version of the kaniko-executor container image * Add Canary installation files to release archives For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.27.0-rc2 Any feedback can be provided on the Strimzi mailing list, on the #strimzi Slack channel on CNCF Slack or as a GitHub issue. Thanks & Regards Strimzi team
|
|
|
