|
[ANNOUNCE] [RELEASE] Strimzi Test Container Operators 0.101.0 is out
Maros Orsak
Strimzi Test Container 0.101.0 has been released. Main changes since the 0.100.0 release include: - adding the ability to configure ZooKeeper container via zookeeper.properties - adding the ability to connect to internal ZooKeeper inside StrimziKafkaContainer using the getInternalZooKeeperConnect() method - constructor for an only specified number of brokers - better logs description - adds support for ppc64le architecture **Github links** - https://github.com/strimzi/test-container/releases/tag/0.101.0 - https://github.com/strimzi/test-container-images/releases/tag/0.101.0
|
|
|
|
[ANNOUCE] [Release Candidate 2] Strimzi test containers 0.101.0
Maros Orsak
Release candidate 2 of Strimzi test containers 0.101.0 is now available for testing. The main changes from version 0.101.0-rc1 in this release include :
Images used by Strimzi test container (no change in images) To test the Maven artefacts which are part of this release, use the staging repository by including the following in your pom.xml: Best regards,
|
|
|
|
[ANNOUCE] [Release Candidate 1] Strimzi test containers 0.101.0
Maros Orsak
Release candidate 1 of Strimzi test containers 0.101.0 is now available for testing. The main changes from version 0.100.0 in this release include : - adding the ability to configure ZooKeeper container via `zookeeper.properties` - adding the ability to connect to internal ZooKeeper inside `StrimziKafkaContainer` using `getInternalZooKeeperConnect()` method - constructor for an only specified number of brokers - better logs description - adds support for `ppc64le` architecture Images used by Strimzi test container To test the Maven artefacts which are part of this release, use the staging repository by including the following in your pom.xml: Best regards,
|
|
|
|
Re: Need help setting UP Strimzi Kafka Cluster with Cruise Control
Jakub Scholz
You have to do it manually. Kafka is not something that is normally scaled up or down very often. It is a stateful application with partitions and their replicas having a fixed assignment to a specific broker. Scaling the cluster up and down means reassigning the partitions between the nodes and since the partitions often contain very large amounts of data, these data need to be transferred. And that has of course its cost (it often takes a long time, generates financial costs etc.). So usually, you would scale the cluster up for example as you onboard new applications or as your applications grow. But you would for example not scale it up for a few minutes because there seems to be some traffic peak. So normally it is more the result of some long term planning rather than some quick decision of the HPA. Thanks & Regards Jakub
On Wed, Feb 23, 2022 at 9:48 PM Ranjeet Ranjan <ranjeet@...> wrote:
|
|
|
|
Re: Need help setting UP Strimzi Kafka Cluster with Cruise Control
Ranjeet Ranjan
Hi Jakub, I have set up Strimzi Kafka Cluster and Operator. I have still one doubt that I did not answer, do I need to scale the Kafka cluster and data balance up/down manually or will it scale up/down based on the Kubernetes HPA? Your help will be much appreciated.
On Tue, Feb 22, 2022 at 11:47 PM Jakub Scholz <jakub@...> wrote:
|
|
|
|
Re: Need help setting UP Strimzi Kafka Cluster with Cruise Control
Jakub Scholz
No, we do not have a Helm Chart for creating the Kafka custom resource. Thanks & Regards Jakub
On Tue, Feb 22, 2022 at 6:40 PM Ranjeet Ranjan <ranjeet@...> wrote:
|
|
|
|
Re: Need help setting UP Strimzi Kafka Cluster with Cruise Control
Ranjeet Ranjan
Hi Jakub, Thank you for your response. DO we have a helm as well for creating Kafka Cluster?
On Tue, Feb 22, 2022 at 11:08 PM Jakub Scholz <jakub@...> wrote:
|
|
|
|
Re: Need help setting UP Strimzi Kafka Cluster with Cruise Control
Jakub Scholz
HI Ranjeet, The Helm Chart installs only the operator for managing the Kafka clusters. Next, you need to create the Kafka custom resource describing the Kafka cluster you want to have, its configuration etc. The docs (https://strimzi.io/documentation/) cover all the different aspects which you can configure. We have also all kind of examples on GitHub: https://github.com/strimzi/strimzi-kafka-operator/tree/main/examples Thanks & Regards Jakub
On Tue, Feb 22, 2022 at 6:33 PM Ranjeet Ranjan <ranjeet@...> wrote:
|
|
|
|
Need help setting UP Strimzi Kafka Cluster with Cruise Control
Ranjeet Ranjan
Hi Team,
Let me introduce myself Ranjeet Ranjan, CEO & Founder Adsizzler Media. We have recently launched one new Ad Tech Startup Named http://adosiz.com/ We wanted to install but need help setting UP Strimzi Kafka Cluster with Cruise Control for Auto Scale. I have downloaded helm from https://strimzi.io/downloads/. I followed the below steps to install. Step -1: helm install strimzi-kafka-operator strimzi-kafka-operator -n test But it only installed strimzi-cluster-operator-5f56c65c5d-4pphv Now please let know how Kafka Cluster pod will be created. Thanks Ranjeet Ranjan
|
|
|
|
Re: adding annotation to service account
Jakub Scholz
Hi Amit, You can use the `template` section to declaratively customize annotations: https://strimzi.io/docs/operators/latest/full/configuring.html#assembly-customizing-kubernetes-resources-str ... for Service Accounts, from 0.27.0 with the ServiceAccountPatching feature gate enabled, it should be possible to use that even for existing clusters / service accounts. In earlier versions (or with the ServiceAccountPatching feature gate disables) it will be taken into account only when a new Service account is created. In that case, you can also annotate it simply with `kubectl annotate ...`. Thanks & Regards Jakub
On Mon, Feb 21, 2022 at 10:54 AM <amit.cahanovich@...> wrote: Hi,
|
|
|
|
adding annotation to service account
amit.cahanovich@...
Hi,
I work with strimzi (as kafka connect) on eks. I would like to add to the service account aws role annotation to s3 (something like: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxx:role/s3-read-role). Is there a trivial way to do it? Thanks, Amit
|
|
|
|
[ANNOUNCE] [RELEASE] Strimzi Kafka Operators 0.28.0
Jakub Scholz
Strimzi Kafka Operators 0.28.0 has been released. The main changes in this release include: * Add support for Kafka 3.1.0; remove Kafka 2.8.0 and 2.8.1 * Add support for `StrimziPodSet` resources (disabled by default through the `UseStrimziPodSets` feature gate) * Support custom authentication mechanisms in Kafka listeners * Intra-broker disk balancing using Cruise Control * Added the option `createBootstrapService` in the Kafka Spec to disable the creation of the bootstrap service for the Load Balancer Type Listener. It will save the cost of one load balancer resource, specially in the public cloud. * Add support for disabling the FIPS mode in OpenJDK * Fix renewing your own CA certificates For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.28.0 We have also created a video about the changes in this release which you can watch on our YouTube channel: https://youtu.be/PZKbrDUU1zo Important: This release supports only the API version v1beta2 and CRD version apiextensions.k8s.io/v1. If upgrading from Strimzi 0.22, migration to v1beta2 needs to be completed for all Strimzi CRDs and CRs before the upgrade to 0.28 is done! If upgrading from Strimzi version earlier than 0.22, you need to first install the CRDs from Strimzi 0.22 and complete the migration to v1beta2 for all Strimzi CRDs and CRs before the upgrade to 0.28 is done! For more details about the CRD upgrades, see the documentation. Thanks to everyone who contributed to these releases! Thanks & Regards Strimzi team
|
|
|
|
RC1 of Strimzi Kafka Operators 0.28.0 is available for testing
Jakub Scholz
Release candidate 1 of Strimzi Kafka Operators 0.28.0 is now available for testing. The main changes in this release include: * Add support for Kafka 3.1.0; remove Kafka 2.8.0 and 2.8.1 * Add support for `StrimziPodSet` resources (disabled by default through the `UseStrimziPodSets` feature gate) * Support custom authentication mechanisms in Kafka listeners * Intra-broker disk balancing using Cruise Control * Added the option `createBootstrapService` in the Kafka Spec to disable the creation of the bootstrap service for the Load Balancer Type Listener. It will save the cost of one load balancer resource, specially in the public cloud. * Add support for disabling the FIPS mode in OpenJDK * Fix renewing your own CA certificates For more details and installation files, go to https://github.com/strimzi/strimzi-kafka-operator/releases/tag/0.28.0-rc1 Any feedback can be provided on the Strimzi mailing list, on the #strimzi Slack channel on CNCF Slack or as a GitHub issue. Thanks & Regards Strimzi team
|
|
|
|
[ANNOUNCE] [RELEASE] Strimzi Kafka Bridge 0.21.4
Jakub Scholz
New version 0.21.4 of Strimzi Kafka Bridge has been released. The main changes in this release are: * Dependency updates (Configuration providers, Vert.x, Netty, Oauth client and more) * Add support for disabling the FIPS mode in OpenJDK * Add transactions `isolation.level` configuration parameter on consumer creation * Support for s390x platform For more details, go to https://github.com/strimzi/strimzi-kafka-bridge/releases/tag/0.21.4 Thanks to everyone who contributed to any of these releases! Regards Strimzi team
|
|
|
|
Re: Broker Hostname mismatch error seen with aiokafka APIs
Jakub Scholz
You are using the external loadbalancer listener with internal service name. That is causing the problem. You should either switch to one of the internal interfaces - e.g. the on on port 9093. Or you should use the proper loadbalancer bootstrap address which you can find in the status section of the Kafka custom resource (`kubectl get kafka -o yaml` should show it for you). In general, if your app runs inside the same Kubernetes, using the internal listener on port 9093 would be the right way to go => it should be cheaper and more performant than going through the loadbalancer. That should be used by apps outside your Kube cluster. Jakub
On Fri, Feb 4, 2022 at 10:49 AM <udaykumartj@...> wrote:
|
|
|
|
Broker Hostname mismatch error seen with aiokafka APIs
udaykumartj@...
Hi all,
I am getting the below error for the code (which uses aiokafka APIs) that i have pasted below (also provided kafka resource yaml snippet). Instead of kafka broker hostname, if i give IP address, it works. Not sure why kafka broker (dns) hostname is not working. Please help. Unable connect to "strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094": [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local'. (_ssl.c:1129) Note: If i use confluent_kafka APIs (Consumer, Producer), i dont see this issue. Code snippet: context = create_ssl_context( cafile='/etc/vcerts/cluster/ca.crt',
certfile='/etc/vcerts/client/user.crt',
keyfile='/etc/vcerts/client/user.key',
)
consumer = AIOKafkaConsumer(
my_topic, bootstrap_servers='strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094',
auto_offset_reset='latest',
group_id=group_id,
security_protocol="SSL",
ssl_context=context)
await consumer.start() <=== This line throws the above error.
Kafka spec: apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: strimzi-kafka
spec:
kafka:
version: 3.0.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: tls
- name: external
port: 9094
type: loadbalancer
tls: true
authentication:
type: tls
template:
clusterCaCert:
metadata:
....
|
|
|
|
Broker Hostname mismatch error seen with aiokafka APIs
udaykumartj@...
Hi all,
I am getting the below error for the code (which uses aiokafka APIs) that i have pasted below (also provided kafka resource yaml snippet). Instead of kafka broker hostname, if i give IP address, it works. Not sure why kafka broker (dns) hostname is not working. Please help. Unable connect to "strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094": [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local'. (_ssl.c:1129) Note: If i use confluent_kafka APIs (Consumer, Producer), i dont see this issue. Code snippet: context = create_ssl_context( cafile='/etc/vcerts/cluster/ca.crt',
certfile='/etc/vcerts/client/user.crt',
keyfile='/etc/vcerts/client/user.key',
)
consumer = AIOKafkaConsumer(
my_topic, bootstrap_servers='strimzi-kafka-kafka-external-bootstrap.kafka.svc.cluster.local:9094',
auto_offset_reset='latest',
group_id=group_id,
security_protocol="SSL",
ssl_context=context)
await consumer.start() <=== This line throws the above error.
Kafka spec: apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: strimzi-kafka
spec:
kafka:
version: 3.0.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: tls
- name: external
port: 9094
type: loadbalancer
tls: true
authentication:
type: tls
template:
clusterCaCert:
metadata:
....
|
|
|
|
[ANNOUNCE] [RELEASE] Mirror Maker 2 Extensions 1.2.0, EnvVar Configuration Provider 1.0.0 and Kubernetes Configuration Provider 1.0.0
Jakub Scholz
New versions of Mirror Maker 2 Extensions, EnvVar Configuration Provider and Kubernetes Configuration Provider have been released. The biggest change involves the Mirror Maker 2 Extensions. The Strimzi Identity Replication Policy is now deprecated and all users of Kafka 3.0.0 and higher are recommended to update their configurations and use the Identity Replication Policy which is directly part of the Apache Kafka project. The 1.2.0 release of the Mirror Maker 2 Extensions is used for backwards compatibility and provides the same behaviour as the Kafka policy but using the original Strimzi class name. The Config Provider releases contain minor improvements and dependency updates. For more details, visit the release pages of each project: * Mirror Maker 2 Extensions: https://github.com/strimzi/mirror-maker-2-extensions/releases/tag/1.2.0 * Kafka EnvVar Configuration Provider: https://github.com/strimzi/kafka-env-var-config-provider/releases/tag/1.0.0 * Kafka Kubernetes Configuration Provider: https://github.com/strimzi/kafka-kubernetes-config-provider/releases/tag/1.0.0 Thanks to everyone who contributed to any of these releases! Regards Strimzi team
|
|
|
|
Re: Adding annotations and limits of kafka connect created pods
amit.cahanovich@...
for reference,
The following is kafkaconnect that is integrated with datadog (named my-connect-cluster) apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster
namespace: kafka
annotations:
strimzi.io/use-connector-resources: "true"
spec:
image: amitca71/strimzi-connect:0.27.1-kafka-2.8.1
version: 2.8.1
replicas: 3
bootstrapServers: "${bootstrat_servers}"
template:
deployment:
metadata:
annotations:
reloader.stakater.com/auto: "true"
pod:
metadata:
annotations:
ad.datadoghq.com/my-connect-cluster-connect.check_names: '["openmetrics"]'
ad.datadoghq.com/my-connect-cluster-connect.init_configs: '[{}]'
ad.datadoghq.com/my-connect-cluster-connect.instances: '[{"prometheus_url": "http://%%host%%:%%port%%/metrics","namespace": "strimzi_kafka_connect","metrics": ["kafka_connect_connector_task_batch_size_avg", "kafka_admin_client_node_incoming_byte_total", "kafka_connect_worker_connector_paused_task_count","kafka_connect_version_info"]}]'
|
|
|
|
RC1 of Mirror Maker 2 Extensions 1.2.0, EnvVar Configuration Provider 1.0.0 and Kubernetes Configuration Provider 0.1.1
Jakub Scholz
As we are preparing for the next release of the Strimzi operators, we are doing different minor and patch releases of some of our smaller projects used as dependencies. Today, we released the Release Candidate 1 of several sub-projects: * Mirror Maker 2 Extensions: https://github.com/strimzi/mirror-maker-2-extensions/releases/tag/1.2.0-rc1 * Kafka EnvVar Configuration Provider: https://github.com/strimzi/kafka-env-var-config-provider/releases/tag/1.0.0-rc1 * Kafka Kubernetes Configuration Provider: https://github.com/strimzi/kafka-kubernetes-config-provider/releases/tag/1.0.0-rc1 They contain some smaller bug fixes and improvements as well as updated dependencies. If you are interested, give them a try! Any feedback can be provided on the Strimzi mailing list, on the #strimzi Slack channel on CNCF Slack or as a GitHub issue. Thanks & Regards Strimzi team
|
|
|
