1. Cncf-Strimzi-Users
  2. Topics

Adding annotations and limits of kafka connect created pods

alonisser@...
 

Trying to add pod annotations (for our log connector) and resource request/limits to the KafkaConnect created resources
What should I add in the resource so it would create the pods/deployment with them(I suspect I already sent this message but can't find it now)

Jakub Scholz
 

Hi,

For specifying the resources, you can do something like this:
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnect
metadata:
  name: my-connect
  # ...
spec:
  # ...
  resources:
    requests:
      memory: 1Gi
      cpu: 500m
    limits:
      memory: 2Gi
      cpu: 1000m
  # ...

For the annotations, it depends what exactly do you want to annotate. But basically you can use this feature: https://strimzi.io/docs/latest/full.html#assembly-customizing-deployments-str
Your YAML would look something like this:
kind: KafkaConnect
metadata:
  name: my-connect
  # ...
spec:
  # ...
  template:
    deployment:
      metadata:
        annotations:
          myanno: myvalue
    pod:
      metadata:
        annotations:
          myanno: myvalue
  # ...

Thanks & Regards
Jakub




On Sun, Feb 23, 2020 at 10:42 AM <alonisser@...> wrote:
Trying to add pod annotations (for our log connector) and resource request/limits to the KafkaConnect created resources
What should I add in the resource so it would create the pods/deployment with them(I suspect I already sent this message but can't find it now)

alonisser@...
 

Thanks I would try that! I was missing the pod/deployment sub level

alonisser@...
 

Thanks, following  your advice I've added:
template:
  pod:
    metadata:
      annotations:
        ad.datadoghq.com/kafka-connect-container-name.logs: [{"type":"file", "source":"java","sourcecategory":"sourcecode", "service":"kafka-connect"}]

And it didn't work and I saw errors in the pods of the operator
following the error log 
at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: io.fabric8.kubernetes.api.model.WatchEvent["object"]->io.strimzi.api.kafka.model.KafkaConnect["spec"]->io.strimzi.api.kafka.model.KafkaConnectSpec["template"]->io.strimzi.api.kafka.model.template.KafkaConnectTemplate["pod"]->io.strimzi.api.kafka.model.template.PodTemplate["metadata"]->io.strimzi.api.kafka.model.template.MetadataTemplate["annotations"]->java.util.LinkedHashMap["ad.datadoghq.com/kafka-connect-container-name.logs"])

I've guessed it's about the array, so wrapping the array as a quoted string fixed the issue 
Thanks for the help again, and I'll hope this would be useful for someone else

Jakub Scholz
 

Yeah, the annotations are strings. So you need to wrap it into quotes to make it a string. Without them it will be interpreted as array.


On Fri, Feb 28, 2020 at 2:04 PM <alonisser@...> wrote:
Thanks, following  your advice I've added:
template:
  pod:
    metadata:
      annotations:
        ad.datadoghq.com/kafka-connect-container-name.logs: [{"type":"file", "source":"java","sourcecategory":"sourcecode", "service":"kafka-connect"}]

And it didn't work and I saw errors in the pods of the operator
following the error log 
at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: io.fabric8.kubernetes.api.model.WatchEvent["object"]->io.strimzi.api.kafka.model.KafkaConnect["spec"]->io.strimzi.api.kafka.model.KafkaConnectSpec["template"]->io.strimzi.api.kafka.model.template.KafkaConnectTemplate["pod"]->io.strimzi.api.kafka.model.template.PodTemplate["metadata"]->io.strimzi.api.kafka.model.template.MetadataTemplate["annotations"]->java.util.LinkedHashMap["ad.datadoghq.com/kafka-connect-container-name.logs"])

I've guessed it's about the array, so wrapping the array as a quoted string fixed the issue 
Thanks for the help again, and I'll hope this would be useful for someone else

amit.cahanovich@...
 

Hi,
i have the following configuration:
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster
namespace: kafka
annotations:
strimzi.io/use-connector-resources: "true"
spec:
template:
pod:
metadata:
annotations:
ad.datadoghq.com/kafka-connect-container-name.logs: '[{"type":"file", "source":"java","sourcecategory":"sourcecode", "service":"kafka-connect"}]'
ad.datadoghq.com/container.check_names: '["openmetrics"]'
ad.datadoghq.com/container.init_configs: '[{}]'
ad.datadoghq.com/container.instances: '[{"prometheus_url": "http://%%host%%:9404/metrics","namespace": "kafka","metrics": ["kafka_connect_connector_task_batch_size_avg"],"type_overrides": {"kafka_connect_incoming_byte_total": "gauge"}}]'
image:


i keep on getting: error: error validating "kafka-connect-prometeus.yaml": error validating data: ValidationError(KafkaConnect): unknown field "template" in io.strimzi.kafka.v1beta2.KafkaConnect; if you choose to ignore these errors, turn validation off with --validate=false

any advise on it (when removing it. all work smoothly...)
Thanks,
Amit

Jakub Scholz
 

The YAML you shared has no indentation. If that is how it really looks, it is indeed invalid. If the indentation was list during copy pasting, nobody can say what the original indentation was and what would be the error.

Thanks & Regards
Jakub

On Mon, Jan 31, 2022 at 4:08 PM <amit.cahanovich@...> wrote:
Hi,
i have the following configuration:
kind: KafkaConnect
metadata:
name: my-connect-cluster
namespace: kafka
annotations:
spec:
template:
pod:
metadata:
annotations:
ad.datadoghq.com/kafka-connect-container-name.logs: '[{"type":"file", "source":"java","sourcecategory":"sourcecode", "service":"kafka-connect"}]'
ad.datadoghq.com/container.instances: '[{"prometheus_url": "http://%%host%%:9404/metrics","namespace": "kafka","metrics": ["kafka_connect_connector_task_batch_size_avg"],"type_overrides": {"kafka_connect_incoming_byte_total": "gauge"}}]'
image:


i keep on getting: error: error validating "kafka-connect-prometeus.yaml": error validating data: ValidationError(KafkaConnect): unknown field "template" in io.strimzi.kafka.v1beta2.KafkaConnect; if you choose to ignore these errors, turn validation off with --validate=false

any advise on it (when removing it. all work smoothly...)
Thanks,
Amit

amit.cahanovich@...
 

for reference,
The following is kafkaconnect that is integrated with datadog (named my-connect-cluster)

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  name: my-connect-cluster
  namespace: kafka
  annotations:
    strimzi.io/use-connector-resources: "true"
spec:
  image: amitca71/strimzi-connect:0.27.1-kafka-2.8.1
  version: 2.8.1
  replicas: 3
  bootstrapServers: "${bootstrat_servers}"
  template:
    deployment:
      metadata:
        annotations:
          reloader.stakater.com/auto: "true"
    pod:
      metadata:
        annotations:
          ad.datadoghq.com/my-connect-cluster-connect.check_names: '["openmetrics"]'
          ad.datadoghq.com/my-connect-cluster-connect.init_configs: '[{}]'
          ad.datadoghq.com/my-connect-cluster-connect.instances: '[{"prometheus_url": "http://%%host%%:%%port%%/metrics","namespace": "strimzi_kafka_connect","metrics": ["kafka_connect_connector_task_batch_size_avg", "kafka_admin_client_node_incoming_byte_total", "kafka_connect_worker_connector_paused_task_count","kafka_connect_version_info"]}]'    
  

1 - 8 of 8