Kafka clients are getting disconnected frequently


Majid Ma
 

Hello Strimizi Users,

We have deployed kafka using strmizi operator v0.20.1 and we see the kafka clients are getting disconnected frequently, though we don't see any stability issues in the applications using kafka.

We see the same behaviour in the other environments where we have deployed kafka using the strimzi operator v0.20.0

We don't see this issue for the environments that does not uses kafka deployed using strimizi operator.

The following is our application logs, If you see there are continuous connection failures.

2021-01-20 05:55:36,651 - WARNING - AA - kafka.client - --- - Node 2 connection failed -- refreshing metadata
2021-01-20 05:55:36,944 - WARNING - AA - kafka.client - --- - Node 0 connection failed -- refreshing metadata
2021-01-20 05:55:39,944 - WARNING - AA - kafka.client - --- - Node 0 connection failed -- refreshing metadata
2021-01-20 05:56:07,464 - WARNING - AA - kafka.client - --- - Node 1 connection failed -- refreshing metadata
2021-01-20 05:56:12,944 - WARNING - AA - kafka.client - --- - Node 0 connection failed -- refreshing metadata
2021-01-20 05:56:13,227 - WARNING - AA - kafka.client - --- - Node 2 connection failed -- refreshing metadata
2021-01-20 05:56:13,384 - WARNING - AA - kafka.client - --- - Node 1 connection failed -- refreshing metadata
2021-01-20 05:56:14,943 - WARNING - AA - kafka.client - --- - Node 0 connection failed -- refreshing metadata

Any thoughts or inputs/suggestions will be appreciated.


Jakub Scholz
 

As I said on Slack ... I wonder about two things ...
* What client does it use? From the log, it does not seem to be like the official Java client. Might be worth trying it (for example with some dummy data) with the official client to eliminate any possible client issues and get a log which is easier to understand.
* Is the connection from outside or inside Kubernetes? If it is from outside, it might be worth trying it from inside and see if that makes any difference.

Jakub

On Wed, Jan 20, 2021 at 10:53 AM Majid Ma <majid.ma@...> wrote:
Hello Strimizi Users,

We have deployed kafka using strmizi operator v0.20.1 and we see the kafka clients are getting disconnected frequently, though we don't see any stability issues in the applications using kafka.

We see the same behaviour in the other environments where we have deployed kafka using the strimzi operator v0.20.0

We don't see this issue for the environments that does not uses kafka deployed using strimizi operator.

The following is our application logs, If you see there are continuous connection failures.

2021-01-20 05:55:36,651 - WARNING - AA - kafka.client - --- - Node 2 connection failed -- refreshing metadata
2021-01-20 05:55:36,944 - WARNING - AA - kafka.client - --- - Node 0 connection failed -- refreshing metadata
2021-01-20 05:55:39,944 - WARNING - AA - kafka.client - --- - Node 0 connection failed -- refreshing metadata
2021-01-20 05:56:07,464 - WARNING - AA - kafka.client - --- - Node 1 connection failed -- refreshing metadata
2021-01-20 05:56:12,944 - WARNING - AA - kafka.client - --- - Node 0 connection failed -- refreshing metadata
2021-01-20 05:56:13,227 - WARNING - AA - kafka.client - --- - Node 2 connection failed -- refreshing metadata
2021-01-20 05:56:13,384 - WARNING - AA - kafka.client - --- - Node 1 connection failed -- refreshing metadata
2021-01-20 05:56:14,943 - WARNING - AA - kafka.client - --- - Node 0 connection failed -- refreshing metadata

Any thoughts or inputs/suggestions will be appreciated.