1. Topics
  2. RFC: Strimzi

RFC: Strimzi

Chris Aniszczyk
 

Hey all, we heard from the Strimzi project today:
https://github.com/cncf/toc/issues/138

There were questions about if the ZK aspects of the operator could be
pulled out as its own project. There were also other questions around
other kafka operators in the ecosystem.

Anyways, the Strimzi project is looking for TOC sponsors for the sandbox.

--
Chris Aniszczyk (@cra) | +1-512-961-6719
alexis richardson
 

I am a potential sponsor.

for me to buy in: I'd like to see a neutral gov & roadmap plus plans
from david on achieving unambiguous neutrality, plus thoughts on what
would be useful as a 'spun out' tool, eg ZK, Quotas, ..

example of lean governance that works for one project
https://github.com/cortexproject/cortex/blob/master/GOVERNANCE.md


On Tue, Apr 9, 2019 at 9:00 AM Chris Aniszczyk
<caniszczyk@...> wrote:

Hey all, we heard from the Strimzi project today:
https://github.com/cncf/toc/issues/138

There were questions about if the ZK aspects of the operator could be
pulled out as its own project. There were also other questions around
other kafka operators in the ecosystem.

Anyways, the Strimzi project is looking for TOC sponsors for the sandbox.

--
Chris Aniszczyk (@cra) | +1-512-961-6719
David Ingham
 

Hi,

Thank you for your comments and the opportunity to present Strimzi to the TOC yesterday.


We are keen to work towards neutral open governance of the project and think that the link to the Cortex governance is a good starting point. Until now the project has prioritised ease of administration over more formal processes for PR review and merging, but now is a good time to start adopting a more formal model. We will therefore start by reducing the number of our own committers and explore expanding the pool of non-Red Hat committers on the project (this might however take some more time). We’ll keep you informed on our progress.


During the call it was clear that other people were also interested in the idea of having a standalone Zookeeper deployment - without Apache Kafka. In order to make a start on some initial design work for this it would be great to get more input from those people about how they think this would be used. For example, if they had other Zookeeper-dependent projects in mind it would be great to know about them.


Can you elaborate on the idea around quotas? We think the idea here would be a messaging-agnostic custom resource/API expressing a quota for authenticated message senders to (abstract) message destinations, but we’re not sure we understood you correctly.


Finally, if we addressed these points are there other TOC members who would be willing to sponsor the project or do others have further issues they would like to discuss?

Best regards,
Dave.
alexis richardson
 

David

Thanks. I would prioritise open governance over reducing your
committers, but up to you.

Re quotas. Any multi-tenant system that holds data for any length of
time faces the question of how available resources may be allocated to
tenants. Typically a quota system is used. If your users are single
tenant, then you don't need this.

a

On Wed, Apr 10, 2019 at 8:19 AM <dingham@...> wrote:

Hi,

Thank you for your comments and the opportunity to present Strimzi to the TOC yesterday.


We are keen to work towards neutral open governance of the project and think that the link to the Cortex governance is a good starting point. Until now the project has prioritised ease of administration over more formal processes for PR review and merging, but now is a good time to start adopting a more formal model. We will therefore start by reducing the number of our own committers and explore expanding the pool of non-Red Hat committers on the project (this might however take some more time). We’ll keep you informed on our progress.


During the call it was clear that other people were also interested in the idea of having a standalone Zookeeper deployment - without Apache Kafka. In order to make a start on some initial design work for this it would be great to get more input from those people about how they think this would be used. For example, if they had other Zookeeper-dependent projects in mind it would be great to know about them.


Can you elaborate on the idea around quotas? We think the idea here would be a messaging-agnostic custom resource/API expressing a quota for authenticated message senders to (abstract) message destinations, but we’re not sure we understood you correctly.


Finally, if we addressed these points are there other TOC members who would be willing to sponsor the project or do others have further issues they would like to discuss?

Best regards,
Dave.
Tom Bentley
 

Hi Alexis,

On Wed, Apr 10, 2019 at 4:47 PM alexis richardson <alexis@...> wrote:
David

Thanks.  I would prioritise open governance over reducing your
committers, but up to you.

I just wanted to let people know that the project has now adopted an open governance model (https://github.com/strimzi/strimzi-kafka-operator/blob/master/GOVERNANCE.md) as a first step. We expect to broaden the maintainers beyond Red Hat over time.
 
Re quotas.  Any multi-tenant system that holds data for any length of
time faces the question of how available resources may be allocated to
tenants.  Typically a quota system is used.  If your users are single
tenant, then you don't need this.

Kafka itself is not multi-tennant. Neither messages nor topics are owned authorised entities. So although it has different retention policies for messages, they're not associated with users or tenants. Rate limiting / quotas of connected clients are features of Kafka itself, rather than being something Strimzi adds. Therefore it might be hard to separate it as a general concept.
alexis richardson
 

Tom




On Wed, 1 May 2019, 04:23 Tom Bentley, <tbentley@...> wrote:
Hi Alexis,

On Wed, Apr 10, 2019 at 4:47 PM alexis richardson <alexis@...> wrote:
David

Thanks.  I would prioritise open governance over reducing your
committers, but up to you.

I just wanted to let people know that the project has now adopted an open governance model (https://github.com/strimzi/strimzi-kafka-operator/blob/master/GOVERNANCE.md) as a first step. We expect to broaden the maintainers beyond Red Hat over time.

Fantastic news - congratulations.


 
Re quotas.  Any multi-tenant system that holds data for any length of
time faces the question of how available resources may be allocated to
tenants.  Typically a quota system is used.  If your users are single
tenant, then you don't need this.

Kafka itself is not multi-tennant. Neither messages nor topics are owned authorised entities. So although it has different retention policies for messages, they're not associated with users or tenants. Rate limiting / quotas of connected clients are features of Kafka itself, rather than being something Strimzi adds. Therefore it might be hard to separate it as a general concept.

Perhaps I misunderstood but isn't one goal of Strimzi to enable Kafka-aaS.
Tom Bentley
 

Hi Alexis,

On Wed, May 1, 2019 at 10:11 AM Alexis Richardson <alexis@...> wrote:
Kafka itself is not multi-tennant. Neither messages nor topics are owned authorised entities. So although it has different retention policies for messages, they're not associated with users or tenants. Rate limiting / quotas of connected clients are features of Kafka itself, rather than being something Strimzi adds. Therefore it might be hard to separate it as a general concept.

Perhaps I misunderstood but isn't one goal of Strimzi to enable Kafka-aaS.


Strimzi provides operators which make it easy for different users to create their own Kafka clusters, (plus topics, users etc within those clusters). It does not provide support for a single Kafka cluster supporting multiple tenants. Obviously it is possible provide abstractions to do that on top of Kafka, but that's not what Strimzi provides.

Regards,

Tom
alexis richardson
 

Tom

On Thu, 2 May 2019, 09:26 Tom Bentley, <tbentley@...> wrote:
Hi Alexis,

On Wed, May 1, 2019 at 10:11 AM Alexis Richardson <alexis@...> wrote:
Kafka itself is not multi-tennant. Neither messages nor topics are owned authorised entities. So although it has different retention policies for messages, they're not associated with users or tenants. Rate limiting / quotas of connected clients are features of Kafka itself, rather than being something Strimzi adds. Therefore it might be hard to separate it as a general concept.

Perhaps I misunderstood but isn't one goal of Strimzi to enable Kafka-aaS.


Strimzi provides operators which make it easy for different users to create their own Kafka clusters, (plus topics, users etc within those clusters). It does not provide support for a single Kafka cluster supporting multiple tenants. Obviously it is possible provide abstractions to do that on top of Kafka, but that's not what Strimzi provides.

Yes but what happens when there are two or more clusters?  Does Strimzi support creation of multiple Kafka clusters in one kubernetes cluster?  If so, then those clusters may require different user policies (even if Strimzi doesn't support that yet).

A






Regards,

Tom
Tom Bentley
 



On Thu, May 2, 2019 at 2:48 PM Alexis Richardson <alexis@...> wrote:

Yes but what happens when there are two or more clusters?  Does Strimzi support creation of multiple Kafka clusters in one kubernetes cluster?

Yes, of course.
 
  If so, then those clusters may require different user policies (even if Strimzi doesn't support that yet).


Each Kafka cluster has a distinct set of users, topics, policies, etc. Nothing would be shared.

Kind regards,

Tom
alexis richardson
 

Tom,




On Thu, 2 May 2019, 09:59 Tom Bentley, <tbentley@...> wrote:


On Thu, May 2, 2019 at 2:48 PM Alexis Richardson <alexis@...> wrote:

Yes but what happens when there are two or more clusters?  Does Strimzi support creation of multiple Kafka clusters in one kubernetes cluster?

Yes, of course.

Excellent

Think of these clusters as "tenants".


 
  If so, then those clusters may require different user policies (even if Strimzi doesn't support that yet).


Each Kafka cluster has a distinct set of users, topics, policies, etc. Nothing would be shared.

Super 

Now imagine traffic flowing into the Kube cluster.  If the kafka clusters become overloaded, is there a way to share traffic fairly, without soliciting more Kubernetes cluster resources.



Kind regards,

Tom
1 - 10 of 10