Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)


Rajeev
 


Hi, 

   We are using amazon Elastic Kubernetes Service with no dedicated nodes to deploy our stateless services. Now i want to use Falco for runtime container security. But as EKS spins ephemeral pods on dynamic nodes, i was not able to install Falco. 

Some developers are suggesting to run it as sidecar container in each pod so that it listens from there and logs which can be transferred to Amazon CloudWatch.

Can someone point me to docker image of falco to run as side car container. also share if there are any published resources available to deploy falco into EKS


Thanks,
Rajeev


Dan Miles (UK)
 

Hi Rajeev,

The correct pattern here is not a sidecar but a daemonset. 

We do something similar with GKE and ephemeral nodes, a daemonset will ensure that Falco is available across each node as they come up. 

A sidecar really isn’t the pattern here

-Dan


On Mon, 22 Feb 2021 at 13:25, Rajeev <rajeev@...> wrote:

Hi, 

   We are using amazon Elastic Kubernetes Service with no dedicated nodes to deploy our stateless services. Now i want to use Falco for runtime container security. But as EKS spins ephemeral pods on dynamic nodes, i was not able to install Falco. 

Some developers are suggesting to run it as sidecar container in each pod so that it listens from there and logs which can be transferred to Amazon CloudWatch.

Can someone point me to docker image of falco to run as side car container. also share if there are any published resources available to deploy falco into EKS


Thanks,
Rajeev


-------------------- End of message text --------------------
We're working with our industry experts to provide businesses with the information they need to respond to COVID-19. Subscribe here to receive our latest insights and podcast episodes straight to your inbox.
----------------------------------------------------------------
This email is confidential and is intended for the addressee only. If you are not the addressee, please delete the email and do not use it in any way.
PricewaterhouseCoopers LLP accepts no liability for any use of or reliance on this email by anyone, other than the intended addressee to the extent agreed in the relevant contract for the matter to which this email relates (if any).
PricewaterhouseCoopers LLP is a limited liability partnership registered in England under registered number OC303525, with its registered address at 1 Embankment Place, London, WC2N 6RH. It is authorised and regulated by the Financial Conduct Authority for designated investment business and by the Solicitors Regulation Authority for regulated legal activities. For security purposes and other lawful business purposes, PwC monitors outgoing and incoming emails and may monitor other telecommunications on its email and telecommunications systems.
----------------------------------------------------------------
Visit our website http://www.pwc.com/uk and see our privacy statement for details of why and how we use personal data and your rights (including your right to object and to stop receiving direct marketing from us).
----------------------------------------------------------------


Rajeev
 


It is not reporting logs. not sure if i missed any important step.

Could you please point me to any reference document which i can replicate and validate working setup 
or
 throw some pointers like installing as daemon and validating installation.




From: Dan Miles (UK) <daniel.miles@...>
Sent: 22 February 2021 19:54
To: Rajeev <rajeev@...>
Cc: cncf-falco-dev@... <cncf-falco-dev@...>
Subject: Re: [cncf-falco-dev] Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)
 
Hi Rajeev,

The correct pattern here is not a sidecar but a daemonset. 

We do something similar with GKE and ephemeral nodes, a daemonset will ensure that Falco is available across each node as they come up. 

A sidecar really isn’t the pattern here

-Dan


On Mon, 22 Feb 2021 at 13:25, Rajeev <rajeev@...> wrote:

Hi, 

   We are using amazon Elastic Kubernetes Service with no dedicated nodes to deploy our stateless services. Now i want to use Falco for runtime container security. But as EKS spins ephemeral pods on dynamic nodes, i was not able to install Falco. 

Some developers are suggesting to run it as sidecar container in each pod so that it listens from there and logs which can be transferred to Amazon CloudWatch.

Can someone point me to docker image of falco to run as side car container. also share if there are any published resources available to deploy falco into EKS


Thanks,
Rajeev


-------------------- End of message text --------------------
We're working with our industry experts to provide businesses with the information they need to respond to COVID-19. Subscribe here to receive our latest insights and podcast episodes straight to your inbox.
----------------------------------------------------------------
This email is confidential and is intended for the addressee only. If you are not the addressee, please delete the email and do not use it in any way.
PricewaterhouseCoopers LLP accepts no liability for any use of or reliance on this email by anyone, other than the intended addressee to the extent agreed in the relevant contract for the matter to which this email relates (if any).
PricewaterhouseCoopers LLP is a limited liability partnership registered in England under registered number OC303525, with its registered address at 1 Embankment Place, London, WC2N 6RH. It is authorised and regulated by the Financial Conduct Authority for designated investment business and by the Solicitors Regulation Authority for regulated legal activities. For security purposes and other lawful business purposes, PwC monitors outgoing and incoming emails and may monitor other telecommunications on its email and telecommunications systems.
----------------------------------------------------------------
Visit our website http://www.pwc.com/uk and see our privacy statement for details of why and how we use personal data and your rights (including your right to object and to stop receiving direct marketing from us).
----------------------------------------------------------------


 

Hi Rajeev,

Can you show the output of several commands for us?

helm list

kubectl get ds -o wide

kubectl get node

kubectl get pod -o wide | grep falco

If you can find a falco pod you can run `kubectl logs <pod>` directly to see if it's generating anything on stdout. And you can create synthetic events as mentioned in that blog post.

Hope this helps!
Spencer

On Mon, Feb 22, 2021, at 10:13 AM, Rajeev wrote:


It is not reporting logs. not sure if i missed any important step.

Could you please point me to any reference document which i can replicate and validate working setup 
or
 throw some pointers like installing as daemon and validating installation.






From: Dan Miles (UK) <daniel.miles@...>
Sent: 22 February 2021 19:54
To: Rajeev <rajeev@...>
Cc: cncf-falco-dev@... <cncf-falco-dev@...>
Subject: Re: [cncf-falco-dev] Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)
 
Hi Rajeev,

The correct pattern here is not a sidecar but a daemonset. 

We do something similar with GKE and ephemeral nodes, a daemonset will ensure that Falco is available across each node as they come up. 

A sidecar really isn’t the pattern here

-Dan


On Mon, 22 Feb 2021 at 13:25, Rajeev <rajeev@...> wrote:

Hi, 

   We are using amazon Elastic Kubernetes Service with no dedicated nodes to deploy our stateless services. Now i want to use Falco for runtime container security. But as EKS spins ephemeral pods on dynamic nodes, i was not able to install Falco. 

Some developers are suggesting to run it as sidecar container in each pod so that it listens from there and logs which can be transferred to Amazon CloudWatch.

Can someone point me to docker image of falco to run as side car container. also share if there are any published resources available to deploy falco into EKS


Thanks,
Rajeev




-------------------- End of message text --------------------
We're working with our industry experts to provide businesses with the information they need to respond to COVID-19. Subscribe here to receive our latest insights and podcast episodes straight to your inbox.
----------------------------------------------------------------
This email is confidential and is intended for the addressee only. If you are not the addressee, please delete the email and do not use it in any way.
PricewaterhouseCoopers LLP accepts no liability for any use of or reliance on this email by anyone, other than the intended addressee to the extent agreed in the relevant contract for the matter to which this email relates (if any).
PricewaterhouseCoopers LLP is a limited liability partnership registered in England under registered number OC303525, with its registered address at 1 Embankment Place, London, WC2N 6RH. It is authorised and regulated by the Financial Conduct Authority for designated investment business and by the Solicitors Regulation Authority for regulated legal activities. For security purposes and other lawful business purposes, PwC monitors outgoing and incoming emails and may monitor other telecommunications on its email and telecommunications systems.
----------------------------------------------------------------
Visit our website http://www.pwc.com/uk and see our privacy statement for details of why and how we use personal data and your rights (including your right to object and to stop receiving direct marketing from us).
----------------------------------------------------------------

-- 
  Spencer Krum



Rajeev
 

it is showing pods properly. 

Generated the activity like below:
touch /etc/2
cat /etc/shadow > /dev/null 2>&1


I don't see any new logs on Falco pods. Falco pods are running in default namespace.  Even i am trying to generate is on different namespace. Will there be any affect due to this?

How can i validate if daemon is running on any node?



From: cncf-falco-dev@... <cncf-falco-dev@...> on behalf of Spencer Krum via lists.cncf.io <nibz=spencerkrum.com@...>
Sent: 22 February 2021 22:02
To: cncf-falco-dev@... <cncf-falco-dev@...>
Subject: Re: [cncf-falco-dev] Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)
 
Hi Rajeev,

Can you show the output of several commands for us?

helm list

kubectl get ds -o wide

kubectl get node

kubectl get pod -o wide | grep falco

If you can find a falco pod you can run `kubectl logs <pod>` directly to see if it's generating anything on stdout. And you can create synthetic events as mentioned in that blog post.

Hope this helps!
Spencer

On Mon, Feb 22, 2021, at 10:13 AM, Rajeev wrote:


It is not reporting logs. not sure if i missed any important step.

Could you please point me to any reference document which i can replicate and validate working setup 
or
 throw some pointers like installing as daemon and validating installation.






From: Dan Miles (UK) <daniel.miles@...>
Sent: 22 February 2021 19:54
To: Rajeev <rajeev@...>
Cc: cncf-falco-dev@... <cncf-falco-dev@...>
Subject: Re: [cncf-falco-dev] Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)
 
Hi Rajeev,

The correct pattern here is not a sidecar but a daemonset. 

We do something similar with GKE and ephemeral nodes, a daemonset will ensure that Falco is available across each node as they come up. 

A sidecar really isn’t the pattern here

-Dan


On Mon, 22 Feb 2021 at 13:25, Rajeev <rajeev@...> wrote:

Hi, 

   We are using amazon Elastic Kubernetes Service with no dedicated nodes to deploy our stateless services. Now i want to use Falco for runtime container security. But as EKS spins ephemeral pods on dynamic nodes, i was not able to install Falco. 

Some developers are suggesting to run it as sidecar container in each pod so that it listens from there and logs which can be transferred to Amazon CloudWatch.

Can someone point me to docker image of falco to run as side car container. also share if there are any published resources available to deploy falco into EKS


Thanks,
Rajeev




-------------------- End of message text --------------------
We're working with our industry experts to provide businesses with the information they need to respond to COVID-19. Subscribe here to receive our latest insights and podcast episodes straight to your inbox.
----------------------------------------------------------------
This email is confidential and is intended for the addressee only. If you are not the addressee, please delete the email and do not use it in any way.
PricewaterhouseCoopers LLP accepts no liability for any use of or reliance on this email by anyone, other than the intended addressee to the extent agreed in the relevant contract for the matter to which this email relates (if any).
PricewaterhouseCoopers LLP is a limited liability partnership registered in England under registered number OC303525, with its registered address at 1 Embankment Place, London, WC2N 6RH. It is authorised and regulated by the Financial Conduct Authority for designated investment business and by the Solicitors Regulation Authority for regulated legal activities. For security purposes and other lawful business purposes, PwC monitors outgoing and incoming emails and may monitor other telecommunications on its email and telecommunications systems.
----------------------------------------------------------------
Visit our website http://www.pwc.com/uk and see our privacy statement for details of why and how we use personal data and your rights (including your right to object and to stop receiving direct marketing from us).
----------------------------------------------------------------

-- 
  Spencer Krum



 

Hi Rajeev,

It looks from your log output that you have successfully installed falco. You need to check for logs on the same node that you are doing the activity, so verify that (`kubectl get pod -o wide` helps). I do see a lot of drops but you should be seeing events.

Cheers,
Spencer

On Mon, Feb 22, 2021, at 10:44 AM, Rajeev wrote:
it is showing pods properly. 

Generated the activity like below:
touch /etc/2
cat /etc/shadow > /dev/null 2>&1

I don't see any new logs on Falco pods. Falco pods are running in default namespace.  Even i am trying to generate is on different namespace. Will there be any affect due to this?

How can i validate if daemon is running on any node?





From: cncf-falco-dev@... <cncf-falco-dev@...> on behalf of Spencer Krum via lists.cncf.io <nibz=spencerkrum.com@...>
Sent: 22 February 2021 22:02
To: cncf-falco-dev@... <cncf-falco-dev@...>
Subject: Re: [cncf-falco-dev] Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)
 
Hi Rajeev,

Can you show the output of several commands for us?

helm list

kubectl get ds -o wide

kubectl get node

kubectl get pod -o wide | grep falco

If you can find a falco pod you can run `kubectl logs <pod>` directly to see if it's generating anything on stdout. And you can create synthetic events as mentioned in that blog post.

Hope this helps!
Spencer

On Mon, Feb 22, 2021, at 10:13 AM, Rajeev wrote:


It is not reporting logs. not sure if i missed any important step.

Could you please point me to any reference document which i can replicate and validate working setup 
or
 throw some pointers like installing as daemon and validating installation.






From: Dan Miles (UK) <daniel.miles@...>
Sent: 22 February 2021 19:54
To: Rajeev <rajeev@...>
Cc: cncf-falco-dev@... <cncf-falco-dev@...>
Subject: Re: [cncf-falco-dev] Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)
 
Hi Rajeev,

The correct pattern here is not a sidecar but a daemonset. 

We do something similar with GKE and ephemeral nodes, a daemonset will ensure that Falco is available across each node as they come up. 

A sidecar really isn’t the pattern here

-Dan


On Mon, 22 Feb 2021 at 13:25, Rajeev <rajeev@...> wrote:

Hi, 

   We are using amazon Elastic Kubernetes Service with no dedicated nodes to deploy our stateless services. Now i want to use Falco for runtime container security. But as EKS spins ephemeral pods on dynamic nodes, i was not able to install Falco. 

Some developers are suggesting to run it as sidecar container in each pod so that it listens from there and logs which can be transferred to Amazon CloudWatch.

Can someone point me to docker image of falco to run as side car container. also share if there are any published resources available to deploy falco into EKS


Thanks,
Rajeev




-------------------- End of message text --------------------
We're working with our industry experts to provide businesses with the information they need to respond to COVID-19. Subscribe here to receive our latest insights and podcast episodes straight to your inbox.
----------------------------------------------------------------
This email is confidential and is intended for the addressee only. If you are not the addressee, please delete the email and do not use it in any way.
PricewaterhouseCoopers LLP accepts no liability for any use of or reliance on this email by anyone, other than the intended addressee to the extent agreed in the relevant contract for the matter to which this email relates (if any).
PricewaterhouseCoopers LLP is a limited liability partnership registered in England under registered number OC303525, with its registered address at 1 Embankment Place, London, WC2N 6RH. It is authorised and regulated by the Financial Conduct Authority for designated investment business and by the Solicitors Regulation Authority for regulated legal activities. For security purposes and other lawful business purposes, PwC monitors outgoing and incoming emails and may monitor other telecommunications on its email and telecommunications systems.
----------------------------------------------------------------
Visit our website http://www.pwc.com/uk and see our privacy statement for details of why and how we use personal data and your rights (including your right to object and to stop receiving direct marketing from us).
----------------------------------------------------------------


-- 
  Spencer Krum



Attachments:
  • Screenshot 2021-02-22 at 10.09.47 PM.png
  • Screenshot 2021-02-22 at 10.10.09 PM.png
  • Screenshot 2021-02-22 at 10.11.29 PM.png

-- 
  Spencer Krum



Rajeev
 

I tried to create event on a specific pod and check its logs. i don't see any audit logs generated.

what exactly i did:
*) selected a specific pod in my containers and did tail of running logs with below command: (in this pod, along with application container name explorer, there are side car containers running to push logs to cloud watch. one is fluentd and other is log rotate)


kubectl logs -f explorer-d4c8cbc76-trxwt -n bhadra-graph-beta -c explorer


*) now i logged into application container terminal and generated event 

touch /etc/2
cat /etc/shadow > /dev/null 2>&1



I do not see any logs in container logs. am i missing anything?


let me know if i can provide any additional information for you to understand this issue.



From: Spencer Krum <nibz@...>
Sent: 22 February 2021 22:39
To: Rajeev <rajeev@...>; cncf-falco-dev@... <cncf-falco-dev@...>
Subject: Re: [cncf-falco-dev] Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)
 
Hi Rajeev,

It looks from your log output that you have successfully installed falco. You need to check for logs on the same node that you are doing the activity, so verify that (`kubectl get pod -o wide` helps). I do see a lot of drops but you should be seeing events.

Cheers,
Spencer

On Mon, Feb 22, 2021, at 10:44 AM, Rajeev wrote:
it is showing pods properly. 

Generated the activity like below:
touch /etc/2
cat /etc/shadow > /dev/null 2>&1

I don't see any new logs on Falco pods. Falco pods are running in default namespace.  Even i am trying to generate is on different namespace. Will there be any affect due to this?

How can i validate if daemon is running on any node?





From: cncf-falco-dev@... <cncf-falco-dev@...> on behalf of Spencer Krum via lists.cncf.io <nibz=spencerkrum.com@...>
Sent: 22 February 2021 22:02
To: cncf-falco-dev@... <cncf-falco-dev@...>
Subject: Re: [cncf-falco-dev] Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)
 
Hi Rajeev,

Can you show the output of several commands for us?

helm list

kubectl get ds -o wide

kubectl get node

kubectl get pod -o wide | grep falco

If you can find a falco pod you can run `kubectl logs <pod>` directly to see if it's generating anything on stdout. And you can create synthetic events as mentioned in that blog post.

Hope this helps!
Spencer

On Mon, Feb 22, 2021, at 10:13 AM, Rajeev wrote:


It is not reporting logs. not sure if i missed any important step.

Could you please point me to any reference document which i can replicate and validate working setup 
or
 throw some pointers like installing as daemon and validating installation.






From: Dan Miles (UK) <daniel.miles@...>
Sent: 22 February 2021 19:54
To: Rajeev <rajeev@...>
Cc: cncf-falco-dev@... <cncf-falco-dev@...>
Subject: Re: [cncf-falco-dev] Run Falco as sidecar container in Amazon Elastic Kubernetes service(EKS)
 
Hi Rajeev,

The correct pattern here is not a sidecar but a daemonset. 

We do something similar with GKE and ephemeral nodes, a daemonset will ensure that Falco is available across each node as they come up. 

A sidecar really isn’t the pattern here

-Dan


On Mon, 22 Feb 2021 at 13:25, Rajeev <rajeev@...> wrote:

Hi, 

   We are using amazon Elastic Kubernetes Service with no dedicated nodes to deploy our stateless services. Now i want to use Falco for runtime container security. But as EKS spins ephemeral pods on dynamic nodes, i was not able to install Falco. 

Some developers are suggesting to run it as sidecar container in each pod so that it listens from there and logs which can be transferred to Amazon CloudWatch.

Can someone point me to docker image of falco to run as side car container. also share if there are any published resources available to deploy falco into EKS


Thanks,
Rajeev




-------------------- End of message text --------------------
We're working with our industry experts to provide businesses with the information they need to respond to COVID-19. Subscribe here to receive our latest insights and podcast episodes straight to your inbox.
----------------------------------------------------------------
This email is confidential and is intended for the addressee only. If you are not the addressee, please delete the email and do not use it in any way.
PricewaterhouseCoopers LLP accepts no liability for any use of or reliance on this email by anyone, other than the intended addressee to the extent agreed in the relevant contract for the matter to which this email relates (if any).
PricewaterhouseCoopers LLP is a limited liability partnership registered in England under registered number OC303525, with its registered address at 1 Embankment Place, London, WC2N 6RH. It is authorised and regulated by the Financial Conduct Authority for designated investment business and by the Solicitors Regulation Authority for regulated legal activities. For security purposes and other lawful business purposes, PwC monitors outgoing and incoming emails and may monitor other telecommunications on its email and telecommunications systems.
----------------------------------------------------------------
Visit our website http://www.pwc.com/uk and see our privacy statement for details of why and how we use personal data and your rights (including your right to object and to stop receiving direct marketing from us).
----------------------------------------------------------------


-- 
  Spencer Krum



Attachments:
  • Screenshot 2021-02-22 at 10.09.47 PM.png
  • Screenshot 2021-02-22 at 10.10.09 PM.png
  • Screenshot 2021-02-22 at 10.11.29 PM.png

-- 
  Spencer Krum