DevSecOps · July 24, 2023 0

Implementing zero-trust Microservices Security with Service Mesh

In this blog, We will discuss how to implement end-to-end security between microservices. I will cover mostly Istio service mesh,linkerd service mesh & a hint of tenzu.
If you want to reproduce the below labs you can use- https://github.com/c0ldb00t3r/k8s-poc/tree/main/service-mesh-demo.Set up a local k8s cluster easily with vagrant- https://github.com/c0ldb00t3r/k8s-vagrant
 

Istio- Istio is an open-source service mesh. It's also the default service mesh for many Kubernetes service providers like GCP, Microsoft, and IBM. It solves 3 major problems faced by large-scale microservices-

  • Traffic management
  • Security
  • Observability
Details of why use istio, benefits, etc- https://www.solo.io/blog/what-is-istio/
Here we will discuss implementing security using ISTIO.
Arch-
ENVOY is the default sidecar proxy in Istio. Envoy proxy is injected as a sidecar in each pod. That is how it intercepts all network communication between microservices, then configures and manages Istio policies using its control plane functionality. In the istiod control plane, Citadel handles certificates, service discovery (Pilot), and configuration (Galley).
 
Installation:
1. Download and setup istioctl
curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH

2. As istiod takes around 2 GB of ram, for the safer side I will be assigning a node using a node selector. For that first generate the yaml.

istioctl manifest generate --set profile=default > istio.yaml
What are configuration profiles you can read them here- https://istio.io/latest/docs/setup/additional-setup/config-profiles/
Note- if you don't have CPU usage issue you can simply install using istioctl install --set profile=default -y.
 
3. Edit the yaml and add nodeName: kworker2 in two of the deployment pod istiod and istio-ingress respectively & apply the yaml file.
update- Create a namespace istio-system before applying the yaml file.
4. Once installed you can see “Istiod” pod which is basically the control plane & generated data plane configurations using CRD, which are stored in etcd.
5. By default Envoy proxy sidecars were not injected. To enable that you have to specifically set the Namespace Label to “istio-injection=enabled”.There are two ways to do this -
  • using imperative command
And now when the service restarts it will automatically add Sidecar to all the associated pods.
  • Another way is adding a label in the deployment manifest file itself.
6. The most scalable way of enabling istio sidecar is setting up a validation webhook so that every time a new service is deployed it automatically checks if istio is enabled or not.
7. In the above examples, we have enabled and disabled injection at the namespace level. Injection can also be controlled on a per-pod basis, by configuring the sidecar.istio.io/inject label on a pod.
 
MTLS USING ISTIO
The istiod component handles all CA capabilities (handling CSR, signing, rotating). By default, the Istio CA generates a self-signed root certificate and key and uses them to sign the workload certificates. The root CA validity is 10 years. You can replace this certificate with your own CA(for example Google CAS etc.).

https://istio.io/latest/docs/tasks/security/cert-management/plugin-ca-cert/For a production cluster setup, it is highly recommended to use a production-ready CA, such as Hashicorp Vault. It is a good practice to manage the root CA on an offline machine with strong security protection.https://medium.com/everything-full-stack/how-to-integrate-vault-as-external-root-ca-with-cert-manager-istio-csr-and-istio-7684baa369db

Workload Certificate Rotation-
On the client side, a service within the istio-proxy (that is, running as a sidecar with the workload) is responsible for creating workload certificates and initiating the CSR process with istiod. The default workload certificate validity is for 24 hours. This can be changed on the workload/client side with an environment variable SECRET_TTL.
I have shamelessly copied the certificate rotation section from https://blog.christianposta.com/diving-into-istio-1-6-certificate-rotation/.
 
 
MODES-
-PERMISSIVE: Workloads accept both mutual TLS and plain text traffic. This mode is most useful during migrations when workloads without sidecar cannot use mutual TLS. Once workloads are migrated with sidecar injection, you should switch the mode to STRICT.
- STRICT: Workloads only accept mutual TLS traffic.
- DISABLE: Mutual TLS is disabled.
 
POC-
I have created 3 namespaces for POC. secured, lax, and legacy. Let's create 2 services a, and b, and deploy them in all namespaces. Let's send traffic from b to a from other namespaces. They are able to communicate with each of them like these - Now let's apply peer authentication mode. Secured has been applied with a strict peer authentication policy, and lax is applied with a permissive policy & as the name suggests lax doesn't have any authentication policy.
kubectl apply -n secured -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
spec:
  mtls:
    mode: STRICT
EOF
As you can see from the below screenshot only legacy is not able to communicate with STRICT mode but able to communicate with PERMISSIVE mode.
As you can see from the below screenshot communication is encrypted. Whereas legacy services can communicate with "lax" namespace. This is very helpful in a production environment where they are many legacy services and you want onboard them.
 
Using AUTHZ & AUTHN for verification
By default any service can communicate with each other in k8s even in the services are in different namespaces. You obviously would want to implement authentication and authorization for communication between one service with another as we used to do even 3-tier arch due to compliance 😉.

This upload microservice is created using chatgpt. let's upload a file from another service.For the demo, we will be using JWT claim-based authentication. for the prod environment, you can use external OIDC providers like Keycloak, google, okta, etc. https://venafi.com/blog/istio-oidc/

Let's apply authn and authz policy that this service only allows requests from signed with specific JWKS. Istio also constructs the requestPrincipal by combining the iss and sub of the JWT token.In the below screenshot, you can see that requests are forbidden without a valid jwt token & allow for signed valid jwt.

Controlling http Traffic using istio AUTHZ & Using this for incident responseFrom the above screenshot, you see it allows only get requests. You can use many selectors as you want depending on your use case where you want your traffic to flow from. https://istio.io/latest/docs/reference/config/security/authorization-policy/

Let's think of a critical production incident, a user is able to upload his profile pic and crash another user or stored xss or any malicious activity. To fix the issue it will require 2 days to fix. By the time how to contain the issue without affecting millions of live users? Now you know how just disable the vulnerable file upload service for 2 days or for the specific URI path field & users will be able to do everything except profile pic upload.

PII data leakage and OWASP threat detection
Although I don't promote paid solutions, I need to show what you can do with service mesh. Tanzu is saas solution from VMware, this is made above istio. With Tanzu customers can create PII data leakage policies to alert or block when data is accessed inappropriately. Tanzu Service Mesh can apply threat detection for the Open Web Application Security Project (OWASP) Top Ten web application security risks to east–west traffic. This means that every proxy in the service mesh can detect and stop an OWASP 10 attack. You can also program custom attacks by importing attack signatures. Unless you really need this feature, I would recommend using istio opensource. Let me know if you know of any other open-source solution that does the same.
 

It's turning out to be a big blog, let's continue on the linkerd on part2-

Spread the love