Minimum Viable Security for a Kubernetised Webapp: TLS everywhere - Part1

Minimum Viable Security (MVSec) is a concept borrowed from the Minimum Viable Product (MVP) concept about the Product Development Strategy and from the Pareto Principle or 80/20 rule. The MVP concept applied to IT Security means the product (application) will contain only the minimum amount (20%) of effort invested in order to prove the viability (80%) of an idea (acceptable security).

The purpose of this post is to explain how to implement TLS everywhere to become MVSec (roughly 80% of security with 20% of working) for a Kubernetised Webapp hosted on AWS.

Minimum Viable Security for a Kubernetised Webapp: TLS everywhere with NGINX Ingress Controller, Cert-Manager and Let's Encrypt

Why TLS everywhere meets 80/20 rule?

Part of the answer gives us Vilfredo Pareto with the Pareto Principle, but to have a complete answer we have to turn to The Security Design Principles that NIST, OWASP, NCSC and other Security References give us.

The Security Design Principles to consider are:

There are many similarities between them at fundamental level, let’s explore the OWASP for example and do a quick analysis to see how TLS can help us to meet these Security Principles.

# OWASP Security Design Principles Explanation How to TLS help us?
1) Defense in Depth. Also known as layered defense. TLS over HTTP provides a new secure layer.
2) Fail Safe. Unless a subject is given explicit access to an object, it should be denied access to that object. TLS Certificates can encrypt at rest and in transit these objects.
3) Least Privilege. A Process is given only the minimum level of access rights (privileges) that is necessary for that Process to complete an assigned operation. Transit of sensitive objects must be over TLS.
4) Separation of Duties. Also known as the compartmentalization principle, or separation of privilege. HTTPS is for sensitive transactions and HTTP for non-sensitive.
5) Economy of Mechanism. Also known as Keep It Simple, Stupid principle. The accepted and supported HTTP/2 includes TLS by default. So, if HTTP is insecure, why is it still being used?.
6) Complete Mediation. All accesses to objects must be checked to ensure that they are allowed. TLS provides Simple and Mutual (Two-Way) Authentication and allways once crypto keys have been verified a secure communication is stablished.
7) Open Design. The security of a mechanism should not depend on the secrecy of its design or implementation. The disclosure of TLS’ using in software designs don’t put in risk the application.
8) Least Common Mechanism It disallows the sharing of mechanisms that are common to more than one user or process if the users and processes are at different levels of privilege. TLS is a common secure mechanism shared along the application.
9) Psychological acceptability. Tries maximizing the usage and adoption of the security functionality in the application by making it more usable. TLS is a feature by default in HTTP/2. HTTP requires implement TLS and deal the crypto keys.
10) Weakest Link. The resiliency of your software against hacker attempts will depend heavily on the protection of its weakest components. TLS helps to harden the access to resources and secure the traffic.
11) Leveraging Existing Components. It focuses on ensuring that the attack surface is not increased. TLS is a feature by default in HTTP/2 for securing data in transit and data at rest, don’t try to bring or implement your own TLS.

If this quick analysis does not convince you, then refer to the Pillars of Security, they are axioms and don’t require be demostration. Organization like OWASP recommends that all security controls should be designed with the Core pillars of Information Security in mind:

# Pillar of Security Description How TLS apply?
1. Confidentiality Only allow access to data for which the user is permitted. Access Control(*) using TLS and Mutual TLS Authn.
2. Integrity Ensure data is not tampered or altered by unauthorised users. Data no altered using TLS(*) encryption at rest.
3. Availability Ensure systems and data are available to authorised users when they need it. Mutual TLS Authn(*) helps to availability only to authorised users.

(*) These Axioms refer to access control, data no altered, unauthorised users, etc. and that is implemented following the Identity-based Security Strategy.

Let’s implement TLS everywhere in Kubernetes.

Create an Kubernetes Cluster

This blog post will use the Building your own affordable K8s - Serie:

1) Clone the Affordable K8s Cluster Git Repo and run the Terraform scripts

I’ll create a Kubernetes Cluster with this configuration:

  • NGINX Ingress Controller will be deployed as DaemonSet (in the namespace ingress-nginx) with hostNetwork: true to ensure the NGINX server is reachable from all K8s Cluster nodes.
    • When a pod is configured with hostNetwork: true, the applications running in such a pod can directly see the network interfaces of the host machine where the pod was started.
    • Further information in the nginx-ingress-mandatory.yaml deployment file.
  • Custom Domain Name System (DNS) is cloud.holisticsecurity.io and the Ingress Subdomain DNS is ingress-nginx.cloud.holisticsecurity.io.
  • TLS Cert enabled using Jetstack Cert-Manager and Let’s Encrypt.
  • NodePort Service for the NGINX Ingress Controller. See its configuration here.
$ git clone https://github.com/chilcano/affordable-k8s
$ cd affordable-k8s

$ terraform init

$ terraform plan \
  -var cluster_name="cheapk8s" \
  -var k8s_ssh_key="ssh-key-for-us-east-1" \
  -var admin_cidr_blocks="<YOUR-IP-ADDRESS>/32" \
  -var region="us-east-1" \
  -var kubernetes_version="1.14.3" \
  -var external_dns_enabled="1" \
  -var nginx_ingress_enabled="1" \
  -var nginx_ingress_domain="ingress-nginx.cloud.holisticsecurity.io" \
  -var cert_manager_enabled="1" \
  -var cert_manager_email="cheapk8s@holisticsecurity.io"

$ terraform apply \
  -var cluster_name="cheapk8s" \
  -var k8s_ssh_key="ssh-key-for-us-east-1" \
  -var admin_cidr_blocks="<YOUR-IP-ADDRESS>/32" \
  -var region="us-east-1" \
  -var kubernetes_version="1.14.3" \
  -var external_dns_enabled="1" \
  -var nginx_ingress_enabled="1" \
  -var nginx_ingress_domain="ingress-nginx.cloud.holisticsecurity.io" \
  -var cert_manager_enabled="1" \
  -var cert_manager_email="cheapk8s@holisticsecurity.io"

2) Checking the K8s Cluster, NGINX Ingress Controller, Cert-Manager are running correctly

// SSH to the cluster
$ ssh ubuntu@$(terraform output master_dns) -i ~/Downloads/ssh-key-for-us-east-1.pem

Checking NGINX Ingress Controller:

// Likely you have to wait a 2-3 minutes till the Cluster is running completely.
$ kubectl get pod,svc -n ingress-nginx -o wide
NAME                                        READY   STATUS    RESTARTS   AGE
pod/default-http-backend-5c9bb94849-9lpn9   1/1     Running   0          7m35s
pod/nginx-ingress-controller-r2j8t          1/1     Running   0          7m34s
pod/nginx-ingress-controller-v22mp          1/1     Running   0          7m34s

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/default-http-backend   ClusterIP   10.110.233.145   <none>        80/TCP                       7m36s
service/ingress-nginx          NodePort    10.111.197.179   <none>        80:30002/TCP,443:30414/TCP   7m33s

If everything is fine, you should get a default backend - 404 response when accessing the NGINX Ingress Controller using its NodePort and its external IP address. This tells us that the controller doesn’t know where to route the request to, so responds with the default backend.

// Calling it from inside the K8s Cluster, here you have to use the `NodePorts` (`30002` for HTTP and `30414` for HTTPS):
$ curl http://localhost:30002/abc
default backend - 404

$ curl https://localhost:30414/pqr -k
default backend - 404

// Calling from Internet, here you have to use `80` and `443` port, and the FQDN `ingress-nginx.cloud.holisticsecurity.io`
$ curl http://ingress-nginx.cloud.holisticsecurity.io/abc
default backend - 404

$ curl https://ingress-nginx.cloud.holisticsecurity.io/pqr -k
default backend - 404

Checking Cert-Manager:

// Cert-Manager logs
$ kubectl logs -f -n cert-manager -lapp=cert-manager

// Lets Encrypt Cert Issuer
$ kubectl get issuer,secret -n default

Deploying a Sample Application

I love Weave Scope, it is a good Web Application that I can use as example to enable/configure its security.

Weave Scope is a visualization and monitoring tool for Docker and Kubernetes. It provides a top down view into your app as well as your entire infrastructure, and allows you to diagnose any problems with your distributed containerized app, in real time, as it is being deployed to a cloud provider.

Installing Weave Scope

# Get ssh access and in your K8s Cluster, run below command to install Weave Scope with everything by default.
$ kubectl apply -f "https://cloud.weave.works/k8s/scope.yaml?k8s-version=$(kubectl version | base64 | tr -d '\n')"

# Check if Weave Scope has been installed successfully. You will see Pods, Service, ReplicaSet, DaemonSet and Deployment.
$ kubectl get all -n weave

# Get the Weave Scope's TargetPort. By default the TargetPort for Weave Scope's ClusterIP service is 4040.
$ kubectl get -n weave svc weave-scope-app -o jsonpath='{.spec.ports[0].targetPort}'
4040

Creating NodePort service for Weave Scope

Since ClusterIP is for internal use only, I’ll need that Weave Scope be exposed and reachable from Internet that I can make a SSH tunnel. I can do it by creating a new NodePort service, also I’ll create and register in AWS Route 53 a fqdn for Weave-Scope, in this case it will be weave-scope.cloud.holisticsecurity.io, although this fqdn isn’t required to make the SSH tunnel.

# Let's create a NodePort Resource for Weave Scope.
$ kubectl apply -f https://raw.githubusercontent.com/chilcano/affordable-k8s/master/examples/weave-scope-app-svc-np.yaml
service/weave-scope-app-svc-np created

# Now, we have 2 services
$ kubectl get svc -n weave
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
weave-scope-app        ClusterIP   10.100.187.247   <none>        80/TCP         45m
weave-scope-app-svc-np NodePort    10.102.182.243   <none>        80:30002/TCP   18m

# Get the Weave Scope's NodePort. Also you can see it in above command or in the 'sample-2-weave-scope-app-svc-np.yml' file.
$ kubectl get -n weave svc weave-scope-app-svc-np -o jsonpath='{.spec.ports[0].nodePort}'
30002

Creating a SSH tunnel to Weave Scope

Now let’s create a SSH tunnel from your Admin Computer over Internet to Weave Scope’s NodePort service.

$ ssh -nNT -L 4002:localhost:30002 ubuntu@$(terraform output master_dns) -i ~/Downloads/ssh-key-for-us-east-1.pem

Now open your favorite browser and enter this url http://localhost:4002 and you will be able to visualize all resources created in your Cluster in real-time.

[

Enabling and configuring Security based on TLS

Since I’m using the Affordable K8s’ Terraform scripts to build a K8s Cluster with the Jetstack Cert-Manager, to get, renew, revoke any kind of X.509 Certificates, and the NGINX Ingress Controller, to manage the traffic, now i would be able to improve security according the Minimum Viable Security (MVSec) and Pareto Principle or 80/20 rule both explained above.
In next posts I’ll explain how to:

  • Part 1 - Minimum Viable Security for a Kubernetised Webapp: TLS everywhere (this post)
  • [Part 2 - Enable and configure HTTP Basic Authentication over TLS in Weave Scope](/2020/03/19/minimum-viable-security-for-a-k8s-webapp-http-basic-auth-on-tls-part2)
  • Part 3 - Enable and configure Mutual TLS Authentication in Weave Scope (Hashicorp Vault as PKI and Secrets Mngmt will be integrated)
  • Part 4 - Integrating an IAM (Identity Access Management - see Security along Container-based SDLC - OSS Tools List) solution in a Kubernetised Webapp

Troubleshooting

  1. Getting Kubernetes installation logs: Access to Cluster via SSH and get the logs.
$ cat /var/log/cloud-init-output.log
  1. Getting NGINX Ingress Controller logs:
$ kubectl get pods -n ingress-nginx 
$ kubectl exec -it -n ingress-nginx nginx-ingress-controller-p5qz5 -- cat /etc/nginx/nginx.conf | grep ssl
$ kubectl logs -f -n ingress-nginx nginx-ingress-controller-p5qz5 | grep Error
$ kubectl logs -f -n ingress-nginx -lapp.kubernetes.io/name=ingress-nginx
$ kubectl logs -f -n ingress-nginx -lapp.kubernetes.io/part-of=ingress-nginx
  1. Getting Jetstack Cert-Manager logs:
$ kubectl get pods -n cert-manager
$ kubectl exec -it -n ingress-nginx cert-manager-54d94bb6fc-fmhcc -- cat /etc/nginx/nginx.conf | grep ssl
$ kubectl logs -f -n cert-manager cert-manager-54d94bb6fc-fmhcc 
$ kubectl logs -f -n cert-manager -lapp=cert-manager
$ kubectl logs -f -n cert-manager -lapp=cainjector
$ kubectl logs -f -n cert-manager -lapp=webhook