Dan Munro - Dev-Sec-Ops Engineer   About

Practical Kubernetes Security

Kubernetes is a fascinating software development that has rapidly revolutionized the container and ops spaces. However, with quick evolution comes new attack vectors.

This article hopes to highlight some important Kubernetes security best practices, including hardening default options, isolation at the network level, and tools for auditing security settings in deployed clusters. Reference links and documentation will be added for further reading.

Some advice is specific to AWS, however this guide aims to be hosting agnostic as much as possible.

A huge thanks to everyone who has documented Kubernetes extensively already. The official docs linked in these tips, and Kubernetes Security Best Practices made this article possible.

API Server Settings

As the only component in a Kubernetes cluster to connect to etcd, as well as being a central control point for worker nodes, the api server is a prime target for attackers.

Authorization Mode

Per this authorization overview, avoid using the AlwaysAllow authorization mode for a cluster. This mode grants any authenticated entity full access to the cluster. With this level of access, an attacker will have the ability to launch new pods within the cluster – a fairly straightforward method for privilege escalation.

Users who have the ability to create pods in a namespace can potentially escalate their privileges within that namespace. They can create pods that access their privileges within that namespace. They can create pods that access secrets the user cannot themselves read, or that run under a service account with different/greater permissions. (read more)

Instead, RBAC should be used. RBAC allows policy storage and retrieval natively via the Kubernetes API. Roles may be applied to users, groups, or service accounts, and a number of roles are defined out of the box. Pod creation is reserved for special roles, which should be restricted.

Check the --authorization-mode parameter of your kube-apiserver processes. RBAC is the desired option.

Bitnami provides an excellent guide for setting up RBAC.

Insecure Port

Deprecated since v1.10, this option really should be disabled. If enabled, a port is open which provides no authorization. No TLS, no auth, no authz.

Disable with --insecure-port=0.

Kubernetes at a high level

Kubelet Settings

Ensure kubelet’s port (10250/tcp) is behind a firewall. Kubelet offers an api for arbitrary command execution on worker nodes, which is utilized by the API server.

Restrict “docker image pull”

Owing to the fact that docker images are uncontrolled environments, pulling an unauthorized image is a convenient way to gain arbitrary code execution within a cluster, among other nasty possibilities.

Use the docker image policy plugin to restrict this attack vector.

Disallow Dashboard Access From Inside The Cluster

Depending on the Kubernetes version, kubernetes-dashboard uses a service account with full cluster access. Access to the dashboard from within the cluster can be disallowed without affecting kubectl proxy. Read this section for more information and to get a NetworkPolicy that will guard against this exploit.

Kube-bench

Kube-bench is a tool for auditing deployed Kubernetes clusters. Some configuration is required, and the readme is well worth the time. There are a large number of configuration checks across different versions of Kubernetes.

Firewall Ports

Following this advice, firewall all ports at the network level that aren’t needed by platform services.

Bastion Host

Disallow direct ssh access to Kubernetes nodes. Instead expose one edge node, which serves as a proxy between the internet and the private network. This is a small detail but has huge advantages. Auditing and logging become much easier concerns to monitor. And in the case of a breach, there is one location to disable ssh.

AWS bastion host setup within a VPC

The bastion host should still be protected by ip rules, such as security groups in AWS. Here’s a good guide for AWS bastion host setup (not for kubernetes but the concept is the same).

Kube2IAM

Kubernetes running directly in EC2 has no native way to represent IAM roles on individual running pods within a node. Instead, roles must be applied to the nodes, which is suboptimal from a security perspective. Ideally, we can follow the principle of least privilege and apply roles on a per-container or per-pod basis.

Kube2IAM does just that. By creating a sidecar container that serves as a proxy to the AWS metadata api, the sidecar then authorizes requests to that api via configured roles, which serves as an effective work-around.

CVEs

This past March, two of the most critical CVEs for Kubernetes to date were reported. Both CVEs involve hacks which allow an attacker to break the normal filesystem isolation afforded by containerization.

The first hack requires shell access inside the container, after which arbitrary root access to the host filesystem can be gained. The second allows arbitrary host path deletion via a few different vectors within the Kubernetes platform.

Fixes for both issues have been deployed and backported as far back as 1.7. These CVEs highlight the importance of software updates and automated vulnerability scanning.



Image attributions:

X-Team

AWS ssh via bastion host

Written on August 18, 2018