CVE-2018-1002105: The Kubernetes API Server Flaw and What to Check on Your Cluster

On Monday the Kubernetes project disclosed CVE-2018-1002105, and it’s the kind of bug that gets a cluster operator’s attention fast. It lives in the API server, the one component every request goes through, and the project’s own issue scores it 9.8 out of 10. If you run Kubernetes yourself, the short version is to upgrade. If you’re on a managed service, the providers say they’ve already handled the control plane, but the bug is a good excuse to look hard at who can do what in your cluster.

I’m sticking to what the project and the big providers published themselves: the Kubernetes issue and announcement, the Kubernetes docs, and bulletins from Google, Microsoft, AWS, and Red Hat.

What the announcement says

Jordan Liggitt posted the security announcement to kubernetes-announce on December 3 on behalf of the Kubernetes Product Security Team. It says v1.10.11, v1.11.5, and v1.12.3 were released to address a critical issue present in all previous versions of the Kubernetes API server, that the upcoming v1.13.0 also includes the fix, and that all clusters on earlier versions should update to one of those releases immediately. The report is credited to Darren Shepherd.

The details are in kubernetes/kubernetes issue #71411, which the project titled “proxy request handling in kube-apiserver can leave vulnerable TCP connections.” The API server doesn’t only answer requests itself. When you run kubectl exec, attach, or port-forward, the API server checks your permissions and then proxies the connection to the kubelet on the node running the pod. It does the same thing for aggregated API servers, the add-on APIs (the metrics server is the example the issue uses) that get plugged in behind the main API. In both cases, the API server talks to the backend with its own TLS credentials.

The flaw was in how the API server handled a failed attempt to upgrade one of those proxied connections. According to the issue, a user who is allowed to open a connection through the API server to a backend could, with a specially crafted request, keep sending arbitrary requests down that same connection straight to the backend. Those requests are authenticated as the API server, not as the user, and they never pass back through the API server’s authorization checks.

Who is actually exposed

The issue describes two configurations, and they’re worth reading separately because the second one goes all the way back to 1.0.

The first is clusters running aggregated API servers that the API server can reach directly over its network. The issue says this applies to clusters on 1.6 or later. In that setup, a call to any endpoint on an aggregated API server can be escalated into any request against that aggregated server. The part that makes this nasty is the default RBAC policy: the issue points out that the built in discovery bindings let every user, authenticated or not, make the discovery calls that open the door. So an anonymous user could get full access to whatever an aggregated API exposes.

The second is clusters that give pod exec, attach, or portforward permissions to people who aren’t supposed to have full access to the kubelet API. Here a user who can exec into one pod can escalate to any request against the kubelet on that pod’s node. The issue’s examples are listing all pods on the node and running commands inside them. And it calls out that exec, attach, and portforward are part of the admin and edit roles, which are the roles meant for users confined to a namespace. If you’ve handed a team edit on their own namespace, that team could reach other workloads that happen to share a node with theirs.

One more line from the issue deserves attention: there’s no simple way to tell if this was used. The escalated requests travel over an already established connection, so they don’t show up in the API server’s audit log or server log. They do show up in kubelet or aggregated API server logs, but they look exactly like properly authorized proxied requests.

What the managed providers said

If you don’t run your own control plane, the upgrade is your provider’s job, and all four big ones posted something.

Google’s GKE security bulletin dated December 3 says all GKE masters were affected, that Google has already upgraded them, and that no action is required. It lists patched GKE versions such as 1.10.9-gke.5 and 1.11.2-gke.18. Google rates it High rather than Critical for GKE, because details of its implementation prevented the unauthenticated escalation path.

Microsoft’s Azure update for AKS, also posted December 3, describes the exposure in AKS in terms of the metrics server API, which the 1.10 and 1.11 releases in AKS include. It says AKS had already patched affected clusters before the announcement by overriding the default configuration to remove unauthenticated access to everything under /apis/. If something outside your cluster relied on reaching those paths without credentials, Microsoft says you’ll need to switch to an authenticated path. It also made 1.11.5 available for anyone who wants the upstream fix, through az aks upgrade.

AWS published security bulletin AWS-2018-020 on December 5. It says the EKS fleet has been patched, every EKS cluster is running a version that isn’t affected, and no customer action is required.

Red Hat’s vulnerability article for OpenShift rates it Critical and says all 3.x versions of OpenShift Container Platform are affected through the pod exec, attach, and portforward path, with 3.6 and later also exposed through aggregated APIs, including the service catalog. It lists the minimum fixed build for each 3.x line, from v3.2.1.34-2 up to v3.11.43-1, and says OpenShift Online and OpenShift Dedicated are impacted too.

GKE’s patched builds include a 1.10.9, which is below the upstream fix of 1.10.11, and OpenShift has its own numbering. The Kubernetes issue itself says that if you get binaries from a distributor, you should ask them which versions resolve this. So comparing your version string to the upstream list only works if you’re running upstream builds.

Check your version

The bug is in the API server, so the version that matters is the server’s, not your laptop’s kubectl and not necessarily the kubelets’. This prints the client and server versions:

kubectl version --short

Look at the Server Version line. On upstream Kubernetes, you want v1.10.11, v1.11.5, v1.12.3, or newer on those lines. Anything on 1.9 or older isn’t getting a fix from the project, so that cluster needs a minor version upgrade. On a managed service or a distribution, check the number against that vendor’s bulletin instead.

Find your aggregated API servers

The issue includes a command that lists the APIService objects backed by a service, which is to say your aggregated API servers:

kubectl get apiservices \
  -o 'jsonpath={range .items[?(@.spec.service.name!="")]}{.metadata.name}{"\n"}{end}'

An empty result, or an older API server that doesn’t have the apiservices API, means you don’t have any. If names come back, an unpatched cluster likely has the setup where anonymous users could escalate, and that’s a strong reason to prioritize the upgrade on that cluster. Even after patching, I’d go through the list and make sure you still know why each one is there.

Review who can exec into pods

This is the part I’d do even on a patched, managed cluster. Exec, attach, and portforward are powerful permissions, and it’s easy to hand them out through edit without thinking about it.

kubectl auth can-i asks the API server’s authorization layer directly, and the authorization docs show pairing it with impersonation to check another user. Since kubectl exec sends a POST to the pod’s exec subresource, and POST maps to the create verb, this is a reasonable check for a given user and namespace:

kubectl auth can-i create pods --subresource=exec --namespace dev --as dave

To see who holds admin or edit, list the bindings with the wide output, which adds the role, users, groups, and service accounts for each binding:

kubectl get rolebindings,clusterrolebindings --all-namespaces -o wide

If a person or a CI service account only needs to deploy and read logs, give them a narrower role without the exec, attach, and portforward subresources. Until you’ve upgraded, the issue lists removing those permissions from users who shouldn’t have full kubelet access as the mitigation for this path.

Anonymous access and discovery

The RBAC docs explain that the default discovery bindings let both authenticated and unauthenticated users read API information considered safe to expose, and that you can turn off anonymous requests by passing --anonymous-auth=false to the API server. For clusters that can’t upgrade right away, the issue lists that flag as one mitigation for the anonymous path, along with removing all anonymous access to aggregated APIs or suspending aggregated API servers entirely. It’s upfront that these can hurt: disabling anonymous auth can break load balancer or kubelet health checks against the API server and breaks kubeadm join flows. The project strongly recommends upgrading over any of them, and so would I.

The bigger takeaway is that the API server is a proxy to very powerful things, and your RBAC is only as tight as the paths through it. Upgrade or confirm your provider has, list your aggregated APIs, and take a real look at who can exec into pods.