Kubernetes 1.0 for People Who Just Learned Docker

Kubernetes is officially 1.0. The v1.0.0 tag was cut on GitHub earlier this month, and on Tuesday, July 21, Google made the launch official at OSCON in Portland. The announcement on the Google Cloud Platform blog says the release was built by over 400 contributors and that Kubernetes is ready for production use. The same post says Google is joining the Linux Foundation and a group of industry partners to form the Cloud Native Computing Foundation, which Google plans to seed with Kubernetes.

If you’ve just gotten comfortable with docker run, the vocabulary is a lot at once, and it’s not obvious which of pods, replication controllers and services is the container. So here are the core objects as they exist in 1.0, how each relates to Docker, a tiny example against the v1 API, and the questions I’d ask before running it.

Pods are the unit, not containers

You never schedule a bare container in Kubernetes. The smallest thing you create is a pod, which the v1.0.0 pod docs describe as a colocated group of Docker containers with shared volumes. Every container in a pod lands on the same machine, shares the pod’s IP address and port space, and can reach the others on localhost.

The networking doc explains how that sits on top of Docker. Plain Docker puts each container on a private docker0 bridge, so reaching it from another machine means publishing a port on the host and making sure nobody else took that port. Kubernetes instead requires that every pod can reach every other pod, and every node can reach every pod, without NAT, and it gives every pod its own IP. It builds that with a “pod container” that holds the network namespace open, while your app containers join it through Docker’s --net=container:<id> option. So a pod is roughly a few docker run calls sharing one network stack, and host ports become the rare case.

The container fields map onto things you know. image is the Docker image, pulled from Docker Hub by default. command overrides the image’s Entrypoint and args stands in for its Cmd. kubectl logs does what docker logs does. What you have to unlearn is permanence. Once a pod is scheduled to a node it stays there, and if that node dies the pod is deleted, never moved. The docs say you should almost always use a controller instead of creating pods directly, even for a single instance.

Labels and selectors hold it all together

Labels are key and value pairs you attach to objects, like app=hello-web or track=canary, and Kubernetes doesn’t assign them any meaning. The useful part is the label selector, a query such as app=hello-web,track!=canary, which the docs call the core grouping primitive. When you query the API you can also use set based selectors like environment in (production, qa), although the selector on a v1 service or controller is a plain map of keys and values that must all match.

Nothing owns pods by name. Replication controllers and services each carry a selector and act on whatever pods match it right now, which makes canaries easy. The docs sketch one controller running 9 replicas labeled track=stable, another running 1 labeled track=canary, and a service whose selector leaves track out, so traffic reaches all ten.

Replication controllers keep the count

A replication controller makes sure a set number of pod replicas is running. If there are too many, it kills some. Too few, and it starts more from its pod template. The docs compare it to a process supervisor for pods across many nodes, and recommend one even for a single pod, because a controller replaces pods lost to node failure while a bare pod is just gone.

It’s deliberately narrow. It doesn’t run health checks, autoscale, or make scheduling decisions, and restarting crashed containers is the kubelet’s job. Changing the template has no effect on pods that already exist. The friendlier commands in kubectl, run, scale, stop and rolling-update, are built on top of it, and a rolling update is really kubectl creating a second controller and shifting replicas from the old one to the new one a pod at a time.

Two behaviors catch people out. Deleting a controller through the raw API leaves its pods running unless you scale it to 0 first (kubectl delete and kubectl stop do that for you). And if you change a pod’s labels so it stops matching, the controller counts it as missing and starts a replacement, while the relabeled pod keeps running so you can poke at it.

Services give pods a stable address

Pod IPs change whenever a controller replaces a pod, so something has to stay put. A service is a selector plus a stable virtual IP, called the cluster IP. Kubernetes keeps evaluating the selector and writes the matching pod addresses to an Endpoints object with the same name. Every node runs kube-proxy, which watches services and endpoints, installs iptables rules that catch cluster IP traffic, and proxies each connection to a backend. In 1.0 services work at the TCP and UDP level, with no notion of HTTP.

Pods find services through environment variables or DNS. The kubelet injects variables like HELLO_WEB_SERVICE_HOST and HELLO_WEB_SERVICE_PORT, plus Docker links style ones, but only for services that already exist when the pod starts. DNS comes from an optional add on that the docs strongly recommend. To reach a service from outside the cluster, set type to NodePort, which opens the same port on every node (picked from 30000 to 32767 by default), or to LoadBalancer on clouds that support it.

A small example against the v1 API

Here’s a web tier with three nginx replicas, a service in front, and a one shot pod that fetches the page through the service. Every field below is defined in the v1 Swagger spec in the v1.0.0 tag. First the service, in hello-svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: hello-web
  labels:
    app: hello-web
spec:
  selector:
    app: hello-web
  ports:
  - port: 80
    targetPort: http

Then the controller, in hello-rc.yaml:

apiVersion: v1
kind: ReplicationController
metadata:
  name: hello-web
spec:
  replicas: 3
  selector:
    app: hello-web
  template:
    metadata:
      labels:
        app: hello-web
    spec:
      containers:
      - name: nginx
        image: nginx:1.9.3
        ports:
        - name: http
          containerPort: 80
        readinessProbe:
          httpGet:
            path: /
            port: 80

And the check, in hello-check.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: hello-check
spec:
  restartPolicy: Never
  containers:
  - name: check
    image: busybox
    command: ["sh", "-c", "wget -qO- http://$HELLO_WEB_SERVICE_HOST:$HELLO_WEB_SERVICE_PORT"]

The controller and the service both select the template’s label. The service’s targetPort names the container port, which the services doc says lets a later version change the port number without breaking clients. The readiness probe keeps each pod out of the endpoints until nginx answers on /. It uses the number rather than the name on purpose: in the v1.0.0 kubelet source, resolving a probe port by name returns the container’s host port, which isn’t set here, so the named version would fail. The image tag is pinned because 1.0 pulls a :latest image every time a container starts but other tags only when missing, so nodes could drift onto different builds.

Create the service before the pods, so they get its environment variables, then check the result:

kubectl create -f hello-svc.yaml
kubectl create -f hello-rc.yaml
kubectl get pods -l app=hello-web
kubectl create -f hello-check.yaml
kubectl logs hello-check

Then delete a hello-web pod with kubectl delete pod and watch a replacement appear, try kubectl scale rc hello-web --replicas=5, and clean up with kubectl stop rc hello-web.

Questions to ask before you run it

Who can reach the API server? According to the 1.0 API access doc, the apiserver serves two ports by default. Port 8080 speaks plain HTTP with no authentication or authorization, bound to localhost and protected only by access to that host. Port 6443 serves HTTPS and authenticates with client certificates, tokens or basic auth. And the v1.0.0 apiserver source defaults --authorization-mode to AlwaysAllow, so any authenticated user can do anything unless someone wrote an ABAC policy file. Tokens and basic auth passwords last indefinitely and can’t change without restarting the apiserver. So ask whoever built the cluster whether 8080 is reachable from anywhere but the master, which authorization mode is set, and who else holds the credentials in your ~/.kube directory.

How are secrets handled? 1.0 has a Secret object. Values are base64 in the manifest, which is an encoding and not encryption, and they reach containers as files in a volume. A secret is only sent to a node running a pod that needs it, where it lives on tmpfs. The docs are honest about the gaps: there’s no way yet to control which users of a cluster can access a secret, anyone who can create a pod that uses a secret can read it, and anyone with root on any node can read any secret from the apiserver by impersonating the kubelet. Treat a shared 1.0 cluster as one trust zone.

Is it production ready? Google says yes, citing scale tests of hundreds of nodes and thousands of containers and a stable API with a formal deprecation policy. The README in the v1.0.0 tag still has a heading calling Kubernetes pre-production beta and tells you to expect bugs and API changes. The services doc warns that the userspace proxy may not scale to thousands of services and hides the client’s source IP. The beta API versions are on their way out (v1beta1 and v1beta2 were deleted in June, and 1.0 stops exposing v1beta3 by default), so tutorials with fields like portalIP or createExternalLoadBalancer need updating. And the docs point out that NodePort and LoadBalancer might expose a service to your corporate network or to the whole internet depending on the environment, so check whether the app does its own authentication.

The takeaway

Kubernetes 1.0 boils down to three ideas layered on Docker: pods that share a network stack, labels that let objects find each other without hard links, and small controllers that keep reality matching what you asked for. Learn those and the YAML stops looking like magic. Just remember the cluster now has its own API, and in 1.0 that API trusts a lot more than you’d guess.