Definition
Kubernetes – κυβερνήτης • (kyvernítis) m
- governor (leader of a region or state)
- (nautical) captain, skipper
- pilot (of an aircraft)
Motivation
We recently moved to a new office and discovered that one of our bare metal Continuous Integration build agents didn’t survive the move. Since other developers were already unhappy with the fact, that the CI could only be reached from within the office, we took that as an incentive to give building on the Google Cloud Platform a shot.
We already experimented with CoreOS for our internal logging system. Although we liked it, we were quite curious how a cluster setup with Kubernetes (K8S if you like) performs.
So in this post, we’ll show you how to set things up, what pitfalls to avoid, and how we managed to create a robust and easy to set up solution.
Cluster Setup
The setup of a Kubernetes cluster on the Google Cloud Platform is quite easy (and fast as a Google engineer will happily demonstrate) with the Google Cloud Container Engine.
I recommend using the command line tool gcloud because there are almost no Kubernetes features implemented in the GUI yet.
For our use case we figured two CPU cores should be enough and used a n1‑standard‑2 machine:
gcloud alpha container clusters create\
-m n1-standard-2\
teamcity
Build Agent Setup
Photo by Serguei S. Dukachev
Now that you have a three node Kubernetes cluster running, it is time to start-up some containers. In Kubernetes a group of containers and their resources (volumes, ports…) form a pod (as in: a pod of dolphins). To ensure that n pods are always running, you can use a replication controller. Almost anything you do in a Kuberntes cluster you do through kubectl which is hidden under:
gcloud alpha container kubectl
By the way, your life will be a bit easier when you setup an alias first:
alias kubectl='gcloud alpha container kubectl'
Here’s an example for our replication controller:
id: teamcity-agent
kind: ReplicationController
apiVersion: v1beta1
labels:
name: teamcity-agent
desiredState:
replicas: 3
replicaSelector:
name: teamcity-agent
podTemplate:
labels:
name: teamcity-agent
desiredState:
manifest:
version: v1beta1
id: teamcity-agent
containers:
- name: teamcityagent
image: smallimprovements/teamcity-agent-docker
env:
- name: TEAMCITY_SERVER
value: http://teamcity:8111
volumeMounts:
- name: agent-data
mountPath: /data
volumes:
- name: agent-data
source:
hostDir:
path: /var/buildAgent
This will spawn a replication controller (line 2) named teamcity-agent (line 1). The controller should fire up three pods (line 7). The pods consists of:
- a container named teamcityagent (line 18), that has its working directory “data mounted to a volume (line 23)
- a Volume named agent-data (line 27) that goes to a host directory (line 29)
If aren’t comfortable writing YAML, don’t worry, Kubernetes understands JSON as you’ll see in Googles example.
For a full reference of replication controllers refer to the excellent docs.
To fire up anything in Kubernetes you use kubectl create:
kubectl create -f ./teamcity-agent-rc.yml
If everything went well, you’ll get a list of your running controller and pods by:
kubectl get pods,rc
With the parameter “-l” you can filter the resources by the labels attached to them. In our example we gave the replication controller (line 5) – and all the pods spawned by him – the label “teamcity-agent” (line 12). So to list the controller and the pods, we run:
POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED
teamcity-agent-acnbt 10.196.1.4 teamcityagent smallimprovements/teamcity-agent-docker:latest k8s-teamcity-node-1.c.eastern-kit-781.internal/146.148.14.166 name=teamcity-agent Running 12 hours
teamcity-agent-hvz97 10.196.2.4 teamcityagent smallimprovements/teamcity-agent-docker:latest k8s-teamcity-node-2.c.eastern-kit-781.internal/104.155.19.117 name=teamcity-agent Running 12 hours
teamcity-agent-tycvv 10.196.3.4 teamcityagent smallimprovements/teamcity-agent-docker:latest k8s-teamcity-node-3.c.eastern-kit-781.internal/104.155.45.72 name=teamcity-agent Running 12 hours
CONTROLLER CONTAINER(S) IMAGE(S) SELECTOR REPLICAS
teamcity-agent teamcityagent smallimprovements/teamcity-agent-docker:latest name=teamcity-agent 3
We tried running an in-memory disk, ramdisk, inside the container, because we weren’t happy with the build times at first (but eventually didn’t see any noticeable performance improvements). If you want to do this, you need to start the whole Kubernetes cluster with –allow_privilged=true so you can mount tmpfs inside a container.
But, it’s much easier to set up a ramdisk on your host system, and use it as a volume for your containers. This is what we are doing in line 26 of the example above.
The fstab on our host system looks like this
tmpfs /var/buildAgent tmpfs noatime,size=2G 0 0
Build Server Setup
Another important part of our setup is the TeamCity Server. Again we use a replication controller.
“Why?” you might ask, do you use a replication controller, if you need only one pod?
That’s because pods are not durable. When a pod fails it’s gone. In their design document Google says that you “should almost always” use controllers to facilitate self-healing in the cluster.
id: teamcity
kind: ReplicationController
apiVersion: v1beta1
labels:
name: teamcity
desiredState:
replicas: 1
replicaSelector:
name: teamcity
podTemplate:
labels:
name: teamcity
desiredState:
manifest:
id: teamcity
containers:
- name: teamcity
image: smallimprovements/teamcity-docker
ports:
- name: teamcity
containerPort: 8111
volumeMounts:
- name: teamcity-data
mountPath: /var/lib/teamcity
volumes:
- name: teamcity-data
source:
persistentDisk:
pdName: teamcity-backup
fsType: ext4
Nothing special here, but note that we exposed the port 8111 of the teamcity container and named it “teamcity” (line 19). Furthermore we have given the label “teamcity” to the pods (line 12). The label will be important for the setup of our TeamCity service.
Which brings us to the last Kubernetes resources of this post, the service.
Service
Because pods are ephemeral and their containers will get a new IP address every time they are fired up, the only way to get a permanent address to access our TeamCity server is to expose it as a service:
id: teamcity
kind: Service
apiVersion: v1beta1
port: 8111
containerPort: teamcity
labels:
name: teamcity
selector:
name: teamcity
Here a service with the id “teamcity” (line 1) is created.
We use the container port with the name “teamcity” (line 5) from the pods that match the selector “teamcity” (line 9), i.e. the pods that have the label “teamcity”.
The container port is exposed as the port 8111 (line 4) on the teamcity service.
So how do you access the service now? From within a container the service is accessible on a host named after the services id. So inside a teamcityagent container you can access TeamCity now through http://teamcity:8111.
But we wanted to access TeamCity from our office. Luckily every service will get an IP address through which it is accessible from the nodes inside the cluster.
To find out the services IP address use kubectl, as in the following example:
NAME LABELS SELECTOR IP PORT
teamcity name=teamcity name=teamcity 10.199.243.51 8111
So now we know that from any node in the cluster TeamCity will be accessible via 10.199.243.51:8111.
Because we didn’t want to expose our TeamCity server to the public, we decided dig a ssh tunnel from our office into the cluster and from there to the TeamCity service.
# we use a load balanced ssh connection
# to reach any node in the teamcity cluster
LB_SSH_CLUSTER_TEAMCITY=123.123.1.2
# from inside the cluster access Teamcity
# on the services ip
SERVICE_TEAMCITY=10.199.243.51
PORT_TEAMCITY=8111
ssh -NfC\
-L $PORT_TEAMCITY:$SERVICE_TC:$PORT_TEAMCITY\
$LB_SSH_CLUSTER_TEAMCITY
You might have stumbled over the load balanced comment: To make things more stable we decided to use a load balancer for the SSH connection. So when we destroy or create nodes in our Kubernetes cluster, we add them to the target pool and they are eligible for the tunnel. How to set up a load balancer on the cloud platform is described here.
To wrap things up, here are the steps to get from zero to a cluster with three nodes running TeamCity and the agents:
#create the cluster
gcloud alpha container clusters create\
-m n1-standard-2\
teamcity
#create TeamCity
kubectl create -f teamcity-rc.yml
#expose it as a service
kubectl create -f teamcity-service.yml
#create the agents
kubectl create -f teamcity-agent-rc.yml
Debugging
To be fair most of the time you’ll spend getting the resource configurations right, or debugging your containers. A nice touch is that you can read the logs from outside the cluster through kubectl:
# kubectl logs <pod> <container>
kubectl logs teamcity-grizy teamcity
If the pods won’t start, this doesn’t help you much, because as Kubernetes will tell you: There are no logs. In that case a “kubectl get events will” usually help. This will give you a list of events that happened in your cluster, before things went bad.
In theory you can execute commands through kubectl in the containers, as we tried in the following example
kubectl exec -it\
-p teamcity-agent-acnbt\
-c teamcityagent\
-- bash
Alas, in practice this consistently failed us, but we guessed it will work once the container engine reaches beta (it might work for you).
Conclusion
Its been fun setting up a build system on Kubernetes. Debugging was easier than on plain CoreOS because we could read the clusters events.
We loved the service abstraction to link containers and were quite impressed by the replication controllers (a feature lacking in plain CoreOS)
What we didn’t like was, that it wasn’t possible to enforce that only one agent might run per node. So it took some tries (kubectl {delete, create}) till they ran where they should.
Per cluster one additional instance is created, that hosts the Kubernetes API: The master. Currently it will have the same machine type as every node in your cluster. So for our 3 node setup an additional two core machine is quite a waste, but this evens out when you do larger clusters.
