Reading time ~13 minutes
Deploy Your Own Kubernetes Lab
- Option 1 - Run Kubernetes Locally
- Option 2 - Deploy a Deliberately Vulnerable Cluster
- Option 3 - Deploy a Multi-Node Production Ready Cluster
- Option 4 - Deploy to Cloud
- Option 5 - Deploy on Baremetal
- Bonus
- Conclusion
This post is part of the “Kubernetes Primer for Security Professionals” series, and is going to cover multiple deployment options for a Kubernetes lab, ranging from more lightweight (like running Kubernetes locally) to more realistic ones (like deploying a multi-node cluster) suitable for security research.
Option 1 - Run Kubernetes Locally
Especially at the beginning, while you are getting accustomed to Kubernetes, a local installation is probably going to be enough. Here we are going to discuss the predominant alternatives currently available.
Minikube vs Docker for Mac
Minikube is a tool that makes it easy to run Kubernetes locally: it implements a local, single-node, Kubernetes cluster inside a VM for users looking to try Kubernetes or develop with it day-to-day. In addition, it is supported on all the three major desktop operating systems (MacOS, Linux, and Windows).
At the same time, if you run MacOS you are lucky because Kubernetes is now bundled in Docker for Mac. So which one is the best option?
The Codefresh team wrote an article describing pros and cons of the two products. To summarize it:
- Minikube is a mature solution available for all major operating systems. Its main advantage is that it provides a unified way of working with a local Kubernetes cluster regardless of the operating system. It is perfect for people that are using multiple OS machines and have some basic familiarity with Kubernetes and Docker.
- Docker for Mac is a very user-friendly solution with good integration for the MacOS UI, but it also comes with limited configuration options.
Between the two, I ended up using Minikube for my first approach towards Kubernetes.
Setup Minikube
First of all, see “Installing Minikube” in the official Kubernetes documentation for instructions on how to obtain the latest release.
Once installed successfully, you can start Minikube by typing minikube start
:
❯ minikube start
😄 minikube v1.0.0 on darwin (amd64)
🤹 Downloading Kubernetes v1.14.0 images in the background ...
🔥 Creating virtualbox VM (CPUs=2, Memory=2048MB, Disk=20000MB) ...
💿 Downloading Minikube ISO ...
142.88 MB / 142.88 MB [============================================] 100.00% 0s
📶 "minikube" IP address is 192.168.0.10
🐳 Configuring Docker as the container runtime ...
🐳 Version of container runtime is 18.06.2-ce
⌛ Waiting for image downloads to complete ...
✨ Preparing Kubernetes environment ...
💾 Downloading kubelet v1.14.0
💾 Downloading kubeadm v1.14.0
🚜 Pulling images required by Kubernetes v1.14.0 ...
🚀 Launching Kubernetes v1.14.0 using kubeadm ...
⌛ Waiting for pods: apiserver proxy etcd scheduler controller dns
🔑 Configuring cluster permissions ...
🤔 Verifying component health .....
💗 kubectl is now configured to use "minikube"
🏄 Done! Thank you for using minikube!
The setup process creates a new virtual machine (based on Virtualbox in my case), pulls all the images needed,
and then creates a new Kubernetes context called “minikube
”.
If you are working with multiple Kubernetes clusters and different environments you should be familiar with the concept of switching contexts. You can view contexts using the kubectl config
command:
❯ kubectl config get-contexts
CURRENT NAME CLUSTER AUTHINFO NAMESPACE
docker-for-desktop docker-for-desktop-cluster docker-for-desktop
* minikube minikube minikube
Once ensured “minikube
” is the default context, let’s obtain the cluster information with kubectl
as a smoke test:
❯ kubectl cluster-info
Kubernetes master is running at https://192.168.99.101:8443
KubeDNS is running at https://192.168.99.101:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
You can also use Minikube to access the Kubernetes dashboard:
❯ minikube dashboard
🔌 Enabling dashboard ...
🤔 Verifying dashboard health ...
🚀 Launching proxy ...
🤔 Verifying proxy health ...
🎉 Opening http://127.0.0.1:52012/api/v1/namespaces/kube-system/services/http:kubernetes-dashboard:/proxy/ in your default browser...

Hello World with Minikube
To complete the example, we can follow the “Hello Minikube” tutorial of the official documentation to run a simple Hello World Node.js app on Kubernetes using Minikube:
❯ kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node
deployment.apps/hello-node created
❯ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
hello-node 1/1 1 1 2m43s
❯ kubectl expose deployment hello-node --type=LoadBalancer --port=8080
service/hello-node exposed
❯ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-node LoadBalancer 10.111.188.237 <pending> 8080:30567/TCP 4s
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 21m
❯ minikube service hello-node
🎉 Opening kubernetes service default/hello-node in default browser...
Option 2 - Deploy a Deliberately Vulnerable Cluster
Minikube and Docker for Mac work fine if you just want to try Kubernetes out or for developers who have to work with it daily (i.e., by having local environments for quick development sprints).
However, if you are doing security research, you will need access to configuration and security settings of the Kubernetes deployment itself.
Run Kubernetes on a Vagrant VM
Liz Rice summarized this in her “Kubernetes in Vagrant with kubeadm” blog post:
[…] However, a lot of what I’m doing at the moment relates to security settings that you might configure on your Kubernetes cluster. For example, if I’m working on kube-bench a lot of the tests look at the parameters passed to the API Server executable. Neither Minikube nor Docker for Mac use standard installation tools like kubeadm or kops that you might use for a production cluster, so for my work I was looking for ways to tweak parameters in ways that are not the same as on a regular production server.
[…] I decided that I’d be better off running exactly the same code that a Kubernetes user might run on a production cluster. And I couldn’t see any reason not to try running that in a regular Linux VM on my local machine.
She ended up providing a fully annotated Vagrant file to reproduce her setup. This is definitely worth a try if you need a quick (and still local) deployment to use for demo, talks, etc.
Run Insecure Configurations with Kind
Kind is a tool for running local Kubernetes clusters using Docker container “nodes”, bootstrapping them with kubeadm
.
What’s interesting here is that Rory McCune put together a collection of kind configuration files recreating some of the common insecure configurations you can see in Kubernetes clusters. The samples Rory put together are not compatible with the latest version of Kind anymore, so you can use my fork which should work out of the box.
# Install kind
❯ go get sigs.k8s.io/kind
# Download sample configs
❯ git clone https://github.com/marco-lancini/kind-of-insecure.git
❯ cd kind-of-insecure
# Create vulnerable cluster
❯ kind --config insecure-port.yaml --name insecure create cluster
Creating cluster "insecure" ...
✓ Ensuring node image (kindest/node:v1.15.0) 🖼
✓ Preparing nodes 📦📦📦📦
✓ Creating kubeadm config 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Cluster creation complete. You can now use the cluster with:
export KUBECONFIG="$(kind get kubeconfig-path --name="insecure")"
kubectl cluster-info
❯ export KUBECONFIG="$(kind get kubeconfig-path --name="insecure")"
❯ kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:57546
KubeDNS is running at https://127.0.0.1:57546/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
❯ k get nodes
NAME STATUS ROLES AGE VERSION
insecure-control-plane Ready master 75s v1.15.0
insecure-worker Ready <none> 30s v1.15.0
insecure-worker2 Ready <none> 28s v1.15.0
insecure-worker3 Ready <none> 30s v1.15.0
# Test for insecure port
❯ curl http://127.0.0.1:8080/
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/",
...
"/logs",
"/metrics",
"/openapi/v2",
"/version"
]
}
Option 3 - Deploy a Multi-Node Production Ready Cluster
Kubespray aims to be a one stop shop to deploy a production ready Kubernetes cluster, which supports the most popular Linux distributions and can be deployed on basically any existing provider (AWS, GCP, Azure, OpenStack, vSphere, Packet, Oracle Cloud Infrastructure, or Baremetal). In addition, it is highly composable, allowing to select core components, network plugins, and applications to be deployed.
Setup Kubespray
To use Kubespray I will assume you have at least two available hosts, which at least 1.5 GB of memory each (although 2048 MB would be better).
For my setup, I created two hosts running Ubuntu Server with statically assigned IP addresses (192.168.1.111
and 192.168.1.112
) and with key-based SSH authentication (for which I created a new key pair named k8s_key
).
With our hosts up and running, we can use Ansible to quickly deploy our components in an automated fashion.
Since I don’t want to run Ansible straight from my host, I created a docker container (hereinafter called the “ansible_worker
”) to play the role of the control machine. This container can be downloaded freely from the Github Docker Registry (more on this in a moment).
Here is the full process to deploy our Kubernetes lab:
- First, let’s clone the Kubespray repository:
❯ git clone https://github.com/kubernetes-sigs/kubespray.git
❯ cd kubespray
- Then, we can pull the
ansible_worker
from the Docker Hub, mounting both thekubespray
and the~/.ssh/
folders as volumes:
$ docker run -ti --rm -v $(pwd):/kubespray -v ~/.ssh/:/root/.ssh/ ghcr.io/marco-lancini/ansible-worker:latest
- From within the
ansible_worker
container, add the identity we configured for the key-based SSH authentication:
/src $ cd /kubespray/
/kubespray $ eval "$(ssh-agent)"
/kubespray $ ssh-add ~/.ssh/k8s_key
Identity added: /root/.ssh/k8s_key (/root/.ssh/k8s_key)
- Install the kubespray dependencies and create a new inventory starting from the sample:
# Install dependencies from requirements.txt
/kubespray $ pip3 install -r requirements.txt
# Copy inventory/sample as inventory/mycluster
/kubespray $ cp -r inventory/sample inventory/mycluster
- Update the Ansible inventory file
inventory/mycluster/inventory.ini
to reflect your setup. Here is mine with just 2 hosts:
[all]
node1 ansible_host=192.168.1.111
node2 ansible_host=192.168.1.112
[kube-master]
node1
[etcd]
node1
[kube-node]
node2
[k8s-cluster:children]
kube-master
kube-node
- Review and change parameters in the
inventory/mycluster/group_vars
folder, specifically ininventory/mycluster/group_vars/all/all.yml
andinventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
-
It can be useful to set the following two variables to
true
ininventory/mycluster/group_vars/k8s-cluster/k8s-cluster.yml
:kubeconfig_localhost
(to make a copy of kubeconfig on the host that runs Ansible in{ inventory_dir }/artifacts
) andkubectl_localhost
(to download kubectl onto the host that runs Ansible in{ bin_dir }
) - Finally, we can deploy Kubernetes by running the
cluster.yml
playbook (herevagrant
is my dummy user on the 2 hosts):
/kubespray $ ansible-playbook -b -v --become-user=root -i inventory/mycluster/inventory.ini -u vagrant --private-key=~/.ssh/k8s_key cluster.yml
PLAY [localhost] **************************************************************************************************************************************************************
TASK [Check ansible version >=2.7.6] ******************************************************************************************************************************************
Monday 15 April 2019 15:07:04 +0000 (0:00:00.156) 0:00:00.156 **********
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
[WARNING]: Could not match supplied host pattern, ignoring: bastion
...[omitted for brevity]...
PLAY RECAP ********************************************************************************************************************************************************************
localhost : ok=1 changed=0 unreachable=0 failed=0
node1 : ok=367 changed=114 unreachable=0 failed=0
node2 : ok=274 changed=81 unreachable=0 failed=0
Monday 15 April 2019 15:17:00 +0000 (0:00:00.042) 0:10:01.818 **********
===============================================================================
bootstrap-os : Install python and pip --------------------------------------------------------------------------------------------------------------------------------- 59.41s
download : file_download | Download item ------------------------------------------------------------------------------------------------------------------------------ 32.60s
kubernetes/master : kubeadm | Initialize first master ----------------------------------------------------------------------------------------------------------------- 27.96s
download : container_download | download images for kubeadm config images --------------------------------------------------------------------------------------------- 27.47s
kubernetes/preinstall : Install packages requirements ----------------------------------------------------------------------------------------------------------------- 26.87s
container-engine/docker : ensure docker packages are installed -------------------------------------------------------------------------------------------------------- 21.75s
download : file_download | Download item ------------------------------------------------------------------------------------------------------------------------------ 17.63s
kubernetes/kubeadm : Join to cluster ---------------------------------------------------------------------------------------------------------------------------------- 16.57s
container-engine/docker : Docker | pause while Docker restarts -------------------------------------------------------------------------------------------------------- 10.24s
etcd : wait for etcd up ------------------------------------------------------------------------------------------------------------------------------------------------ 8.30s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------------------------------- 8.28s
download : file_download | Download item ------------------------------------------------------------------------------------------------------------------------------- 8.12s
download : file_download | Download item ------------------------------------------------------------------------------------------------------------------------------- 8.07s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------------------------------- 8.05s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------------------------------- 7.75s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------------------------------- 7.69s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------------------------------- 6.53s
kubernetes-apps/ansible : Kubernetes Apps | Start Resources ------------------------------------------------------------------------------------------------------------ 6.28s
download : container_download | Download containers if pull is required or told to always pull (all nodes) ------------------------------------------------------------- 6.20s
etcd : Configure | Check if etcd cluster is healthy -------------------------------------------------------------------------------------------------------------------- 5.61s
Interact with the Cluster
Having set kubeconfig_localhost: true
means that Ansible will automatically make a copy of the kubeconfig
file on the host that runs Ansible (ansible_worker
). Since we are sharing the working folder as a docker volume, we can then access the same kubeconfig
from our host (on which I assume you’ll have already installed kubectl).
Let’s perform a smoke test to ensure everything works as expected (notice how we are manually specifying the --kubeconfig
location so that kubectl
knows how to access our new cluster):
❯ kubectl --kubeconfig=./inventory/mycluster/artifacts/admin.conf cluster-info
Kubernetes master is running at https://192.168.1.111:6443
coredns is running at https://192.168.1.111:6443/api/v1/namespaces/kube-system/services/coredns:dns/proxy
kubernetes-dashboard is running at https://192.168.1.111:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
❯ kubectl --kubeconfig=./inventory/mycluster/artifacts/admin.conf get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready master 8m v1.13.5
node2 Ready <none> 7m25s v1.13.5
❯ kubectl --kubeconfig=./inventory/mycluster/artifacts/admin.conf proxy
Starting to serve on 127.0.0.1:8001
# To access the dashboard browse to:
# http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
To avoid specifying the --kubeconfig
location for every single command, you can copy admin.conf
to ~/.kube/config
.
If you’ve never heard of the kubeconfig
file, I’d recommend you to read the “Configure Access to Multiple Clusters” article from the Kubernetes docs.
Hello World with Kubespray
To complete the example, let’s deploy again the “Hello Minikube” tutorial:
❯ kubectl create deployment hello-node --image=gcr.io/hello-minikube-zero-install/hello-node
deployment.apps/hello-node created
❯ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
hello-node 1/1 1 1 50s
❯ kubectl expose deployment hello-node --type=LoadBalancer --port=8080
service/hello-node exposed
❯ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-node LoadBalancer 10.233.35.126 <pending> 8080:30723/TCP 40s
kubernetes ClusterIP 10.233.0.1 <none> 443/TCP 103m
# Expose the service locally
❯ kubectl port-forward svc/hello-node 30723:8080
Forwarding from [::1]:30723 -> 8080
Forwarding from 127.0.0.1:30723 -> 8080
Handling connection for 30723
Handling connection for 30723

Additional documentation on the usage of Kubespray can be found on its Github page.
Option 4 - Deploy to Cloud
Although I haven’t explored this avenue yet because I didn’t want to worry about ongoing expenditure of a cloud lab, I felt like I had to mention at least one solution for sake of completeness.
The quickest (and most configurable) way to deploy to cloud is “Kubernetes the Easy Way”, which bootstraps Kubernetes on Google Cloud Platform. Otherwise, Kubespray can deploy to AWS/GCP/Azure.
Option 5 - Deploy on Baremetal
In Kubernetes Lab on Baremetal, I described my personal approach to deploy my own Kubernetes Lab on baremetal, and on an Intel NUC in particular.
The setup described in that post has been automated as part of k8s-lab-plz, a modular Kubernetes Lab which provides an easy and streamlined way to deploy a test cluster with support for different components. You can read more about it at: Introducing k8s-lab-plz: A modular Kubernetes Lab.
In particular, you can refer to the Baremetal Setup page of the documentation for specific instructions.
Bonus
I’ve also came across some other interesting installation methods that might come up handy at some point:
Name | Description |
---|---|
k8s-lab-plz | A modular Kubernetes lab which provides an easy and streamlined way to deploy a test cluster with support for different components. |
k3s | A lightweight Kubernetes. Claimed to be easy to install, half the memory, all in a binary less than 40mb. Great for: Edge, IoT, CI, ARM. |
kind-of-insecure | A collection of kind configuration files that can be used to create deliberately vulnerable clusters, for the purposes of security testing/training. |
Conclusion
In this post, part of the “Kubernetes Primer for Security Professionals” series, we explored some deployment options for a custom Kubernetes lab.
We saw how, although Minikube and Docker for Mac work fine for anyone who just want to try Kubernetes out or for developers who have to work with it daily, if you are doing security research you will need access to configuration and security settings of the Kubernetes deployment itself. Here, Kubespray can help deploying a “production ready” cluster using Ansible to automate provisioning.
The full Kubespray configuration used in this article, together with a handy cheatsheet, can be found in the related Github repository: https://github.com/marco-lancini/offensive-infrastructure/tree/master/kubernetes.
I hope you found this post useful and interesting, and I’m keen to get feedback on it! Therefore, if you find the information shared in this series is useful, or if something is missing, or if you have ideas on how to improve it, please leave a comment in the area below, or let me know on Twitter.