Production ready Kubernetes Cluster on Hetzer

Production ready Kubernetes Cluster on Hetzer

MicroK8s and Rook

2021-02-01 06:00:53

We decided to start using Kubernetes in production. We were looking for a solution easy that is easy to install for us and your customers. As a German company we love to use Hetzner services. Hetzner does not provide a managed Kubernetes Cluster, so you need to install it on our own. There is already a tool called hetzner-kube to build a K8s Cluster based on kubeadm, but I fall in love with MicroK8s and wanted to create a production ready Kubernteres Cluster based on it.

This is a step by step tutorial how to create a production Kubernetes Cluste base on MicroK8s on the Hetzner infrastucture.

Hardware overview:

  • A load balancer with a domain and wildcald domain assigned to it

  • 3 servers (or more)

Software overview:

  • MicroK8s v1.19

  • Nginx ingress controller

  • Cert-manager with Let's Encrypt Cluster Issuer

  • Rook v1.5 with ceph v15.2.7

  • Kasten K10


  • a domain

  • installed hcloud and configured a project

  • basic Kubernetes knowledge (not realy needed, but realy welcome)

Create a load balancer

First of all create a load balancer to have one IP address for the whole cluster.

hcloud load-balancer create --type=lb11 --name=k8s --location=fsn1
hcloud load-balancer add-service k8s --protocol=tcp --listen-port=80 --destination-port=80
hcloud load-balancer add-service k8s --protocol=tcp --listen-port=443 --destination-port=443
hcloud load-balancer add-service k8s --protocol=tcp --listen-port=16443 --destination-port=16443

We create a label selector so the load balancer will automaticly target our servers

hcloud load-balancer add-target k8s --label-selector=k8s

Now we grab the load balancer IP, go to your domain provider and create a DNS entry for it. (e.g Also create a wildcard for subdomains (e.g *

Create a server

hcloud server create \
--image=ubuntu-20.04 \
--type=cpx31 \
--datacenter=fsn1-dc14 \
--ssh-key=<your_ssh_key_name> \
--label=k8s= \

Grab the server IP from the output

Waiting for server XXXXX to have started
... done
Server XXXXX created

Adjust partition table (optional)

We need to adjust the partition table for Rook. I decided to left 20GB for the system and everything else for Rook.

This step is optional. You can also attach Volumes to you servers.

Enable rescue

hcloud server enable-rescue --ssh-key=<your_ssh_key_name> k8s-1
hcloud server reboot k8s-1

Ssh into the rescue system

hcloud server ssh k8s-1

Shrink the main partition and create an unformatted one

e2fsck -f /dev/sda1
resize2fs /dev/sda1 40G
printf "d\n1\nn\n1\n\n+40G\nn\n2\n\n\nt\n2\n31\nw\n" | fdisk -B /dev/sda
e2fsck -f /dev/sda1

Update the system and install snapd

hcloud server ssh k8s-1

apt update && apt upgrade -y
apt install -y snapd

Install MicroK8s

hcloud server ssh k8s-1

snap install microk8s --classic --channel 1.19

Make MicroK8s aware of you load balancer IP

sed -i '/#MOREIPS/a IP.100 = <load_balancer_ip>' /var/snap/microk8s/current/certs/csr.conf.template

Install basic addons

microk8s.enable dns:
microk8s.enable rbac ingress metrics-server helm3

Create a join token

Replate <token> with a 32 charakters random string.

This token will be valid for 10 years.

microk8s.add-node --token-ttl 315360000 --token <token>

Create two more servers

We need minimum 3 servers to have a proper cluster. To avoid repetition of the partition adjustments, we will create a snapshot and then create 2 more servers from it.

Create a snapshot as base for other servers

hcloud server create-image --type=snapshot k8s-1

This will take ~10min, so go have some coffee 😉

Gram the snaphot_id from the output.

Image <snaphost_id> created from server YYY

Create more servers

We will use a the snapshot and a simpe cloud init script to bootstap the servers and let them join the cluster.

Create a cloud-config.yaml file. Remember to deplate <first_node_ip> and <token>.

- /snap/bin/microk8s join <first_node_ip>:25000/<token>

You can run this commands parallel.

hcloud server create \
--label k8s= \
--image=<snaphot_id> \
--type=cpx31 \
--datacenter=fsn1-dc14 \
--ssh-key=<your_ssh_key_name> \

hcloud server create \
--label k8s= \
--image=<snaphot_id> \
--type=cpx31 \
--datacenter=fsn1-dc14 \
--ssh-key=<your_ssh_key_name> \

Install cert-manager

Deploy the cert manager using helm

hcloud server ssh k8s-1

microk8s.kubectl create namespace cert-manager
microk8s.helm3 repo add jetstack
microk8s.helm3 repo update
microk8s.helm3 upgrade --install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.1.0 \
  --set installCRDs=true \
  --set ingressShim.defaultIssuerName=letsencrypt-prod \
  --set ingressShim.defaultIssuerKind=ClusterIssuer \

Create a cluster issuer for Let's Encrypt

Create a cluster-issuer.yaml file.

kind: ClusterIssuer
  name: letsencrypt-prod
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
      name: letsencrypt-prod
    - http01:
          class: nginx
microk8s.kubectl create -f cluster-issuer.yaml

Install kubernetes dashboard

hcloud server ssh k8s-1

microk8s.enable dashboard

To be able to access the dashboard we need to create an ingress resource.

Create a dashboard.yaml file.

apiVersion: extensions/v1beta1
kind: Ingress
  annotations: "true" nginx "HTTPS"
  name: dashboard
  namespace: kube-system
  - host: <your_domain_here>
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 443
        path: /
  - hosts:
    - <your_domain_here>
    secretName: dashboard-ingress-cert
microk8s.kubectl create -f dashboard.yaml

Install Rook

hclound server ssh k8s-1

Ceph Operator

Install the Ceph Operator using Helm

Becouse microk8s comes as a snap package with bundled kubelet, we need to tell the rook operator about it.

We also set enableDiscoveryDaemon=true to enable autodiscovery of hardware changes.

We use the 1.5 version, because after a whole night of tries I was unable to make version 1.6 working.

microk8s.helm3 repo add rook-release

microk8s.kubectl create namespace rook-ceph
microk8s.helm3 upgrade --install \
--set csi.kubeletDirPath=/var/snap/microk8s/common/var/lib/kubelet/ \
--set enableDiscoveryDaemon=true \
--namespace rook-ceph \
rook-ceph rook-release/rook-ceph \
--version 1.5

Ceph Cluster

Again, becouse of the snap, we need adjust some paths.

dataDirHostPath: /var/snap/microk8s/common/var/lib/rook

Additionalaly we change the cehp/ceph image version to an older revision, becouse the current one has a bug and will not discover our partition.

Create a rook-cluster.yaml file with following content.

kind: CephCluster
  name: rook-ceph
  namespace: rook-ceph
    image: ceph/ceph:v15.2.7
    allowUnsupported: false
  dataDirHostPath: /var/snap/microk8s/common/var/lib/rook
    count: 3
    allowMultiplePerNode: false
    - name: pg_autoscaler
      enabled: true
enabled: true
ssl: true
    enabled: true
    rulesNamespace: rook-ceph
    disable: false
    confirmation: ""
      method: quick
      dataSource: zero
      iteration: 1
    allowUninstallWithVolumes: false
    useAllNodes: true
    useAllDevices: true
    managePodBudgets: false
    osdMaintenanceTimeout: 30
    pgHealthCheckTimeout: 0
    manageMachineDisruptionBudgets: false
    machineDisruptionBudgetNamespace: openshift-machine-api
        disabled: false
        interval: 45s
        disabled: false
        interval: 60s
        disabled: false
        interval: 60s
        disabled: false
        disabled: false
        disabled: false
microk8s.kubectl create -f rook-cluster.yaml

Block Storage Class

This will be the default storage class.

Create a storageclass.yaml file with following content.

kind: CephBlockPool
  name: replicapool
  namespace: rook-ceph
  failureDomain: host
    size: 3
kind: StorageClass
   name: rook-ceph-block
annotations: "true" provisioner: parameters:
clusterID: rook-ceph pool: replicapool imageFormat: "2" imageFeatures: layering rook-csi-rbd-provisioner rook-ceph rook-csi-rbd-provisioner rook-ceph rook-csi-rbd-node rook-ceph ext4
reclaimPolicy: Delete
allowVolumeExpansion: true

For detailed information see

microk8s.kubectl create -f storageclass.yaml

Shared Filesystem (ReadWriteMany)

create a rwm-storageclass.yaml file with following content

kind: CephFilesystem
name: myfs
namespace: rook-ceph
size: 3
- replicated:
size: 3
preserveFilesystemOnDelete: true
activeCount: 1
activeStandby: true
kind: StorageClass
name: rook-cephfs
clusterID: rook-ceph
fsName: myfs
pool: myfs-data0 rook-csi-cephfs-provisioner rook-ceph rook-csi-cephfs-provisioner rook-ceph rook-csi-cephfs-node rook-ceph

reclaimPolicy: Delete
microk8s.kubectl create -f rwm-storageclass.yaml

When you don't specify the storageClassName in PermisionVolumeClaim, the default one would be used.

Here is a example PermisionVolumeClaim that consumes this storage class:

apiVersion: v1
kind: PersistentVolumeClaim
name: cephfs-pvc
namespace: kube-system
- ReadWriteMany
storage: 1Gi
storageClassName: rook-cephfs

Why is this storage class not the default one when it is more flexible? Because block storage is faster and is what you will need most the times.

Object Storage (S3 API)

create a object-storageclass.yaml file with following content

kind: CephObjectStore
name: s3-store
namespace: rook-ceph
failureDomain: host
size: 3
failureDomain: host
dataChunks: 2
codingChunks: 1
preservePoolsOnDelete: true
type: s3
port: 80
# securePort: 443
instances: 1
disabled: false
interval: 60s
microk8s.kubectl create -f object-storageclass.yaml
The easiest way to create butkets is through the Ceph Dashboard

Accessing Ceph dashboard with Ingress

Create a rook-ingress.yaml file with the following content:

apiVersion: extensions/v1beta1
kind: Ingress
name: rook-ingress
namespace: rook-ceph
annotations: "nginx" "true" "HTTPS" |
proxy_ssl_verify off;
- hosts:
- ceph.<your-domain-here>
secretName: rook-tls
- host: ceph.<your-domain-here>
- path: /
serviceName: rook-ceph-mgr-dashboard
servicePort: https-dashboard

Install the chart using the values file

microk8s.kubectl create -f rook-ingress.yaml

The dashboard is now accessible under https://ceph.<your_domain_here>

Login Credentials

Rook creates a default user named admin

To retrieve the generated password, you can run the following:

microk8s.kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo

Backups with Kasten K10

Login to the k8s-1 server where we have helm installed

hcloud server ssh k8s-1

Enable Snaphosts

Snapshot Beta CRDs

microk8s.kubectl create -f
microk8s.kubectl create -f
microk8s.kubectl create -f

Common Snapshot Controller

microk8s.kubectl create -f
microk8s.kubectl create -f


We create a volumesnapshotclass for your block storage and annotate it with

microk8s.kubectl create -f
microk8s.kubectl annotate volumesnapshotclass csi-rbdplugin-snapclass

We do the same for the cehpfs storage.

microk8s.kubectl create -f
microk8s.kubectl annotate volumesnapshotclass csi-cephfsplugin-snapclass

Installing Kasten K10

Now we will install Kasten K10 to take care of our backups. We will use a helm chart for it. As part of it we will create a ingress to access the dashboard.

microk8s.helm3 repo add kasten
microk8s.kubectl create namespace kasten-io

Create a k10-values.yaml file with the following content:

  create: true
  class: nginx
  host: <your_doamin_here>
  annotations: "true"
    enabled: true

Install the chart using the values file

microk8s.helm3 install k10 kasten/k10 -n kasten-io -f k10-values.yaml

The dashboard is now accessible under https://<your_domain_here>/k10

Obtaining access token to access K10 Dashboard

sa_secret=$(microk8s.kubectl get serviceaccount k10-k10 -o jsonpath="{.secrets[0].name}" --namespace kasten-io)

microk8s.kubectl get secret $sa_secret --namespace kasten-io -ojsonpath="{.data.token}" | base64 --decode && echo

Some Questions?

Don't hesitate to contact us. We will answer all your questions as soon as possible.