Usage

All k8s-tew commands accept the argument --base-directory, which defines where all the files (binaries, certificates, configurations and so on) will be stored. If no value is defined then it will create a subdirectory called “assets” in the working directory. Additionally, the environment variable K8S_TEW_BASE_DIRECTORY can be set to point to the assets directory instead of using --base-directory.

To see all the commands and their arguments use the -h argument.

To activate completion for bash enter:

source <(k8s-tew completion)

For zsh run:

source <(k8s-tew completion zsh)

Requirements

k8s-tew was tested so far on Ubuntu 20.04 and CentOS 8.2. But it should be able to run on other Linux distributions.

Host related dependencies such as socat, conntrack, ipset and rbd are embedded in k8s-tew and put in place, once the cluster is running. Thus it is fairly portable to other Linux distributions.

Labels

k8s-tew uses labels to specify which files belong on a node, which commands should be executed on a node and also which components need to be started on a node. They are similar to Kubernetes’ Labels and Selectors.

bootstrapper - This label marks bootstrapping node/commands
controller - Kubernetes Controler. At least three controller nodes are required for a HA cluster.
worker - Kubernetes Node/Minion. At least one worker node is required.
storage - The Ceph Monitors are installed on the controller nodes. The storage nodes are used to store the actual data of the pods, and also to run the containers related to Ceph and CSI.

Workflow

To setup a local single-node cluster, the workflow is: initialize -> configure -> generate -> run

For a remote cluster, the workflow is: initialize -> configure -> generate -> deploy

If something in one of the steps is changed, (e.g. configuration), then all the following steps have to be performed.

Initialization

The first step in using k8s-tew is to create a config file. This is achieved by executing this command:

k8s-tew initialize

That command generates the config file called base-directory/etc/k8s-tew/config.yaml. To overwrite the existing configuration use the argument -f.

Configuration

After the initialization step the parameters of the cluster should be be adapted. These are the configure parameters and their defaults:

--alert-manager-count uint16

Number of Alert Manager Servers (default 1)

--alert-manager-size uint16

Size of Alert Manager Persistent Volume (default 2)

--apiserver-port uint16

API Server Port (default 6443)

--ca-certificate-validity-period uint16

CA Certificate Validity Period (default 20)

--calico-typha-ip string

Calico Typha IP (default “10.32.0.5”)

--ceph-cluster-name string

Ceph Cluster Name (default “ceph”)

--ceph-expected-number-of-objects uint

Ceph Expected Number of Objects (default 1000000)

--ceph-placement-groups uint

Ceph Placement Groups (default 32)

--client-certificate-validity-period uint16

Client Certificate Validity Period (default 15)

--cluster-cidr string

Cluster CIDR (default “10.200.0.0/16”)

--cluster-dns-ip string

Cluster DNS IP (default “10.32.0.10”)

--cluster-domain string

Cluster domain (default “cluster.local”)

--cluster-ip-range string

Cluster IP range (default “10.32.0.0/24”)

--cluster-name string

Cluster Name used for Kubernetes Dashboard (default “k8s-tew”)

--controller-virtual-ip string

Controller Virtual/Floating IP for the cluster

--controller-virtual-ip-interface string

Controller Virtual/Floating IP interface for the cluster

--deployment-directory string

Deployment directory (default “/”)

--drain-grace-period-seconds uint16

Drain Grace Period in Seconds

--elasticsearch-count uint16

Number of Elasticsearch Servers (default 1)

--elasticsearch-size uint16

Size of Elasticsearch Persistent Volume (default 10)

--email string

Email address used for example for Let’s Encrypt (default “k8s-tew@gmail.com”)

--grafana-size uint16

Size of Grafana Persistent Volume (default 2)

--help

help for configure

--ingress-domain string

Ingress domain name (default “k8s-tew.net”)

--kube-state-metrics-count uint16

Number of Kube State Metrics Servers (default 1)

--kubernetes-dashboard-port uint16

Kubernetes Dashboard Port (default 32443)

--load-balancer-port uint16

Load Balancer Port (default 32443)

--max-pods uint16

MaxPods (default 110)

--metallb-addresses string

Comma separated MetalLB address ranges and CIDR (e.g 192.168.0.16/28,192.168.0.75-192.168.0.100) (default “192.168.0.16/28”)

--minio-size uint16

Size of Minio Persistent Volume (default 2)

--prometheus-size uint16

Size of Prometheus Persistent Volume (default 2)

--public-network string

Public Network (default “192.168.100.0/24”)

--resolv-conf string

Custom resolv.conf (default “/etc/resolv.conf”)

--rsa-key-size uint16

RSA Key Size (default 2048)

--san-dns-names string

SAN DNS Names (comma separated)

--san-ip-addresses string

SAN IP Addresses (comma separated)

--version-alert-manager string

Alert Manager version (default “quay.io/prometheus/alertmanager:v0.21.0”)

--version-busybox string

Busybox version (default “docker.io/library/busybox:1.36.1”)

--version-calico-cni string

Calico CNI version (default “quay.io/calico/cni:v3.27.3”)

--version-calico-kube-controllers string

Calico Kube Controllers version (default “quay.io/calico/kube-controllers:v3.27.3”)

--version-calico-node string

Calico Node version (default “quay.io/calico/node:v3.27.3”)

--version-calico-typha string

Calico Typha version (default “quay.io/calico/typha:v3.27.3”)

--version-ceph string

Ceph version (default “quay.io/ceph/ceph:v18.2.2”)

--version-cerebro string

Cerebro version (default “docker.io/lmenezes/cerebro:0.9.3”)

--version-cert-manager-acme-solver string

Cert Manager ACME Solver version (default “quay.io/jetstack/cert-manager-acmesolver:v1.14.5”)

--version-cert-manager-cainjector string

Cert Manager CA Injector version (default “quay.io/jetstack/cert-manager-cainjector:v1.14.5”)

--version-cert-manager-controller string

Cert Manager Controller version (default “quay.io/jetstack/cert-manager-controller:v1.14.5”)

--version-cert-manager-ctl string

Cert Manager Ctl version (default “quay.io/jetstack/cert-manager-ctl:v1.9.1”)

--version-cert-manager-startup-api-check string

Cert Manager Startup API Check version (default “quay.io/jetstack/cert-manager-startupapicheck:v1.14.5”)

--version-cert-manager-webhook string

Cert Manager Web Hook version (default “quay.io/jetstack/cert-manager-webhook:v1.14.5”)

--version-containerd string

Containerd version (default “1.7.16”)

--version-coredns string

CoreDNS version (default “docker.io/coredns/coredns:1.11.1”)

--version-crictl string

CriCtl version (default “1.30.0”)

--version-csi-attacher string

CSI Attacher version (default “registry.k8s.io/sig-storage/csi-attacher:v4.5.0”)

--version-csi-ceph-plugin string

CSI Ceph Plugin version (default “quay.io/cephcsi/cephcsi:v3.11.0”)

--version-csi-driver-registrar string

CSI Driver Registrar version (default “registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0”)

--version-csi-provisioner string

CSI Provisioner version (default “registry.k8s.io/sig-storage/csi-provisioner:v4.0.0”)

--version-csi-resizer string

CSI Resizer version (default “registry.k8s.io/sig-storage/csi-resizer:v1.10.0”)

--version-csi-snapshot-controller string

CSI Snapshot Controller version (default “registry.k8s.io/sig-storage/snapshot-controller:v7.0.1”)

--version-csi-snapshot-validation-webhook string

CSI Snapshot Validation Webhook version (default “registry.k8s.io/sig-storage/snapshot-validation-webhook:v7.0.1”)

--version-csi-snapshotter string

CSI Snapshotter version (default “registry.k8s.io/sig-storage/csi-snapshotter:v7.0.0”)

--version-elasticsearch string

Elasticsearch version (default “docker.elastic.co/elasticsearch/elasticsearch:7.11.2”)

--version-etcd string

Etcd version (default “quay.io/coreos/etcd:v3.5.13”)

--version-fluent-bit string

Fluent-Bit version (default “docker.io/fluent/fluent-bit:1.7.2”)

--version-gobetween string

Gobetween version (default “docker.io/yyyar/gobetween:0.8.0”)

--version-grafana string

Grafana version (default “docker.io/grafana/grafana:7.4.3”)

--version-helm string

Helm version (default “3.14.4”)

--version-k8s string

Kubernetes version (default “v1.30.0”)

--version-kibana string

Kibana version (default “docker.elastic.co/kibana/kibana:7.11.2”)

--version-kube-apiserver string

Kubernetes API Server version (default “registry.k8s.io/kube-apiserver:v1.30.0”)

--version-kube-controller-manager string

Kubernetes Controller Manager (default “registry.k8s.io/kube-controller-manager:v1.30.0”)

--version-kube-proxy string

Kubernetes Proxy version (default “registry.k8s.io/kube-proxy:v1.30.0”)

--version-kube-scheduler string

Kubernetes Scheduler (default “registry.k8s.io/kube-scheduler:v1.30.0”)

--version-kube-state-metrics string

Kube State Metrics version (default “registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.12.0”)

--version-kubernetes-dashboard-api string

Kubernetes Dashboard API version (default “docker.io/kubernetesui/dashboard-api:1.5.0”)

--version-kubernetes-dashboard-auth string

Kubernetes Dashboard Auth version (default “docker.io/kubernetesui/dashboard-auth:1.1.3”)

--version-kubernetes-dashboard-kong string

Kubernetes Dashboard Kong version (default “docker.io/library/kong:3.6”)

--version-kubernetes-dashboard-metrics-scraper string

Kubernetes Dashboard Metrics Scraper version (default “docker.io/kubernetesui/dashboard-metrics-scraper:1.1.1”)

--version-kubernetes-dashboard-web string

Kubernetes Dashboard Web version (default “docker.io/kubernetesui/dashboard-web:1.3.0”)

--version-metallb-controller string

MetalLB Controller version (default “quay.io/metallb/controller:v0.9.5”)

--version-metallb-speaker string

MetalLB Speaker version (default “quay.io/metallb/speaker:v0.9.5”)

--version-metrics-server string

Metrics Server version (default “registry.k8s.io/metrics-server/metrics-server:v0.7.1”)

--version-minio-client string

Minio client version (default “docker.io/minio/mc:RELEASE.2021-03-12T03-36-59Z”)

--version-minio-server string

Minio server version (default “docker.io/minio/minio:RELEASE.2021-03-12T00-00-47Z”)

--version-mysql string

MySQL version (default “docker.io/library/mysql:8.0.23”)

--version-nginx-ingress-admission-webhook string

Nginx Ingress Admission Webhook version (default “registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.1”)

--version-nginx-ingress-controller string

Nginx Ingress Controller version (default “registry.k8s.io/ingress-nginx/controller:v1.10.1”)

--version-node-exporter string

Node Exporter version (default “quay.io/prometheus/node-exporter:v1.7.0”)

--version-pause string

Pause version (default “registry.k8s.io/pause:3.9”)

--version-prometheus string

Prometheus version (default “quay.io/prometheus/prometheus:v2.22.0”)

--version-runc string

Runc version (default “1.1.12”)

--version-velero string

Velero version (default “docker.io/velero/velero:v1.9.0”)

--version-velero-plugin-aws string

Velero Plugin AWS version (default “docker.io/velero/velero-plugin-for-aws:v1.6.1”)

--version-velero-plugin-csi string

Velero Plugin CSI version (default “docker.io/velero/velero-plugin-for-csi:v0.4.2”)

--version-velero-restic-restore-helper string

Velero Restic Restore Helper (default “docker.io/velero/velero-restic-restore-helper:v1.5.3”)

--version-virtual-ip string

Virtual-IP version (default “docker.io/darxkies/virtual-ip:1.0.1”)

--version-wordpress string

WordPress version (default “docker.io/library/wordpress:5.7.0-apache”)

--vip-raft-controller-port uint16

VIP Raft Controller Port (default 16277)

--vip-raft-worker-port uint16

VIP Raft Worker Port (default 16728)

--worker-virtual-ip string

Worker Virtual/Floating IP for the cluster

--worker-virtual-ip-interface string

Worker Virtual/Floating IP interface for the cluster

The email and the ingress-domain parameters need to be changed if you want a working Ingress and Lets’ Encrypt configuration. It goes like this:

k8s-tew configure --email john.doe@gmail.com --ingress-domain example.com

Another important argument is --resolv-conf which is used to define which resolv.conf file should be used for DNS.

The Virtual/Floating IP parameters should be accordingly changed if you want true HA. k8s-tew uses internally RAFT and its leader election functionality to select one node on which the Virtual IP is set. If the leader fails, one of the remaining nodes gets the Virtual IP assigned.

Add Remote Node

A remote node can be added with the following command:

k8s-tew node-add -n controller00 -i 192.168.100.100 -l controller,node,storage

The arguments:

-x, --index uint

The unique index of the node which should never be reused; if it is already in use a new one is assigned

-i, --ip string

IP of the node (default “192.168.100.50”)

-l, --labels string

The labels of the node which define the attributes of the node (default “controller,worker”)

-n, --name string

The hostname of the node (default “single-node”)

-s, --self

Add this machine by inferring the host’s name & IP and by setting the labels controller,worker,bootstrapper - The public-network and the deployment-directory are also updated

-r, --storage-index uint

The unique index of the storage node which should never be reused; if it is already in use a new one is assigned

Note

Make sure the IP address of the node matches the public network set using the configuration argument --public-network.

Add Local Node

k8s-tew is also able to start a cluster on the local computer and for that the local computer has to be added as a node:

k8s-tew node-add -s

The arguments:

-s, --self

Add this machine by inferring the host’s name & IP and by setting the labels controller,worker,bootstrapper - The public-network and the deployment-directory are also updated

Remove Node

A node can be removed like this:

k8s-tew node-remove -n controller00

List Nodes

And all the nodes can be listed with the command:
k8s-tew node-list

Generating Files

Once all the nodes were added, the required files (third party binares, certificates, kubeconfigs and so on) have to be put in place. And this goes like this:

k8s-tew generate

The arguments:

-r, --command-retries uint

The count of command retries during the setup (default 300)

--force-download

Force downloading all binary dependencies from the internet

--parallel

Download binary dependencies in parallel

--pull-images

Pull and convert images to OCI to be deployed later on

Run

With this command the local cluster can be started:

k8s-tew run

Note

This command will run in the foreground and it will supervise all the programs it started in the background.

Deploy

In case remote nodes were added with the deploy command, the remotely missing files are copied to the nodes. k8s-tew is installed and started as a service.

The deployment is executed with the command:

k8s-tew deploy

The arguments:

-r, --command-retries uint

The number of command retries during the setup (default 1200)

--force-upload

Files are uploaded without checking if they are already installed

-h, --help

help for deploy

-i, --identity-file string

SSH identity file (default “/home/darxkies/.ssh/id_rsa”)

--import-images

Install images

--parallel

Run steps in parallel

--skip-backup-setup

Skip backup setup

--skip-ingress-setup

Skip ingress setup

--skip-logging-setup

Skip logging setup

--skip-monitoring-setup

Skip monitoring setup

--skip-restart

Skip restart steps

--skip-setup

Skip setup steps

--skip-showcase-setup

Skip showcase setup

--skip-storage-setup

Skip storage setup and all other feature setup steps

--skip-upload

Skip upload steps

--wait uint

Wait for all cluster relevant pods to be ready and jobs to be completed. The parameter reflects the number of seconds in which the pods have to run stable.

The files are copied using scp and the ssh private key $HOME/.ssh/id_rsa. In case the file $HOME/.ssh/id_rsa does not exist it should be generated using the command ssh-keygen. If another private key should be used, it can be specified using the command line argument -i.

Note

The argument --pull-images downloads the required Docker Images on the nodes, before the setup process is executed. That could speed up the whole setup process later on. Furthermore, by using --parallel the process of uploading files to the nodes and the download of Docker Images can be again considerable shortened. Use these parameters with caution, as they can starve your network.

Environment

After starting the cluster, the user will need some environment variables set locally to make the interaction with the cluster easier. This is done with this command:

source <(k8s-tew environment)

This command sets KUBECONFIG needed by kubectl to communicate with the cluster and it also updates PATH to point to the downloaded third-party binaries.

Dashboards & Websites

Depending on the configuration of the cluster, the installation of all containers can take a while. Once everything is installed, the following command can be used to display the credentials (username + generated password) and open the web browser (by passing -o) pointing to the web sites hosted by the cluster:

k8s-tew dashboard XXX

XXX is a place holder and can be replaced with one of the following values:

ceph-manager - Display Ceph Manager website related information

ceph-rados-gateway - Display Ceph Rados Gateway website related information

cerebro - Display Cerebro website related information

grafana - Display Grafana website related information

kibana - Display Kibana website related information

kubernetes - Display Kuberenetes Dashboard website related information

minio - Display Minio website related information

wordpress-ingress - Display WordPress Ingress website related information

wordpress-nodeport - Display WordPress Nodeport website related information

Ingress

For working Ingress make sure ports 80 and 443 are available. The Ingress Domain have to be also configured before ‘generate’ and ‘deploy’ are executed:

k8s-tew configure --ingress-domain [ingress-domain]