Banzai Cloud Logo Close
Home Products Benefits Blog Company Contact
Sign in
Author Tamas Ecsedi

Installing and upgrading Pipeline Kubernetes Engine's multi-master version

At Banzai Cloud we blog mostly about our container management platform, Pipeline. We frequently gloss over the workhorse underneath Pipeline, our CNCF certified Kubernetes distribution, PKE. That’s because our customers usually install and manage PKE with Pipeline, and thus benefit from all the enterprise-grade features we’ve already made available on that platform. However, PKE is also available on its own, and is one of the simplest ways to kickstart a Kubernetes cluster across multiple supported environments.

Just like with Pipeline, we battle tested PKE against the CIS Kubernetes benchmark. PKE is designed to work on any cloud, VM or even on bare metal nodes, and is often our customers’ preferred Kubernetes distribution, even when cloud providers offer alternative management solutions. There are several reasons for this, but, if we’re limiting ourselves to the realm of security, customers prefer PKE because it employs strong defaults for its pod security and network policies.

PKE is available in single node, multi node and single/multi master versions. However, in this post we’ll be focusing on its multi-master version and its version upgrade capabilities.

Note that while there will be lots of CLI commands in the upcoming sections. The CLI machinery behind the scenes - whether it pertains to multi-master installations or to seamless upgrades - is entirely automated with Pipeline. The overall goal of this post is to highlight how you can still accomplish these tasks even without the Pipeline platform.

Pipeline

Just what are we doing here, exactly?

We’ll be installing a multi-master PKE cluster and upgrading it, on the fly, using virtual machines. Since (at time of writing) the two Kubernetes versions we support are v1.15 and v1.14, we’ll be starting with the latter, and upgrading to the former. Note that Banzai Cloud-managed Kubernetes clusters are constantly being upgraded, patched and tested, so that our customers can focus exclusively on the application side of their containerized experience, and leave the heavy lifting to us.

This post serves as a guide through the step-by-step installation of PKE using either a local machine (Vagrant) or AWS EC2. The differences will be noted within each step. (local machine, AWS EC2 or common commands respectively).

We’ll be doing the following:

  • Launching four machines (three master nodes + one worker node)
  • Installing PKE Kubernetes 1.14
  • Upgrading PKE Kubernetes to 1.15

PKE manual upgrade

Local machine

A four node cluster of virtual machines will be launched on our host machine. The first three of these nodes will serve as master nodes, and the final node will be our worker (we could, of course, add as many workers nodes as we need).

Please be prepared for your virtual machines to consume eight CPU cores and eight GiB of memory. If your machine is low on resources, you may need to edit Vagrantfile; master Kubernetes nodes usually require at least two CPU cores.

What are we not doing?

This post will not cover how to add a load-balancer in front of our master nodes. Stay tuned, though, as that topic will be the subject of an upcoming blog post. Also we will not be covering how we to seamlessly upgrade, snapshot (and recover) etcd; for the purposes of this post, it’s enough to bear in mind that these features are covered and automated by the Pipeline platform, and that PKE clusters install or can connect to existing single/multi node etcd clusters.

Note that this is not meant to be a production ready setup, it just serves as a demonstration: additional security, storage, cluster backups, automated security scans, and lots lots more are required for a production ready Kubernetes cluster (again, all these come out-of-the-box with Kubernetes clusters launched on Pipeline.

Prerequisites

Local machine

Here, we’ll cover the installation of Vagrant on a MacOS machine. For different OSes, please take a look at the official instructions.

Install VirtualBox, VirtualBox Extension Pack and Vagrant.

brew cask install virtualbox
brew cask install vagrant
brew cask install virtualbox-extension-pack
vagrant plugin install vagrant-vbguest

Check out the PKE source code.

git clone git@github.com:banzaicloud/pke.git
cd pke

You can download the latest PKE build or you can build it yourself.

curl -Lv https://banzaicloud.com/downloads/pke/pke-latest -o build/pke
chmod +x build/pke

Alternative manual build process. make pke-linux builds the project and cp build/pke-linux build/pke puts it in place so the rest of the commands apply.

Once PKE is in place we can proceed to cluster installation.

AWS EC2

We’ll assume that the AWS CLI is already installed and configured. If you don’t have the AWS CLI installed, please follow these instructions.

OK, now that we’ve finished with the prerequisites, let’s move on to creating our Kubernetes cluster.

PKE cluster installation

Local machine

First, we need to fire up four virtual machines.

vagrant up

This will take a few minutes depending on your internet connection, so grab a coffee and take a short break. Once all the machines are up and running, we can start to bring up our PKE cluster. Let’s proceed to the installation of our first master node.

AWS EC2

I will not go into detail about each and every command, but feel free to check them out in the corresponding PKE documentation.

First, you need to create AWS instance profiles and roles.

export CLUSTER_NAME="banzaicloud-pke"
export AWS_SSH_KEY_PAIR_NAME="pke-ssh-key"
export AWS_DEFAULT_REGION="eu-central-1"
export IMAGE_ID="ami-0dc9154691d8a1757" # for other regions check: https://github.com/banzaicloud/pipeline/blob/0.25.1/internal/providers/pke/pkeworkflow/create_cluster.go#L87

aws cloudformation create-stack \
--stack-name pke-global \
--capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
--template-body https://raw.githubusercontent.com/banzaicloud/pipeline/master/templates/pke/global.cf.yaml

Wait for the stack to finish being created; run the following command until you see the status, CREATE_COMPLETE. This may take a few minutes.

$ aws cloudformation describe-stacks --stack-name pke-global --query "Stacks[*].StackStatus"
[
    "CREATE_COMPLETE"
]

Set up the security group.

aws ec2 create-security-group --group-name pke-cluster --description "PKE security group"
aws ec2 authorize-security-group-ingress --group-name pke-cluster --protocol tcp --port 22 --cidr 0.0.0.0/0
PKE_CLUSTER_SEC_GROUP_ID=$(aws ec2 describe-security-groups --group-name pke-cluster --region $AWS_DEFAULT_REGION --query "SecurityGroups[*].GroupId" --output=text)
aws ec2 authorize-security-group-ingress --group-name pke-cluster --source-group $PKE_CLUSTER_SEC_GROUP_ID --protocol -1

aws ec2 create-security-group --group-name pke-master --description "PKE master security group"
aws ec2 authorize-security-group-ingress --group-name pke-master --protocol tcp --port 6443 --cidr 0.0.0.0/0

Import your SSH key and start the machines.

aws ec2 import-key-pair --key-name $AWS_SSH_KEY_PAIR_NAME --public-key-material file://~/.ssh/id_rsa.pub

aws ec2 run-instances --image-id $IMAGE_ID \
--count 3 \
--instance-type c5.large \
--key-name $AWS_SSH_KEY_PAIR_NAME \
--tag-specifications "ResourceType=instance,Tags=[{Key=kubernetes.io/cluster/$CLUSTER_NAME,Value=owned},{Key=Name,Value=pke-master}]" \
--security-groups pke-cluster pke-master \
--iam-instance-profile Name=pke-global-master-profile

aws ec2 run-instances --image-id $IMAGE_ID \
--count 1 \
--instance-type c5.large \
--key-name $AWS_SSH_KEY_PAIR_NAME \
--tag-specifications "ResourceType=instance,Tags=[{Key=kubernetes.io/cluster/$CLUSTER_NAME,Value=owned},{Key=Name,Value=pke-worker}]" \
--security-groups pke-cluster \
--iam-instance-profile Name=pke-global-worker-profile

Using SSH, connect to each initiated EC2 instance and run the following commands as root:

Query master instances:

> $ aws ec2 describe-instances --filter="Name=tag:Name,Values=pke-master" --query="Reservations[*].Instances[*].{ID:InstanceId,PublicIP:PublicIpAddress,PrivateIP:PrivateIpAddress,State:State.Name}" --output=table
> ----------------------------------------------------------------------
> |                          DescribeInstances                         |
> +----------------------+---------------+------------------+----------+
> |          ID          |   PrivateIP   |    PublicIP      |  State   |
> +----------------------+---------------+------------------+----------+
> |  i-03e71469d404b05bf |  172.31.1.152 |  18.184.255.3    |  running |
> |  i-0fec42374d451be3b |  172.31.7.121 |  3.123.3.106     |  running |
> |  i-04260e07984eaa756 |  172.31.3.129 |  18.184.137.210  |  running |
> +----------------------+---------------+------------------+----------+
> ```
>
> SSH using PublicIP and gain root access.
>
> ```bash
> ssh centos@18.184.255.3
> sudo -s # root access
> ```

```bash
mkdir -p /banzaicloud
curl -vL https://banzaicloud.com/downloads/pke/pke-latest -o /banzaicloud/pke
chmod +x /banzaicloud/pke

export CLUSTER_NAME="banzaicloud-pke"
export EXTERNAL_IP=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)
export INTERNAL_IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)

export MAC=$(curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/)
export VPC_CIDR=$(curl -s http://169.254.169.254/latest/meta-data/network/interfaces/macs/$MAC/vpc-ipv4-cidr-block/)

First master node installation

Local Machine

Log into the first node.

vagrant ssh node1 -c 'sudo -s'

Now run the PKE installer:

/banzaicloud/pke install master \
--kubernetes-master-mode=ha \
--kubernetes-advertise-address=192.168.64.11:6443 \
--kubernetes-api-server=192.168.64.11:6443 \
--kubernetes-version=1.14.3

AWS EC2

Log into the first master node.

/banzaicloud/pke install master \
--kubernetes-master-mode=ha \
--kubernetes-advertise-address=${INTERNAL_IP} \
--kubernetes-api-server=${EXTERNAL_IP}:6443 \
--kubernetes-cloud-provider=aws \
--kubernetes-cluster-name=${CLUSTER_NAME} \
--kubernetes-infrastructure-cidr=${VPC_CIDR} \
--kubernetes-version=1.14.3

Get the internal ip of the first master node. We’ll use this later for APISERVER_IP.

echo $INTERNAL_IP
172.31.1.152

Common commands for both local setup, and AWS EC2

Let’s makes ure the first installed master node is up and running.

export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get node -o wide
NAME    STATUS   ROLES    AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
node1   Ready    master   55s   v1.14.3   192.168.64.11   <none>        CentOS Linux 7 (Core)   3.10.0-957.5.1.el7.x86_64   containerd://1.2.5

Once PKE is successfully installed, we need to distribute the generated certificates. Please note, /banzaicloud/ is a folder that’s shared between virtual machines. We use this fortuitous circumstance within Vagrant to distribute any necessary certificates. For AWS you have to manually scp certs.tgz to each of the other tow machines. In a production environment, you have to find your own way of doing this, e.g. using scp to copy certificates between machines. If you are using Pipeline to set up a multi-master PKE cluster, we do it for you. Also, note that we use Vault and our Bank-Vault project, behind the scenes.

# pack certificates
tar -cvzf /banzaicloud/certs.tgz \
  /etc/kubernetes/pki/*ca.* \
  /etc/kubernetes/pki/ca.* \
  /etc/kubernetes/pki/etcd/*ca.* \
  /etc/kubernetes/pki/*sa* \
  /etc/kubernetes/admission-control/encryption-provider-config.yaml

To join other nodes to the cluster, we need a token and a certificate hash so as to be able to verify their authenticity. Luckily, the installation has already created one for us, we just need to query it.

/banzaicloud/pke token list

This command will output a token and a certificate hash. We’ll save these for later use on additional nodes.

Second and third master node installation

Log into the second and third node.

Extract the shared certificates we got from the first node, and use the token and the certificate hash we just mentioned.

# extract certificates
tar -xvzf /banzaicloud/certs.tgz -C /
# copy token here from first master
export TOKEN="0qztey.dv0o99zdcljb5une"
# copy certificate hash here from first master
export CERTHASH="sha256:24034ed4322ee85c3ef1aa7b380d64d7931b994b7836bad2980110dc43a20db1"

Local machine

/banzaicloud/pke install master \
--kubernetes-master-mode=ha \
--kubernetes-version=1.14.3 \
--kubernetes-join-control-plane \
--kubernetes-node-token=$TOKEN \
--kubernetes-api-server-ca-cert-hash=$CERTHASH \
--kubernetes-api-server=192.168.64.11

AWS EC2

# replace IP with the value of the first master node's private IP address, as mentioned above
export APISERVER_IP="172.31.1.152"
/banzaicloud/pke install master \
--kubernetes-master-mode=ha \
--kubernetes-version=1.14.3 \
--kubernetes-join-control-plane \
--kubernetes-node-token=$TOKEN \
--kubernetes-api-server-ca-cert-hash=$CERTHASH \
--kubernetes-api-server=${APISERVER_IP} \
--kubernetes-advertise-address=${INTERNAL_IP} \
--kubernetes-cloud-provider=aws \
--kubernetes-cluster-name=${CLUSTER_NAME} \
--kubernetes-infrastructure-cidr=${VPC_CIDR}

Worker node installation

At this point, all our master nodes should be up and running. Now we’re ready to add a worker node to the cluster.

Log into the fourth node, which we’ll use as our worker.

Run PKE installer:

# copy token here from first master
export TOKEN="0qztey.dv0o99zdcljb5une"
# copy certificate hash here from first master
export CERTHASH="sha256:24034ed4322ee85c3ef1aa7b380d64d7931b994b7836bad2980110dc43a20db1"
/banzaicloud/pke install worker \
--kubernetes-version=1.14.3 \
--kubernetes-node-token=$TOKEN \
--kubernetes-api-server-ca-cert-hash=$CERTHASH \
--kubernetes-api-server=192.168.64.11

AWS EC2

To get the PublicIP of the worker, run:

> aws ec2 describe-instances --filter="Name=tag:Name,Values=pke-worker" --query="Reservations[*].Instances[*].{ID:InstanceId,PublicIP:PublicIpAddress,PrivateIP:PrivateIpAddress,State:State.Name}" --output=table
> ```

```bash
export APISERVER_IP="172.31.1.152"
/banzaicloud/pke install worker \
--kubernetes-version=1.14.3 \
--kubernetes-node-token=$TOKEN \
--kubernetes-api-server-ca-cert-hash=$CERTHASH \
--kubernetes-api-server=${APISERVER_IP} \
--kubernetes-cloud-provider=aws

Verifying the running cluster

We can check if all the nodes are joined to our Kubernetes cluster from our first node.

[root@node1 vagrant]$ kubectl get node -o wide
NAME    STATUS   ROLES    AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
node1   Ready    master   54m     v1.14.3   192.168.64.11   <none>        CentOS Linux 7 (Core)   3.10.0-957.5.1.el7.x86_64   containerd://1.2.5
node2   Ready    master   13m     v1.14.3   192.168.64.12   <none>        CentOS Linux 7 (Core)   3.10.0-957.5.1.el7.x86_64   containerd://1.2.5
node3   Ready    master   8m25s   v1.14.3   192.168.64.13   <none>        CentOS Linux 7 (Core)   3.10.0-957.5.1.el7.x86_64   containerd://1.2.5
node4   Ready    <none>   114s    v1.14.3   192.168.64.14   <none>        CentOS Linux 7 (Core)   3.10.0-957.5.1.el7.x86_64   containerd://1.2.5

Kubernetes version upgrade; PKE cluster upgrade!

Since we now have a cluster running on K8s 1.14.3, we can start to upgrade it to the latest version. At time of writing, only 1.15.0 is available.

The upgrade flow for a patch version upgrade is exactly the same. In the future, you need only to specify --kubernetes-version=1.15.1 after it has been made generally available.

First master node upgrade

/banzaicloud/pke upgrade master --kubernetes-version=1.15.0

Second and third master node upgrade

/banzaicloud/pke upgrade master --kubernetes-version=1.15.0 --kubernetes-additional-control-plane

Upgrading worker node

/banzaicloud/pke upgrade worker --kubernetes-version=1.15.0

Verifying running cluster

We can check if all the nodes are joined to our Kubernetes cluster from our first node.

[root@node1 vagrant]$ kubectl get node -o wide
NAME    STATUS   ROLES    AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
node1   Ready    master   14m     v1.15.0   192.168.64.11   <none>        CentOS Linux 7 (Core)   3.10.0-957.5.1.el7.x86_64   containerd://1.2.5
node2   Ready    master   10m     v1.15.0   192.168.64.12   <none>        CentOS Linux 7 (Core)   3.10.0-957.5.1.el7.x86_64   containerd://1.2.5
node3   Ready    master   8m7s    v1.15.0   192.168.64.13   <none>        CentOS Linux 7 (Core)   3.10.0-957.5.1.el7.x86_64   containerd://1.2.5
node4   Ready    <none>   4m26s   v1.15.0   192.168.64.14   <none>        CentOS Linux 7 (Core)   3.10.0-957.5.1.el7.x86_64   containerd://1.2.5

Conclusion

As you can see, upgrading an existing PKE Kubernetes cluster is as easy as issuing a single command on every node. You don’t need to mess with Kubernetes upgrades, because the hard work is done by PKE. Again, when using this in combination with the Pipeline platform even less work is required, since Banzai Cloud Pipeline already supports multi-master installations. If you are interested in trying this out for yourself, we’re running a free developer version of the platform, available at: https://beta.banzaicloud.io/

PKE upgrade with Pipeline

Resources - simplifying things even more

All the above examples can be run on your local machine with scripts that are included in the source repository.

For a single node installation:

./centos-single.sh v1.14.3
./centos-single-upgrade.sh v1.15.0

For multi-master setup as described in this blogpost:

./centos-multi.sh v1.14.3
./centos-multi-upgrade.sh v1.15.0

About PKE

PKE is a simple, lightweight Kubernetes installer and distribution. Designed to work anywhere, it’s the preferred means of managing Banzai Cloud’s cloud native application and devops platform, Pipeline. Banzai Cloud Pipeline supercharges the development, deployment and scaling of container-based applications, and includes native support for multi-, hybrid, and edge cloud environments.

About Pipeline

Banzai Cloud’s Pipeline provides a platform which allows enterprises to develop, deploy and scale container-based applications. It leverages best-of-breed cloud components, such as Kubernetes, to create a highly productive, yet flexible environment for developers and operations teams alike. Strong security measures—multiple authentication backends, fine-grained authorization, dynamic secret management, automated secure communications between components using TLS, vulnerability scans, static code analysis, CI/CD, etc.—are a tier zero feature of the Pipeline platform, which we strive to automate and enable for all enterprises.

About Banzai Cloud

Banzai Cloud is changing how private clouds are built: simplifying the development, deployment, and scaling of complex applications, and putting the power of Kubernetes and Cloud Native technologies in the hands of developers and enterprises, everywhere.

#multicloud #hybridcloud #BanzaiCloud

If you are interested in our technology and open source projects, follow us on GitHub, LinkedIn or Twitter:


Comments

comments powered by Disqus