# Installing KVM, libvirt, Docker-CE and Kubernetes on Centos 7.x First of all be root while doing this... `sudo su` ### Set hostname Set the hostname: `hostnamectl set-hostname 'k8s-master'` ## Installing KVM ### Install KVM ``` yum install qemu-kvm qemu-img virt-manager libvirt libvirt-python libvirt-client virt-install virt-viewer bridge-utils systemctl start libvirtd systemctl enable libvirtd lsmod | grep kvm ``` If needed, install xwindows for use of graphical virt manager: ``` sudo yum groupinstall "GNOME Desktop" "Graphical Administration Tools" sudo ln -sf /lib/systemd/system/runlevel5.target /etc/systemd/system/default.target reboot` ``` Before Start creating VMs, let’s first create the bridge interface. Bridge interface is required if you want to access virtual machines from outside of your hypervisor network. ``` cd /etc/sysconfig/network-scripts/ cp ifcfg-eno1 ifcfg-br0 ``` Edit the Interface file and set followings: ``` [root@ network-scripts]# vi ifcfg-eno1 TYPE=Ethernet BOOTPROTO=static DEVICE=eno1 ONBOOT=yes BRIDGE=br0 ``` Edit the Bridge file (ifcfg-br0) and set the followings: ``` [root@ network-scripts]# vi ifcfg-br0 TYPE=Bridge BOOTPROTO=static DEVICE=br0 ONBOOT=yes ``` Replace the IP address and DNS server details as per your setup. Restart the network Service to enable the bridge interface. `systemctl restart network` Check the Bridge interface using below command: `ip addr show br0` ## Installing Docker and K8S ### Disable SELinux First we need to disable both SELinux and swap. Issue the following commands: ``` setenforce 0 sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/sysconfig/selinux ``` ### Disable swap Next, disable swap with the following command: `swapoff -a` We must also ensure that swap isn't re-enabled during a reboot on each server. Open up the `/etc/fstab` and comment out the swap entry like this: `# /dev/mapper/centos-swap swap swap defaults 0 0` `vi /etc/fstab` ### Configure the firewall At this moment just stop the firewall with: `systemctl stop firewalld` Or load the correct firewall rules for Kubernetes master in the firewall: ``` firewall-cmd --permanent --add-port=6443/tcp firewall-cmd --permanent --add-port=2379-2380/tcp firewall-cmd --permanent --add-port=10250/tcp firewall-cmd --permanent --add-port=10251/tcp firewall-cmd --permanent --add-port=10252/tcp firewall-cmd --permanent --add-port=10255/tcp firewall-cmd --reload ``` Or load the correct firewall rules for Kubernetes workers in the firewall: ``` firewall-cmd --permanent --add-port=10250/tcp firewall-cmd --permanent --add-port=10255/tcp firewall-cmd --permanent --add-port=30000-32767/tcp firewall-cmd --permanent --add-port=6783/tcp firewall-cmd --reload ``` ### Enable br_netfilter Enable the br_netfilter kernel module. This is done with the following commands: ``` modprobe br_netfilter echo '1' > /proc/sys/net/bridge/bridge-nf-call-iptables ``` ### Install Docker-ce Install the Docker-ce dependencies with the following command: `yum install -y yum-utils device-mapper-persistent-data lvm2` Next, add the Docker-ce repository with the command: ``` yum-config-manager --add-repo https://gist.githubusercontent.com/stolsma/12457f4db016a86fea631fecee419989/raw/ef412fc4e282a3d83fa216b0be53757ecf1edf37/docker-ce.repo yum update ``` Install Docker-ce with the command: `yum install -y docker-ce` Start docker automatically on reboot and also now: ``` systemctl start docker systemctl enable docker ``` ### Install Kubernetes First we need to create a repository entry for yum. To do this, issue the following command : ``` yum-config-manager --add-repo https://gist.githubusercontent.com/stolsma/12457f4db016a86fea631fecee419989/raw/ef412fc4e282a3d83fa216b0be53757ecf1edf37/kubernetes.repo yum update ``` Install Kubernetes with the command: `yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes` and start the k8s kubelet deamon with: ``` systemctl enable kubelet systemctl start kubelet ``` Once this part of the installation completes, you could reboot the machine. (TODO: but this is not needed?) ### Cgroup changes Now we need to ensure that both Docker-ce and Kubernetes belong to the same control group (cgroup). By default, Docker should already belong to cgroupfs (you can check this with the command `docker info | grep -i cgroup`). Add Kubernetes to this too, issue the command: `sed -i 's/cgroup-driver=systemd/cgroup-driver=cgroupfs/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf` Restart the systemd daemon and the kubelet service with the commands: ``` systemctl daemon-reload systemctl restart kubelet ``` ### Initialize the Kubernetes cluster We're now ready to initialize the Kubernetes cluster. This is done on kubemaster (and only on that machine). On kubemaster, issue the command (again, adjusting the IP addresses to fit your needs): `kubeadm init --apiserver-advertise-address= --pod-network-cidr=/` With flannel as CNI plugin: `kubeadm init --apiserver-advertise-address= --pod-network-cidr=10.244.0.0/16` When this completes (it'll take anywhere from 30 seconds to 5 minutes), the output should include the joining command for your nodes. Once that completes, head over to kube2 and issue the command (adjusting the IP address to fit your needs): `kubeadm join :6443 --token --discovery-token-ca-cert-hash sha256:` Where TOKEN and DISCOVERY_TOKEN_HASH are the tokens displayed after the initialization command completes. Those are only 24 hour valid! If you do not have the token, you can get it by running the following command on the master node: `kubeadm token list` The output is similar to this: ``` TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS 8ewj1p.9r9hcjoqgajrj4gi 23h 2018-06-12T02:51:28Z authentication, The default bootstrap system: signing token generated by bootstrappers: 'kubeadm init'. kubeadm: default-node-token ``` By default, tokens expire after 24 hours. If you are joining a node to the cluster after the current token has expired, you can create a new token by running the following command on the master node: `kubeadm token create` The output is similar to this: `5didvk.d09sbcov8ph2amjw` If you don’t have the value of `--discovery-token-ca-cert-hash`, you can get it by running the following command chain on the master node: ``` openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \ openssl dgst -sha256 -hex | sed 's/^.* //' ``` The output is similar to this: `8cb2de97839780a412b93877f8507ad6c94f73add17d5d7058e91741c9d5ec78` ### Configuring Kubernetes Before Kubernetes can be used, we must take care of a bit of configuration. Come out of root and issue the following three commands (to create a new .kube configuration directory, copy the necessary configuration file, and give the file the proper ownership): ``` mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` ### Deploy flannel network First set something in iptables to pass bridged IPv4 traffic to iptables’ chains `sysctl net.bridge.bridge-nf-call-iptables=1` Now we must deploy the flannel network to the cluster with the command: `kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml` ### Master Isolation By default, your cluster will not schedule pods on the master for security reasons. If you want to be able to schedule pods on the master, e.g. for a single-machine Kubernetes cluster for development, run: `kubectl taint nodes --all node-role.kubernetes.io/master-` With output looking something like: ``` node "test-01" untainted taint "node-role.kubernetes.io/master:" not found taint "node-role.kubernetes.io/master:" not found ``` This will remove the `node-role.kubernetes.io/master` taint from any nodes that have it, including the master node, meaning that the scheduler will then be able to schedule pods everywhere. ### Checking your nodes Once the deploy command completes, you should be able to see both nodes on the master, by issuing the command `kubectl get nodes` ### All ready Congratulations, you now have a Kubernetes cluster ready for pods. ### Remove Kubernetes from your system `sudo kubeadm reset`