Installing Openshift 4 on VMware
Installing Openshift 4 on Vmware is complex with many things to consider and services to configure
Troubleshooting an installation is more complex and usually all those errors comes from badly configured pre-requisites
So in order to make it a little easier we created a helper node with all services needed:
- DNS
- Apache serving ignition files
- Haproxy for the load balancer
- Terraform and all the tools
- Ova template for the nodes
- DHCPD for booting the VMs with temporal IPs, fixed IPs will be configured after rebooting
Let’s begin
Installing Ansible and requisites
yum install ansible python2-pyvmomi python-requests -y
ansible-galaxy install robertdebock.dns robertdebock.bootstrap robertdebock.core_dependencies robertdebock.dhcpd
Clone repo with helper node image and all playbooks
Please be aware this repo contains a OVF template of about 700Mb
git clone https://gitlab.com/Giondo/helper-node-ocp4.git
Modify all vars used to create all personalization
cd helper-node-ocp4/vars
Modify both files:
- vmware-credentials.yaml
- vmware-vars.yaml
btw: the OCP Pull Secret var you will need to get it from https://cloud.redhat.com/openshift/install/vsphere/user-provisioned you need to have a Red Hat account
Running Playbooks
ansible-playbook -i hosts/helper.inv main.yaml
- I would recommend to execute this on a computer in the same network as VMware Vcenter is, you will be deploying a OVF template and this is a time consuming task over slow networks
Installing Openshift 4
ssh [email protected]
cd ocp4-install/gojeaqui/upi/vsphere/
terraform plan
terraform apply
- root password is on the comments of the VM deployed
That’s it all the nodes will boot and configure it
Checking status of the deployment
ssh [email protected]
[root@helper01 root] cd ocp4-install/ignition/
[root@helper01 ignition] export KUBECONFIG=`pwd`/auth/kubeconfig
[root@helper01 ignition]
[root@helper01 ignition]# openshift-install wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.4.11
DEBUG Built from commit db69e0456f2f7d6b937a8e88fc1ee6be32bf61fd
INFO Waiting up to 20m0s for the Kubernetes API at https://api.ocp4.virtualinfra.online:6443...
INFO API v1.17.1+166b070 up
INFO Waiting up to 40m0s for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources
[root@helper01 ignition]# openshift-install wait-for install-complete --log-level=debug
DEBUG OpenShift Installer 4.4.11
DEBUG Built from commit db69e0456f2f7d6b937a8e88fc1ee6be32bf61fd
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG Loading SSH Key...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Cluster Name...
DEBUG Loading Base Domain...
DEBUG Loading Platform...
DEBUG Loading Pull Secret...
DEBUG Loading Platform...
DEBUG Using Install Config loaded from state file
DEBUG Reusing previously-fetched Install Config
INFO Waiting up to 30m0s for the cluster at https://api.ocp4.virtualinfra.online:6443 to initialize...
DEBUG Cluster is initialized
INFO Waiting up to 10m0s for the openshift-console route to be created...
DEBUG Route found in openshift-console namespace: console
DEBUG Route found in openshift-console namespace: downloads
DEBUG OpenShift console route is created
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/ocp4-install/ignition/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp4.virtualinfra.online
INFO Login to the console with user: kubeadmin, password: somescrambledtext
Useful commands
1: Will indicate the pods trying to get up, you can get the logs if something goes wrong
2: Cluster operators will be True you will see how they appear, authentication is one of the last ones
3: Get the logs of the pods indicating the namespace will help you troubleshoot
ssh [email protected]
[root@helper01 root] cd ocp4-install/ignition/
[root@helper01 ignition] export KUBECONFIG=`pwd`/auth/kubeconfig
1. oc get pods --all-namespaces |egrep -v "Running|Completed"
2. oc get clusteroperators
3. oc logs -f kube-controller-manager-master01.ocp4.virtualinfra.online -n openshift-kube-controller-manager
Approve Certs
Around the 90% of the installation you will need to approve some certs
[root@helper01 ignition]# oc get csr
NAME AGE REQUESTOR CONDITION
csr-2qwcs 38m system:node:master01.ocp4.virtualinfra.online Approved,Issued
csr-5qpcp 10m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-c2qrv 25m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
csr-krxrc 38m system:serviceaccount:openshift-machine-config-operator:node-bootstrapper Approved,Issued
````
Approve them all
```bash
[root@helper01 ignition]# oc get csr -o name | xargs oc adm certificate approve
Operators
Example of deployed Operators
[root@helper01 ignition]# oc get clusteroperators
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE
authentication 4.4.11 True False False 3m53s
cloud-credential 4.4.11 True False False 55m
cluster-autoscaler 4.4.11 True False False 40m
console 4.4.11 True False False 8m29s
csi-snapshot-controller 4.4.11 True False False 9m33s
dns 4.4.11 True False False 47m
etcd 4.4.11 True False False 46m
image-registry 4.4.11 True False False 44m
ingress 4.4.11 True False False 3m49s
insights 4.4.11 True False False 40m
kube-apiserver 4.4.11 True False False 46m
kube-controller-manager 4.4.11 True False False 45m
kube-scheduler 4.4.11 True False False 46m
kube-storage-version-migrator 4.4.11 True False False 3m40s
machine-api 4.4.11 True False False 40m
machine-config 4.4.11 True False False 27m
marketplace 4.4.11 True False False 32m
monitoring 4.4.11 True False False 8m52s
network 4.4.11 True False False 49m
node-tuning 4.4.11 True False False 49m
openshift-apiserver 4.4.11 True False False 37m
openshift-controller-manager 4.4.11 True False False 41m
openshift-samples 4.4.11 True False False 40m
operator-lifecycle-manager 4.4.11 True False False 46m
operator-lifecycle-manager-catalog 4.4.11 True False False 46m
operator-lifecycle-manager-packageserver 4.4.11 True False False 32m
service-ca 4.4.11 True False False 49m
service-catalog-apiserver 4.4.11 True False False 50m
service-catalog-controller-manager 4.4.11 True False False 49m
storage 4.4.11 True False False 40m
Haproxy
Is always good to check the load balancer to see if there is some kind of network issue
Haproxy stats are enabled on port 9000 with authentication
http://{{HELPERNODE_IP}}:9000/
Hardware
What will you need:
- VSphere 6.5 or newer
- 3 Master nodes :: 16G RAM, 120GB HD, 4VCPU
- 1 Worker (at least) :: 16G RAM, 120GB HD, 4VCPU
- 1 Bootstrap Node (Can be deleted after installation) :: 16G RAM, 120GB HD, 4VCPU
- Helper node (To run all services, can be deleted once all services are migrated) :: 8G RAM, 50GB HD, 2VCPU
Tested
Yes, this has been tested with Openshift 4.4.11 and VMware 6.5
Openshift 4 Installed
Reference links: