Installing Openshift 4 on VMware

Installing Openshift 4 on Vmware is complex with many things to consider and services to configure

Troubleshooting an installation is more complex and usually all those errors comes from badly configured pre-requisites

So in order to make it a little easier we created a helper node with all services needed:

  • DNS
  • Apache serving ignition files
  • Haproxy for the load balancer
  • Terraform and all the tools
  • Ova template for the nodes
  • DHCPD for booting the VMs with temporal IPs, fixed IPs will be configured after rebooting

Let’s begin

Installing Ansible and requisites

yum install ansible python2-pyvmomi python-requests -y

ansible-galaxy install robertdebock.dns robertdebock.bootstrap robertdebock.core_dependencies robertdebock.dhcpd

Clone repo with helper node image and all playbooks

Please be aware this repo contains a OVF template of about 700Mb

git clone https://gitlab.com/Giondo/helper-node-ocp4.git

Modify all vars used to create all personalization

cd helper-node-ocp4/vars

Modify both files:

  • vmware-credentials.yaml
  • vmware-vars.yaml

btw: the OCP Pull Secret var you will need to get it from https://cloud.redhat.com/openshift/install/vsphere/user-provisioned you need to have a Red Hat account

Running Playbooks

ansible-playbook -i hosts/helper.inv main.yaml
  • I would recommend to execute this on a computer in the same network as VMware Vcenter is, you will be deploying a OVF template and this is a time consuming task over slow networks

Installing Openshift 4

ssh root@helpernode.domain 
cd ocp4-install/gojeaqui/upi/vsphere/
terraform plan
terraform apply 
  • root password is on the comments of the VM deployed

That’s it all the nodes will boot and configure it

Checking status of the deployment

ssh root@helpernode.domain 
[root@helper01 root] cd ocp4-install/ignition/
[root@helper01 ignition] export KUBECONFIG=`pwd`/auth/kubeconfig
[root@helper01 ignition]
[root@helper01 ignition]# openshift-install wait-for bootstrap-complete   --log-level=debug
DEBUG OpenShift Installer 4.4.11
DEBUG Built from commit db69e0456f2f7d6b937a8e88fc1ee6be32bf61fd
INFO Waiting up to 20m0s for the Kubernetes API at https://api.ocp4.virtualinfra.online:6443...
INFO API v1.17.1+166b070 up
INFO Waiting up to 40m0s for bootstrapping to complete...
DEBUG Bootstrap status: complete
INFO It is now safe to remove the bootstrap resources

[root@helper01 ignition]# openshift-install wait-for install-complete   --log-level=debug
DEBUG OpenShift Installer 4.4.11
DEBUG Built from commit db69e0456f2f7d6b937a8e88fc1ee6be32bf61fd
DEBUG Fetching Install Config...
DEBUG Loading Install Config...
DEBUG   Loading SSH Key...
DEBUG   Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Cluster Name...
DEBUG     Loading Base Domain...
DEBUG     Loading Platform...
DEBUG   Loading Pull Secret...
DEBUG   Loading Platform...
DEBUG Using Install Config loaded from state file
DEBUG Reusing previously-fetched Install Config
INFO Waiting up to 30m0s for the cluster at https://api.ocp4.virtualinfra.online:6443 to initialize...
DEBUG Cluster is initialized
INFO Waiting up to 10m0s for the openshift-console route to be created...
DEBUG Route found in openshift-console namespace: console
DEBUG Route found in openshift-console namespace: downloads
DEBUG OpenShift console route is created
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/ocp4-install/ignition/auth/kubeconfig'
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.ocp4.virtualinfra.online
INFO Login to the console with user: kubeadmin, password: somescrambledtext

Useful commands

1: Will indicate the pods trying to get up, you can get the logs if something goes wrong

2: Cluster operators will be True you will see how they appear, authentication is one of the last ones

3: Get the logs of the pods indicating the namespace will help you troubleshoot

ssh root@helpernode.domain 
[root@helper01 root] cd ocp4-install/ignition/
[root@helper01 ignition] export KUBECONFIG=`pwd`/auth/kubeconfig
1. oc get pods --all-namespaces |egrep -v  "Running|Completed"
2. oc get clusteroperators
3. oc logs -f kube-controller-manager-master01.ocp4.virtualinfra.online -n openshift-kube-controller-manager

Approve Certs

Around the 90% of the installation you will need to approve some certs

[root@helper01 ignition]# oc get csr
NAME        AGE   REQUESTOR                                                                   CONDITION
csr-2qwcs   38m   system:node:master01.ocp4.virtualinfra.online                               Approved,Issued
csr-5qpcp   10m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-c2qrv   25m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued
csr-krxrc   38m   system:serviceaccount:openshift-machine-config-operator:node-bootstrapper   Approved,Issued

Approve them all

[root@helper01 ignition]# oc get csr -o name | xargs oc adm certificate approve

Operators

Example of deployed Operators

[root@helper01 ignition]# oc get clusteroperators
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.4.11    True        False         False      3m53s
cloud-credential                           4.4.11    True        False         False      55m
cluster-autoscaler                         4.4.11    True        False         False      40m
console                                    4.4.11    True        False         False      8m29s
csi-snapshot-controller                    4.4.11    True        False         False      9m33s
dns                                        4.4.11    True        False         False      47m
etcd                                       4.4.11    True        False         False      46m
image-registry                             4.4.11    True        False         False      44m
ingress                                    4.4.11    True        False         False      3m49s
insights                                   4.4.11    True        False         False      40m
kube-apiserver                             4.4.11    True        False         False      46m
kube-controller-manager                    4.4.11    True        False         False      45m
kube-scheduler                             4.4.11    True        False         False      46m
kube-storage-version-migrator              4.4.11    True        False         False      3m40s
machine-api                                4.4.11    True        False         False      40m
machine-config                             4.4.11    True        False         False      27m
marketplace                                4.4.11    True        False         False      32m
monitoring                                 4.4.11    True        False         False      8m52s
network                                    4.4.11    True        False         False      49m
node-tuning                                4.4.11    True        False         False      49m
openshift-apiserver                        4.4.11    True        False         False      37m
openshift-controller-manager               4.4.11    True        False         False      41m
openshift-samples                          4.4.11    True        False         False      40m
operator-lifecycle-manager                 4.4.11    True        False         False      46m
operator-lifecycle-manager-catalog         4.4.11    True        False         False      46m
operator-lifecycle-manager-packageserver   4.4.11    True        False         False      32m
service-ca                                 4.4.11    True        False         False      49m
service-catalog-apiserver                  4.4.11    True        False         False      50m
service-catalog-controller-manager         4.4.11    True        False         False      49m
storage                                    4.4.11    True        False         False      40m

Haproxy

Is always good to check the load balancer to see if there is some kind of network issue

Haproxy stats are enabled on port 9000 with authentication

http://{{HELPERNODE_IP}}:9000/

Hardware

What will you need:

  • VSphere 6.5 or newer
  • 3 Master nodes :: 16G RAM, 120GB HD, 4VCPU
  • 1 Worker (at least) :: 16G RAM, 120GB HD, 4VCPU
  • 1 Bootstrap Node (Can be deleted after installation) :: 16G RAM, 120GB HD, 4VCPU
  • Helper node (To run all services, can be deleted once all services are migrated) :: 8G RAM, 50GB HD, 2VCPU

Tested

Yes, this has been tested with Openshift 4.4.11 and VMware 6.5

Openshift 4 Installed

openshift-installed

Reference links: