Skip to content

Building windows custom machine image for creating Tanzu Kubernetes Grid workload clusters

In this post, we will delve into building windows custom machine images. You can use this image later to create Tanzu Kubernetes Grid workload clusters. The instructions assume that you have already reviewed the pre-requisites.

High level the process:

Download Windows ISO

We will use a windows evaluation ISO from Microsoft Eval Center. It is neither recommended nor supported approach. It is used here for demo purposes only.

A recent (newer than April 2021) Windows Server 2019 ISO image. Download through your Microsoft Developer Network (MSDN) or Volume Licensing (VL) account. The use of evaluation media is not supported or recommended.

Download VMware Tools

Download the latest version from VMware Tools.

vmware_tools_iso_url='https://packages.vmware.com/tools/releases/latest/windows/VMware-tools-windows-11.3.5-18557794.iso'
wget -nv ${vmware_tools_iso_url}

Install govc

Install and setup govc

govc_url="https://github.com/vmware/govmomi/releases/download/v0.24.0/govc_linux_amd64.gz"
wget -nv ${govc_url}
gunzip govc_linux_amd64.gz
chmod +x govc_linux_amd64
sudo mv govc_linux_amd64 /usr/local/bin/govc

Configure Environment Variables

Setting these up is helpful as these values are used multiple times throughout the instructions

export VMWARE_TOOLS_ISO_NAME='VMware-tools-windows-11.3.5-18557794.iso'
export WINDOWS_ISO_NAME='17763.737.190906-2324.rs5_release_svc_refresh_SERVER_EVAL_x64FRE_en-us_1.iso'
export GOVC_URL=10.190.99.201
export GOVC_USERNAME=administrator@vsphere.local # vCenter username
export GOVC_PASSWORD=VMware\!23 # vCenter password
export GOVC_DATACENTER=Datacenter
export GOVC_CLUSTER=Cluster
export GOVC_DATASTORE=vsanDatastore
export GOVC_INSECURE=1
export GOVC_NETWORK='VM Network'

You can confirm if govc is setup correctly by using govc about.

Upload Windows ISO and VMware tools ISO to the datastore

This operation can take several minutes to complete as it uploads the Windows and VMware Tool ISOs to the datastore

govc datastore.mkdir iso
govc datastore.upload ${HOME}/${WINDOWS_ISO_NAME} iso/${WINDOWS_ISO_NAME}
govc datastore.upload ${HOME}/${VMWARE_TOOLS_ISO_NAME} iso/${VMWARE_TOOLS_ISO_NAME}

Deploy image-builder-resource-kit K8s deployment to management clusters

In this step we will apply the yaml below to our management clusters. This in turn creates a namespace, service and a deployment which hosts the binaries needed to Kubernetify your windows images. I really like this method of serving binaries as this reduces the effort that a user would have to spend getting these from multiple resources, verifying compatibility etc.

On my local system I have saved this yaml as builder.yaml

apiVersion: v1
kind: Namespace
metadata:
 name: imagebuilder
---
apiVersion: v1
kind: Service
metadata:
 name: imagebuilder-wrs
 namespace: imagebuilder
spec:
 selector:
   app: image-builder-resource-kit
 type: NodePort
 ports:
 - port: 3000
   targetPort: 3000
   nodePort: 30008
---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: image-builder-resource-kit
 namespace: imagebuilder
spec:
 selector:
   matchLabels:
     app: image-builder-resource-kit
 template:
   metadata:
     labels:
       app: image-builder-resource-kit
   spec:
     nodeSelector:
       kubernetes.io/os: linux
     containers:
     - name: windows-imagebuilder-resourcekit
       image: projects.registry.vmware.com/tkg/windows-resource-bundle:v1.22.9_vmware.1-tkg.1
       imagePullPolicy: Always
       ports:
         - containerPort: 3000

To deploy the objects that we discussed above, switch to the management cluster context

# Switch context to management cluster
kubectl config use-context mgmt-e-admin@mgmt-e

kubectl apply -f builder.yaml

Sample output

kubectl config use-context mgmt-e-admin@mgmt-e
Switched to context "mgmt-e-admin@mgmt-e".

kubectl apply -f builder.yaml
namespace/imagebuilder created
service/imagebuilder-wrs created
deployment.apps/image-builder-resource-kit created

# Verify that image-builder-resource-kit deployment is running
kubectl get all -n imagebuilder
NAME                                              READY   STATUS    RESTARTS   AGE
pod/image-builder-resource-kit-7f5cbc8cc9-6tc72   1/1     Running   0          86s

NAME                       TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/imagebuilder-wrs   NodePort   100.71.222.48   <none>        3000:30008/TCP   86s

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/image-builder-resource-kit   1/1     1            1           87s

NAME                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/image-builder-resource-kit-7f5cbc8cc9   1         1         1       87s

Set NODE_IP, this is used for validation and later used when configuring windows.json

# Used to configure windows.json later
export NODE_IP=$(kubectl get nodes -o jsonpath='{.items[*].status.addresses[?(@.type=="ExternalIP")].address}' --selector='!node-role.kubernetes.io/master' | awk '{print $1}')

You can also verify if the deployment is ready to serve the needed binaries

curl $NODE_IP:30008

{
    "containerd": {
        "version": "v1.5.11+vmware.1",
        "path": "files/containerd/cri-containerd-v1.5.11+vmware.1.windows-amd64.tar",
        "sha256": "ce2ceea7295e5cbd95b2773071d7004ca68dc5c91d688b0edfae176daf8077e5"
    },
    "antrea-windows": {
        "version": "v1.2.3+vmware.4-advanced",
        "path": "files/antrea-windows/antrea-windows-advanced.zip",
        "sha256": "5ced2e68f7ebaf79d56e12371cdb4734c3fa5300c1536630b095079a83b2c7c8"
    },
    "kubelet": {
        "version": "v1.22.9+vmware.1",
        "path": "files/kubernetes/kubelet.exe",
        "sha256": "62c5e547e01fbdc4347fad0917f4575ea7d884729509fac974a23c8e57916605"
    },
    "kubeadm": {
        "version": "v1.22.9+vmware.1",
        "path": "files/kubernetes/kubeadm.exe",
        "sha256": "dd78099643effeea60e0e86c585c81e7efe0e4b673da40b9f1c3913bee109915"
    },
    "kubectl": {
        "version": "v1.22.9+vmware.1",
        "path": "files/kubernetes/kubectl.exe",
        "sha256": "e2ea38bddea4d231dd5dc27b47aa2cd388d31798e2a64660a11fdca9524c40cd"
    },
    "kube-proxy": {
        "version": "v1.22.9+vmware.1",
        "path": "files/kubernetes/kube-proxy.exe",
        "sha256": "c763913cfb3220f6ce2cd55b5848a5989ff0aabb022011c5efc3c1a146bc0ff5"
    }
}

Configure windows.json

If you have been following the steps in this post than the only thing that needs update in the json file below is the property unattend_timezone. Please choose a value that suits your setup.

cat <<EOF > $HOME/windows.json
{
  "unattend_timezone": "UTC",
  "windows_updates_categories": "CriticalUpdates SecurityUpdates UpdateRollups",
  "kubernetes_semver": "v1.22.9",
  "cluster": "${GOVC_CLUSTER}",
  "template": "",
  "password": "${GOVC_PASSWORD}",
  "folder": "",
  "runtime": "containerd",
  "username": "${GOVC_USERNAME}",
  "datastore": "${GOVC_DATASTORE}",
  "datacenter": "${GOVC_DATACENTER}",
  "convert_to_template": "true",
  "vmtools_iso_path": "[${GOVC_DATASTORE}] iso/${VMWARE_TOOLS_ISO_NAME}",
  "insecure_connection": "true",
  "disable_hypervisor": "false",
  "network": "${GOVC_NETWORK}",
  "linked_clone": "false",
  "os_iso_path": "[${GOVC_DATASTORE}] iso/${WINDOWS_ISO_NAME}",
  "resource_pool": "${GOVC_RESOURCE_POOL}",
  "vcenter_server": "${GOVC_URL}",
  "create_snapshot": "false",
  "netbios_host_name_compatibility": "false",
  "kubernetes_base_url": "http://$NODE_IP:30008/files/kubernetes/",
  "containerd_url": "http://$NODE_IP:30008/files/containerd/cri-containerd-v1.5.11+vmware.1.windows-amd64.tar",
  "containerd_sha256_windows": "ce2ceea7295e5cbd95b2773071d7004ca68dc5c91d688b0edfae176daf8077e5",
  "pause_image": "mcr.microsoft.com/oss/kubernetes/pause:3.5",
  "prepull": "false",
  "additional_prepull_images": "mcr.microsoft.com/windows/servercore:ltsc2019",
  "additional_download_files": "",
  "additional_executables": "true",
  "additional_executables_destination_path": "c:/k/antrea/",
  "additional_executables_list": "http://$NODE_IP:30008/files/antrea-windows/antrea-windows-advanced.zip",
  "load_additional_components": "true"
}
EOF

Configure autounattend.xml

Download and Update autounattend.xml

curl https://vdc-download.vmware.com/sampleExchange/v1/downloads/7719 --output autounattend.xml

If you are using evaluation copy remove all occurrences of section ProductKey from autounattend.xml. If this is not done the windows boot will get stuck with the message on the UI No Image Available

Build windows custom machine image

docker run -it --rm --mount type=bind,source=$(pwd)/windows.json,target=/windows.json \
--mount type=bind,source=$(pwd)/autounattend.xml,target=/home/imagebuilder/packer/ova/windows/windows-2019/autounattend.xml \
-e PACKER_VAR_FILES="/windows.json" \
-e IB_OVFTOOL=1 \
-e IB_OVFTOOL_ARGS='--skipManifestCheck' \
-e PACKER_FLAGS='-force -on-error=ask' \
-e PACKER_LOG=1 \
-t projects.registry.vmware.com/tkg/image-builder:v0.1.11_vmware.3 \
build-node-ova-vsphere-windows-2019