Skip to content

How to create custom AMIs for TKG clusters using Image Builder

With the recent release of Tanzu Kubernetes Grid, TKG 1.3, the custom image building process has become a lot smoother. In previous versions image building required managing a bunch of pre-requisites, cloning the image-builder GitHub repository, installing required dependencies and a number of other steps. Although, even the earlier process, was a huge step up over dabbling with packer, ansible, ovftool, AMI builder etc. individually. That said, the new process is fairly less involved thanks to the introduction of image-builder docker images.

The entire process can now be completed in three easy steps:

  • Download and configure AWS CLI
  • Download Image Builder configuration files from VMware {code} Sample Exchange
  • Let the image-builder docker image do its magic

Download and configure AWS CLI

If you are using this process for the first time or starting off on a newly built jumpbox. Downloading and configuring the aws cli is one of the pre-requisites. Configuring the CLI is fairly simple, below is an example. Configure this with your credentials, region and default output format accordingly.

aws configure

AWS Access Key ID [None]: UPTBQ
AWS Secret Access Key [None]: HixMWvjL2Jx
Default region name [None]: us-west-2
Default output format [None]: json

Configuring the CLI will create the following files which are later referenced when building an AMI.

ls -lrth $HOME/.aws
total 8.0K
-rw------- 1 ubuntu ubuntu 116 Apr  7 21:24 credentials
-rw------- 1 ubuntu ubuntu  29 Apr  7 21:24 config

Download Image Builder configuration files

The Image Builder configuration files can be found at VMware {code} Sample Exchange. For this post I have opted to build AMI associated with Kubernetes v1.18.16.

On the VMware {code} Sample Exchange look for TKG-Image-Builder-for-Kubernetes-v1-18-16-master.zip

unzip TKG-Image-Builder-for-Kubernetes-v1-18-16-master.zip

# Docker command in the next step is run from this directory
cd TKG-Image-Builder-for-Kubernetes-v1-18-16-master/TKG-Image-Builder-for-Kubernetes-v1.18.16

tree .
├── goss
│   ├── amazon-amazon-linux-1.18.16+vmware.1-goss-spec.yaml
│   ├── amazon-ubuntu-1.18.16+vmware.1-goss-spec.yaml
│   ├── azure-ubuntu-1.18.16+vmware.1-goss-spec.yaml
│   ├── vsphere-photon-1.18.16+vmware.1-goss-spec.yaml
│   ├── vsphere-rhel-1.18.16+vmware.1-goss-spec.yaml
│   └── vsphere-ubuntu-1.18.16+vmware.1-goss-spec.yaml
├── tkg.json
├── tkgm
│   ├── defaults
│   │   └── main.yml
│   └── tasks
│       ├── amazon-2.yml
│       ├── aws.yml
│       ├── azure.yml
│       ├── main.yml
│       ├── ubuntu.yml
│       └── vsphere.yml

At a very high level, there are three artifacts present in this folder:

  • tkg.json - This contains information regarding
  • Kubernetes, cni and containerd versions and source
  • Location of custom ansible roles and goss spec files
  • tkgm folder - This contains custom ansible roles
  • goss folder - goss spec files for image validation
  • goss peeked my interest while exploring image-builder project. Will try to do a separate post on this later

Final Step - Building AMI

This can be done by simply running a single docker command. A key point to note, the user whose credentials were used to configure the aws CLI must have a default VPC pre-created in their default AWS region. The command is straight forward and runs a docker container using k8s.gcr.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.8.

docker run -it --rm \
-v $HOME/.aws:/home/imagebuilder/.aws \
-v $(pwd)/tkg.json:/home/imagebuilder/tkg.json \
-v $(pwd)/tkgm:/home/imagebuilder/tkgm \
-v $(pwd)/goss/amazon-ubuntu-1.18.16+vmware.1-goss-spec.yaml:/home/imagebuilder/goss/goss.yaml \
-v /home/ubuntu/imagebuilder/aws:/home/imagebuilder/output \
--env PACKER_VAR_FILES="tkg.json" \
k8s.gcr.io/scl-image-builder/cluster-node-image-builder-amd64:v0.1.8 \
build-ami-ubuntu-2004

Let's dive a bit into the Docker command parameters:

  • -v $HOME/.aws:/home/imagebuilder/.aws: Mounts credentials from your local machine to the docker container
  • -v $(pwd)/tkg.json:/home/imagebuilder/tkg.json: Adds TKG customization info to the container
  • -v $(pwd)/tkgm:/home/imagebuilder/tkgm: Mounts custom ansible roles into the image-builder container
  • -v $(pwd)/goss/amazon-ubuntu-1.18.16+vmware.1-goss-spec.yaml:/home/imagebuilder/goss/goss.yaml: Specifies the goss spec file used for image validation.
  • -v /home/ubuntu/imagebuilder/aws:/home/imagebuilder/output: Maps the output directory where the image cache will be written on the local machine
  • build-ami-ubuntu-2004: Instructs image builder to build ubuntu AMI

Once finished, you will see a list of AMI created in each region.

--> ubuntu-20.04: AMIs were created:
ap-northeast-1: ami-0c65b9e731025b9ef
ap-northeast-2: ami-0e10a3d845b9e492c
ap-south-1: ami-081bd5b9ef81abafe
ap-southeast-1: ami-039d55b9eaf48518f
ap-southeast-2: ami-055b9ef92dc30ac3d
ca-central-1: ami-0c0bc71c388d1c2eb
eu-central-1: ami-5b9e7f45597e5bfd5
eu-west-1: ami-0c26efa5b9e9557b3
eu-west-2: ami-05380e794655b9e4a
eu-west-3: ami-033586952295b9e40
sa-east-1: ami-0878757dd909a5b9e
us-east-1: ami-0bf332e07e7225b9e
us-east-2: ami-01b3365b9ec35c085
us-west-1: ami-019e885b9ed965b9e
us-west-2: ami-065b9ee14d6df5b9e