Skip to content

Enabling and troubleshooting etcd encryption in TKGI

This post describes how you can enable etcd encryption in you TKGI clusters using TKGI kubernetes profiles. Towards the end of the post we will look into how you can troubleshoot and verify your configuratio if you run into any issues while enabling etcd encryption.

Generate random secret

head -c 32 /dev/urandom | base64 -w
ByLszajcuB06/zeVnpUrLsJaVA+gRmXMVN1oKHcWt8w=

Create EncryptionConfiguration

Info

Typo in this configuration will lead to encryption errors

cat <<EOF > encryption-etcd-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
    - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: ByLszajcuB06/zeVnpUrLsJaVA+gRmXMVN1oKHcWt8w=
    - identity: {}
EOF

Create kubernetes profile to enable etcd encryption

  • "encryption-provider-config": "encryption-etcd-config.yaml" this parameter and file is read from the local directory and pushed to clusters. Make sure the file path is correct
cat <<EOF > encryption-compute-profile.json
{
  "name": "etcd-encryption",
  "description": "encrypts etcd secrets",
  "customizations": [
    {
      "component": "kube-apiserver",
      "arguments": {
      },
      "file-arguments": {
        "encryption-provider-config": "encryption-etcd-config.yaml"
      }
    }
  ]
}

Create and list kubernetes profile

tkgi create-k8s-profile encryption-compute-profile.json
Kubernetes profile etcd-encryption successfully created

tkgi kubernetes-profiles
Name             Description            Created Date
etcd-encryption  encrypts etcd secrets  Thu, 28 Jul 2022 07:12:23 UTC

tkgi kubernetes-profile etcd-encryption
Name:          etcd-encryption
Owner:         admin
Created Date:  Thu, 28 Jul 2022 07:12:23 UTC
Description:   encrypts etcd secrets
Customizations:

  Component:  kube-apiserver
  File Arguments:
      encryption-provider-config:
        apiVersion: apiserver.config.k8s.io/v1
        kind: EncryptionConfiguration
        resources:
          - resources:
            - secrets
            providers:
            - aescbc:
                keys:
                - name: key1
                  secret: ByLszajcuB06/zeVnpUrLsJaVA+gRmXMVN1oKHcWt8w=
            - identity: {}

Apply profile to existing TKGI cluster

tkgi update-cluster encr --kubernetes-profile etcd-encryption

Update summary for cluster encr:
Kubernetes Profile Name: etcd-encryption
Are you sure you want to continue? (y/n): y
Use 'pks cluster encr' to monitor the state of your cluster

Successful etcd encryption task

  • Controle plane nodes are restarted and not recreated
bosh task
Using environment '172.16.100.3' as client 'ops_manager'

Task 126

Task 126 | 07:16:34 | Deprecation: Global 'properties' are deprecated. Please define 'properties' at the job level.
Task 126 | 07:16:36 | Preparing deployment: Preparing deployment
Task 126 | 07:16:37 | Warning: DNS address not available for the link provider instance: pivotal-container-service/e22eef4e-7ad0-41d2-9ed1-8961fb4b60e9
Task 126 | 07:16:38 | Warning: DNS address not available for the link provider instance: pivotal-container-service/e22eef4e-7ad0-41d2-9ed1-8961fb4b60e9
Task 126 | 07:16:38 | Warning: DNS address not available for the link provider instance: pivotal-container-service/e22eef4e-7ad0-41d2-9ed1-8961fb4b60e9
Task 126 | 07:16:50 | Preparing deployment: Preparing deployment (00:00:14)
Task 126 | 07:16:50 | Preparing deployment: Rendering templates (00:00:09)
Task 126 | 07:16:59 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 126 | 07:17:00 | Updating instance master: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary)
Task 126 | 07:17:04 | L executing pre-stop: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary)
Task 126 | 07:17:04 | L executing drain: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary)
Task 126 | 07:17:07 | L stopping jobs: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary)
Task 126 | 07:18:06 | L executing post-stop: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary)
Task 126 | 07:18:23 | L installing packages: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary)
Task 126 | 07:18:26 | L configuring jobs: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary)
Task 126 | 07:18:27 | L executing pre-start: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary)
Task 126 | 07:19:07 | L starting jobs: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary)
Task 126 | 07:19:33 | L executing post-start: master/c4d1fd43-2803-4291-a577-1d640376e32d (0) (canary) (00:02:41)

Task 126 Started  Thu Jul 28 07:16:34 UTC 2022
Task 126 Finished Thu Jul 28 07:19:41 UTC 2022
Task 126 Duration 00:03:07
Task 126 done

Succeeded

Testing and verifying etcd encryption

k create secret generic test-secret --from-literal=name=tkgi
secret/test-secret created

k get secret test-secret -oyaml
apiVersion: v1
data:
  name: dGtnaQ==
kind: Secret
metadata:
  creationTimestamp: "2022-07-28T07:57:16Z"
  name: test-secret
  namespace: default
  resourceVersion: "71578"
  uid: 62698c3b-68ef-465b-bd6f-5ad81b9c1892
type: Opaque

export ETCDCTL_API=3
etcdctl get /registry/secrets/default/test-secret
/registry/secrets/default/test-secret
k8s:enc:aescbc:v1:key1:2U��
�b����&��5TD�n�lL� ����'*�V!��wc��go�r�[���D���fY�o� ���`ߕ=�����u��2��5K�Nu�g�

Verifying using bosh cli

bosh ssh -d service-instance_8e4c492e-daae-4e72-94fb-1f6430f7f4a7 master "ps -ef | grep -v tini | grep '\-\-encryption-provider-config'"

View configuration using bosh cli

bosh ssh -d service-instance_8e4c492e-daae-4e72-94fb-1f6430f7f4a7 master "sudo cat /var/vcap/jobs/kube-apiserver/config/encryption-provider-config"
Using environment '172.16.100.3' as client 'ops_manager'

Using deployment 'service-instance_8e4c492e-daae-4e72-94fb-1f6430f7f4a7'

Task 182. Done
master/c4d1fd43-2803-4291-a577-1d640376e32d: stderr | Unauthorized use is strictly prohibited. All access and activity
master/c4d1fd43-2803-4291-a577-1d640376e32d: stderr | is subject to logging and monitoring.
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout | apiVersion: apiserver.config.k8s.io/v1
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout | kind: EncryptionConfiguration
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout | resources:
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout |   - resources:
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout |     - secrets
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout |     providers:
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout |     - aescbc:
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout |         keys:
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout |         - name: key1
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout |           secret: ByLszajcuB06/zeVnpUrLsJaVA+gRmXMVN1oKHcWt8w=
master/c4d1fd43-2803-4291-a577-1d640376e32d: stdout |     - identity: {}
master/c4d1fd43-2803-4291-a577-1d640376e32d: stderr | Connection to 10.20.1.2 closed.

Succeeded

Troubleshooting

# Verify and collect profile configuration
tkgi kubernetes-profile etcd-encryption
bosh ssh -d service-instance_<> master "sudo cat /var/vcap/jobs/kube-apiserver/config/encryption-provider-config"

# Verify kube-apiserver has the following parameter `--encryption-provider-config=/var/vcap/jobs/kube-apiserver/config/encryption-provider-config`
bosh ssh -d service-instance_<> master "ps -ef | grep -v tini | grep '\-\-encryption-provider-config'"

# Collect bosh task debug logs
bosh task <failed task id> --debug

# Collect cluster logs
bosh logs -d service-instance-xxxxx

# Verify if secrets can be accessed and they are encrypted
export ETCDCTL_API=3
etcdctl get /registry/secrets/pks-system/fluent-bit

# Other helpful commands

# Get only keys for all etcd secrets
etcdctl get /registry/secrets --prefix --key-only | grep / > all-secrets-keys.out

Log and Configuration files

  • kube-apiserver logs and etcd logs will provide the best information
  • Encryption configuration /var/vcap/jobs/kube-apiserver/config/encryption-provider-config