Installing the Anjuna Kubernetes Toolset to your cluster

Configure your environment

Before installing the Anjuna Kubernetes Toolset to your cluster, make sure that the environment is correctly configured, as shown in previous sections.

Since each Anjuna Confidential Pod is deployed as a Confidential VM, specify the virtual machine size to be used. Currently, the following virtual machine families support Confidential Computing:

DCasv5/DCadsv5 family
ECasv5/ECadsv5 family

$ export AZURE_INSTANCE_SIZE="Standard_DC2as_v5"

Configure the environment with your cluster information. You can either use an existing AKS cluster or create a new one through Terraform. Select one of the tabs below according to your use case.

Create a new cluster
Use an existing cluster

If you do not have a cluster created already, you can use the cluster module included in the Anjuna Kubernetes Toolset installer bundle to create a new cluster, using the following commands.

First, create a new terraform.tfvars file with your desired configuration, such as public AKS version, Azure region, network prefix, and public SSH key path:

cat > ${WORKSPACE}/iac/azure/cluster/terraform.tfvars <<EOF
# Kubernetes version to use for the cluster
k8s_version = "1.28"

# Location where resources should be provisioned in Azure
location = "eastus"

# Network address prefix for installing AKS in. E.g: 10.1 or 10.2
# (It is a best practice to ensure that this range does not overlap
# with other subnets within the same Azure region)
base_network_address_prefix = "10.1"

# Path to the SSH public key to use to manage the cluster
ssh_public_key_path = "~/.ssh/id_rsa.pub"

# Prefix to use for all relevant resources created by this configuration
prefix = "anjunaakssev"

# The VM size for the cluster's system agent nodes
agents_size = "standard_d4lds_v5"

# Initial number of k8s system agent nodes
agents_count = 1

# Tags to use on the resources deployed with this configuration
tags = {}
EOF

To create your cluster, run:

$ terraform -chdir=${WORKSPACE}/iac/azure/cluster init
$ terraform -chdir=${WORKSPACE}/iac/azure/cluster apply

Note that creating an AKS cluster usually takes between 5-15 minutes.

After Terraform runs successfully, configure your environment to use the newly created cluster:

$ source ${WORKSPACE}/iac/azure/cluster/env.sh
$ az aks get-credentials \
  --resource-group "${AZURE_AKS_RG}" \
  --name "${AZURE_CLUSTER_NAME}" --admin

If you already have a cluster, ensure the environment is correctly configured to point to it, and skip the creation of the cluster entirely by running the following commands:

$ export AZURE_SSH_KEY="<path to your SSH key>"
$ export AZURE_AKS_WORKER_USER_NAME="<username for the worker node>"
$ export AZURE_CLUSTER_NAME="<cluster name>"
$ export AZURE_AKS_RG="<resource group>"
$ az aks get-credentials \
  --resource-group "${AZURE_AKS_RG}" \
  --name "${AZURE_CLUSTER_NAME}"

The AZURE_SSH_KEY and AZURE_AKS_WORKER_USER_NAME variables are needed to install the Anjuna Kubernetes Toolset to your cluster’s Nodes.

Retrieve the name of the virtual network and subnet used by the AKS cluster so that the Confidential Pods can be part of the same network:

$ export AZURE_VNET_NAME=$(az network vnet list \
  --resource-group "${AZURE_AKS_RG}" \
  --query "[0].name" \
  --output tsv)

$ export AZURE_SUBNET_ID=$(az network vnet subnet list \
  --resource-group "${AZURE_AKS_RG}" \
  --vnet-name "${AZURE_VNET_NAME}" \
  --query "[0].id" \
  --output tsv)

Load and push the Anjuna Kubernetes Toolset images

In this section, you will push the Anjuna Kubernetes Toolset images to a container registry, where they can be accessed by your Kubernetes cluster.

The Anjuna Kubernetes Toolset images include the Anjuna Cloud Adaptor and the Anjuna Custom Container Runtime components.

First, specify the name of the Anjuna Kubernetes Toolset images.

$ export REGISTRY="${AZURE_REGISTRY_NAME}.azurecr.io"
$ export TOOLSET_ADAPTOR_IMAGE="${REGISTRY}/anjuna-k8s-cc-toolset:cloud-adaptor-1.0.0002"
$ export TOOLSET_RUNTIME_IMAGE="${REGISTRY}/anjuna-k8s-cc-toolset:runtime-1.0.0002"

Make sure that you are authenticated and have permission to push to the chosen container registry. For example, if you are using Azure Container Registry (ACR), run the following command:

$ az acr login -n "${AZURE_REGISTRY_NAME}"

You should see output like Login Succeeded when the command completes.

The following commands load the images locally from the Anjuna Kubernetes Toolset installer bundle, and then push it to your chosen container registry. This might take a few seconds depending on the size of the images.

$ cd ${WORKSPACE}
$ docker load -i ${WORKSPACE}/anjuna-k8s-sev-adaptor-image.tar
$ docker tag anjuna-k8s-sev-adaptor ${TOOLSET_ADAPTOR_IMAGE}
$ docker push ${TOOLSET_ADAPTOR_IMAGE}
$ docker load -i ${WORKSPACE}/anjuna-k8s-sev-runtime-image.tar
$ docker tag anjuna-k8s-sev-runtime ${TOOLSET_RUNTIME_IMAGE}
$ docker push ${TOOLSET_RUNTIME_IMAGE}

Then, you must ensure that the cluster is able to pull the images from your container registry.

For example, if you are pushing the images to an Azure Container Registry (ACR), you can generate an authentication token as follows:

$ export ACR_TOKEN_NAME="acr-token-${RANDOM}"
$ export ACR_TOKEN=$(az acr token create -n ${ACR_TOKEN_NAME} \
  -r ${AZURE_REGISTRY_NAME} \
  --repository anjuna-k8s-cc-toolset content/read \
  | jq -r '.credentials.passwords[] | select(.name == "password1") | .value')

As part of the Anjuna Kubernetes Toolset installation, you can create a new image pull secret from the authentication token generated above, so that the cluster can pull the toolset images.

Alternatively, if you do not want to use registry tokens as illustrated above, you can attach the ACR to the AKS cluster instead (requires Owner permission).

Install the Anjuna Kubernetes Toolset

Label the Nodes of the cluster so that they can host the Anjuna Kubernetes Toolset:

$ kubectl label nodes --all node.kubernetes.io/worker=

Move to the iac folder, and install the needed Custom Resource Definition (CRD) for the Anjuna Kubernetes Toolset to your cluster:

$ cd ${WORKSPACE}/iac
$ kubectl apply -f k8s/crd.yaml
$ sed "s~ANJUNA_RUNTIME_IMAGE~$TOOLSET_RUNTIME_IMAGE~g" k8s/ccruntime.template.yaml > k8s/ccruntime.yaml
$ kubectl apply -f k8s/ccruntime.yaml

For regular Pods, the fields resources.requests and resources.limits of the Pod specification help the Kubernetes scheduler make placement decisions based on the capacity and constraints of each worker Node.

Anjuna Confidential Pods are deployed as standalone Confidential VMs and not as containers on the same worker Node. Therefore, a Anjuna Confidential Pod’s spec.resources could mislead the scheduler regarding the actual capacity of the worker Nodes.

To address this issue, the Anjuna Kubernetes Toolset includes a mutating webhook and a controller that adjust the resource requests and limits of an Anjuna Confidential Pod to be more accurate regarding Node resource allocation.

cert-manager is required for the mutating webhook, and can be installed as follows to the cluster:

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.9.1/cert-manager.yaml

Since each Anjuna Confidential Pod runs in its own confidential VM, the Anjuna Kubernetes Toolset limits the number of simultaneous Anjuna Confidential Pods to 20 per cluster. If you want to change this number, you can define it in the values.yaml file (see below) by setting maxConfidentialPodVMs to the desired value on line 10.

You will use a Helm Chart to install the Anjuna Kubernetes Toolset to your cluster. To configure it, create a values.yaml file with the following content:

cat > values.yaml <<EOF
sshPublicKey: ${AZURE_SSH_KEY}
cloud: azure
image: ${TOOLSET_ADAPTOR_IMAGE}
imagePullSecret:
  name: anjuna-kubernetes-toolset
  registry: ${AZURE_REGISTRY_NAME}.azurecr.io
  username: ${ACR_TOKEN_NAME}
  password: ${ACR_TOKEN}
maxConfidentialPodVMs: 20
azure:
  resourceGroup: ${AZURE_RESOURCE_GROUP}
  subnetId: ${AZURE_SUBNET_ID}
  subscriptionId: ${AZURE_SUBSCRIPTION_ID}
  instanceSize: "Standard_DC2as_v5"
  location: ${AZURE_LOCATION}
  storageAccount: ${AZURE_STORAGE_ACC_NAME}
  credentials:
    clientId: ${AZURE_CLIENT_ID}
    clientSecret: ${AZURE_CLIENT_SECRET}
    tenantId: ${AZURE_TENANT_ID}
EOF

Deploy the Anjuna Kubernetes Toolset to your cluster:

$ helm install anjuna-cc k8s/chart --values values.yaml --wait

Verify the installation

All resources are created in the anjuna-system namespace.

To ensure that all components are running, you can verify the Pods running in the anjuna-system namespace:

$ kubectl get pods -n anjuna-system

The output should include the following Pods:

NAME                                                    READY   STATUS    RESTARTS   AGE
cc-operator-controller-manager-68ff8494b7-bwcx9         2/2     Running   0          5m10s
cc-operator-daemon-install-w4t82                        1/1     Running   0          2m16s
cc-operator-pre-install-daemon-f4f96                    1/1     Running   0          2m16s
anjuna-cloud-adaptor-daemonset-z4kmg                    1/1     Running   0          4m54s
peer-pods-webhook-controller-manager-5d4675fc4b-6kjnn   2/2     Running   0          65s

Check that the Runtime Class anjuna-remote was added to the cluster:

$ kubectl get runtimeclass

The output should resemble the following. Note that the Runtime Class might take a couple of minutes to be created.

NAME            HANDLER         AGE
anjuna-remote   anjuna-remote   1m

Your Kubernetes cluster is now ready to deploy applications as Anjuna Confidential Pods. Refer to Deploying Pods as Anjuna Confidential Pods in AKS for examples on how to deploy applications.

Upgrade the Anjuna Kubernetes Toolset

To upgrade the installed version of the Anjuna Kubernetes Toolset, Uninstall the Anjuna Kubernetes Toolset, and follow the installation instructions in Load and push the Anjuna Kubernetes Toolset images again to install the new version.

You do not need to recreate the Shared Resources or the cluster in order to upgrade.

All Anjuna Confidential Pods must be stopped before an upgrade. After the upgrade, the Anjuna Confidential Pod images need to be rebuilt with an anjuna-k8s-cli that matches the new Anjuna Kubernetes Toolset version.

Uninstall the Anjuna Kubernetes Toolset

To uninstall the Anjuna Kubernetes Toolset, first stop all Anjuna Confidential Pods.

Then, run the following commands from the ${WORKSPACE}/iac folder:

$ cd "${WORKSPACE}/iac"
$ kubectl delete -f k8s/crd.yaml
$ helm uninstall anjuna-cc --wait
$ kubectl delete runtimeclass anjuna-remote --ignore-not-found

This operation might take about a minute to fully complete.

To delete cert-manager, run the following command:

$ kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/v1.9.1/cert-manager.yaml

Refer to the next section, Cleaning up resources, to see how to destroy the resources created by Terraform in your Azure subscription.