Installing the Anjuna Kubernetes Toolset to your cluster

Configure your environment

Before installing the Anjuna Kubernetes Toolset to your cluster, make sure that the environment is correctly configured, as shown in previous sections.

Since each Anjuna Confidential Pod is deployed as a Confidential VM, specify the virtual machine size to be used. Currently, the following virtual machine families support Confidential Computing:

DCasv5/DCadsv5 family
ECasv5/ECadsv5 family

$ export AZURE_INSTANCE_SIZE="Standard_DC2as_v5"

Configure the environment with your cluster information. You can either use an existing AKS cluster or create a new one through Terraform. Select one of the tabs below according to your use case.

Create a new cluster
Use an existing cluster

If you do not have a cluster created already, you can use the cluster module included in the Anjuna Kubernetes Toolset installer bundle to create a new cluster, using the following commands.

First, create a new terraform.tfvars file with your desired configuration, such as public AKS version, Azure region, and network prefix:

cat > ${WORKSPACE}/iac/azure/cluster/terraform.tfvars <<EOF
# Kubernetes version to use for the cluster
k8s_version = "1.30"

# Location where resources should be provisioned in Azure
location = "eastus"

# Network address prefix for installing AKS in. E.g: 10.1 or 10.2
# (It is a best practice to ensure that this range does not overlap
# with other subnets within the same Azure region)
base_network_address_prefix = "10.1"

# Prefix to use for all relevant resources created by this configuration
prefix = "anjunaakssev"

# The VM size for the cluster's system agent nodes
agents_size = "standard_d4lds_v5"

# Initial number of k8s system agent nodes
agents_count = 1

# Tags to use on the resources deployed with this configuration
tags = {}
EOF

To create your cluster, run:

$ terraform -chdir=${WORKSPACE}/iac/azure/cluster init
$ terraform -chdir=${WORKSPACE}/iac/azure/cluster apply

Note that creating an AKS cluster usually takes between 5-15 minutes.

After Terraform runs successfully, configure your environment to use the newly created cluster:

$ source ${WORKSPACE}/iac/azure/cluster/env.sh
$ az aks get-credentials \
  --resource-group "${AZURE_AKS_RG}" \
  --name "${AZURE_CLUSTER_NAME}"

If you already have a cluster, ensure the environment is correctly configured to point to it, and skip the creation of the cluster entirely by running the following commands:

$ export AZURE_CLUSTER_NAME="<cluster name>"
$ export AZURE_AKS_RG="<resource group>"
$ az aks get-credentials \
  --resource-group "${AZURE_AKS_RG}" \
  --name "${AZURE_CLUSTER_NAME}"

Retrieve the name of the virtual network and subnet used by the AKS cluster so that the Confidential Pods can be part of the same network:

$ export AZURE_VNET_NAME=$(az network vnet list \
  --resource-group "${AZURE_AKS_RG}" \
  --query "[0].name" \
  --output tsv)

$ export AZURE_SUBNET_ID=$(az network vnet subnet list \
  --resource-group "${AZURE_AKS_RG}" \
  --vnet-name "${AZURE_VNET_NAME}" \
  --query "[0].id" \
  --output tsv)

Load and push the Anjuna Kubernetes Toolset images

In this section, you will push the Anjuna Kubernetes Toolset images to a container registry, where they can be accessed by your Kubernetes cluster.

The Anjuna Kubernetes Toolset images include the Anjuna Cloud Adaptor and the Anjuna Custom Container Runtime components.

First, specify the name of the Anjuna Kubernetes Toolset images.

$ export REGISTRY="${AZURE_REGISTRY_NAME}.azurecr.io"
$ export ANJUNA_ADAPTOR_IMAGE="${REGISTRY}/anjuna-k8s-cc-toolset:cloud-adaptor-1.3.0005"
$ export ANJUNA_RUNTIME_IMAGE="${REGISTRY}/anjuna-k8s-cc-toolset:runtime-1.3.0005"

Make sure that you are authenticated and have permission to push to the chosen container registry. For example, if you are using Azure Container Registry (ACR), run the following command:

$ az acr login -n "${AZURE_REGISTRY_NAME}"

You should see output like Login Succeeded when the command completes.

The following commands load the images locally from the Anjuna Kubernetes Toolset installer bundle, and then push them to your chosen container registry. This might take a few seconds depending on the size of the images.

$ cd ${WORKSPACE}
$ docker load -i ${WORKSPACE}/anjuna-k8s-sev-adaptor-image.tar
$ docker tag anjuna-k8s-sev-adaptor ${ANJUNA_ADAPTOR_IMAGE}
$ docker push ${ANJUNA_ADAPTOR_IMAGE}
$ docker load -i ${WORKSPACE}/anjuna-k8s-sev-runtime-image.tar
$ docker tag anjuna-k8s-sev-runtime ${ANJUNA_RUNTIME_IMAGE}
$ docker push ${ANJUNA_RUNTIME_IMAGE}

Then, you must ensure that the cluster is able to pull the images from your container registry.

For example, if you are pushing the images to an Azure Container Registry (ACR), you can generate an authentication token as follows:

$ export ACR_TOKEN_NAME="acr-token-${RANDOM}"
$ export ACR_TOKEN=$(az acr token create -n ${ACR_TOKEN_NAME} \
  -r ${AZURE_REGISTRY_NAME} \
  --repository anjuna-k8s-cc-toolset content/read \
  | jq -r '.credentials.passwords[] | select(.name == "password1") | .value')

As part of the Anjuna Kubernetes Toolset installation, you can create a new image pull secret from the authentication token generated above, so that the cluster can pull the toolset images.

Alternatively, if you do not want to use registry tokens as illustrated above, you can attach the ACR to the AKS cluster instead (requires Owner permission).

Install the Anjuna Kubernetes Toolset

The Anjuna Kubernetes Toolset relies on Node labels to select in which Nodes of your cluster it will be installed. This operation can be applied to all Nodes, to a set of Nodes, or even to a single Node.

To query the existing Node labels of your cluster, run kubectl get nodes --show-labels. To add a new label to a specific Node, run kubectl label nodes <node-name> <label-key>=<label-value>.

In this guide, the Anjuna Kubernetes Toolset selects the label kubernetes.azure.com/role=agent, which is applied by default to all AKS Nodes.

Run the following command to configure the Node label to be used. If you want to specify a different label selector to install the Anjuna Kubernetes Toolset to a different set of Nodes, change the values below.

$ export ANJUNA_NODE_LABEL_KEY=kubernetes.azure.com/role
$ export ANJUNA_NODE_LABEL_VALUE=agent

With the label key and value defined, move to the iac folder, and install the needed Custom Resource Definition (CRD) for the Anjuna Kubernetes Toolset to your cluster:

$ cd ${WORKSPACE}/iac
$ kubectl apply -f k8s/crd.yaml
$ envsubst < k8s/ccruntime.template.yaml > k8s/ccruntime.yaml
$ kubectl apply -f k8s/ccruntime.yaml

For regular Pods, the resources.requests and resources.limits fields of the Pod specification help the Kubernetes scheduler make placement decisions based on the capacity and constraints of each worker Node.

Anjuna Confidential Pods are deployed as standalone Confidential VMs and not as containers on the same worker Node. Therefore, an Anjuna Confidential Pod’s spec.resources could mislead the scheduler regarding the actual capacity of the worker Nodes.

To address this issue, the Anjuna Kubernetes Toolset includes a mutating webhook and a controller that adjust the resource requests and limits of an Anjuna Confidential Pod. This allows the Pod to be more accurate regarding Node resource allocation.

cert-manager is required for the mutating webhook, and can be installed as follows to the cluster:

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.9.1/cert-manager.yaml

Since each Anjuna Confidential Pod runs in its own confidential VM, the Anjuna Kubernetes Toolset limits the number of simultaneous Anjuna Confidential Pods to 20 per cluster. If you want to change this number, you can define it in the values.yaml file (see below) by setting maxConfidentialPodVMs to the desired value on line 12.

You will use a Helm Chart to install the Anjuna Kubernetes Toolset to your cluster. To configure it, create a values.yaml file with the following content:

cat > values.yaml <<EOF
cloud: azure
image: ${ANJUNA_ADAPTOR_IMAGE}
nodeSelector:
  key: "${ANJUNA_NODE_LABEL_KEY}"
  value: "${ANJUNA_NODE_LABEL_VALUE}"
imagePullSecret:
  name: anjuna-kubernetes-toolset
  registry: ${AZURE_REGISTRY_NAME}.azurecr.io
  username: ${ACR_TOKEN_NAME}
  password: ${ACR_TOKEN}
maxConfidentialPodVMs: 20
azure:
  resourceGroup: ${AZURE_RESOURCE_GROUP}
  subnetId: ${AZURE_SUBNET_ID}
  subscriptionId: ${AZURE_SUBSCRIPTION_ID}
  instanceSize: "Standard_DC2as_v5"
  location: ${AZURE_LOCATION}
  storageAccount: ${AZURE_STORAGE_ACC_NAME}
  credentials:
    clientId: ${AZURE_CLIENT_ID}
    clientSecret: ${AZURE_CLIENT_SECRET}
    tenantId: ${AZURE_TENANT_ID}
EOF

Deploy the Anjuna Kubernetes Toolset to your cluster:

$ helm install anjuna-cc k8s/chart --values values.yaml --wait

Verify the installation

All resources are created in the anjuna-system namespace.

To ensure that all components are running, you can verify the Pods running in the anjuna-system namespace:

$ kubectl get pods -n anjuna-system

The output should include the following Pods:

NAME                                                    READY   STATUS    RESTARTS   AGE
cc-operator-controller-manager-68ff8494b7-bwcx9         2/2     Running   0          5m10s
cc-operator-daemon-install-w4t82                        1/1     Running   0          2m16s
cc-operator-pre-install-daemon-f4f96                    1/1     Running   0          2m16s
anjuna-cloud-adaptor-daemonset-z4kmg                    1/1     Running   0          4m54s
peer-pods-webhook-controller-manager-5d4675fc4b-6kjnn   2/2     Running   0          65s

Check that the Runtime Class anjuna-remote was added to the cluster:

$ kubectl get runtimeclass

The output should resemble the following. Note that the Runtime Class might take a couple of minutes to be created.

NAME            HANDLER         AGE
anjuna-remote   anjuna-remote   1m

Your Kubernetes cluster is now ready to deploy applications as Anjuna Confidential Pods. Refer to Deploying Pods as Anjuna Confidential Pods in AKS for examples on how to deploy applications.

Upgrade the Anjuna Kubernetes Toolset

To upgrade the installed version of the Anjuna Kubernetes Toolset, Uninstall the Anjuna Kubernetes Toolset, and follow the installation instructions in Load and push the Anjuna Kubernetes Toolset images again to install the new version.

You do not need to recreate the Shared Resources or the cluster in order to upgrade.

All Anjuna Confidential Pods must be stopped before an upgrade. After the upgrade, the Anjuna Confidential Pod images need to be rebuilt with an anjuna-k8s-cli that matches the new Anjuna Kubernetes Toolset version.

Uninstall the Anjuna Kubernetes Toolset

To uninstall the Anjuna Kubernetes Toolset, first stop all Anjuna Confidential Pods.

Then, run the following commands from the ${WORKSPACE}/iac folder:

$ cd "${WORKSPACE}/iac"
$ kubectl delete -f k8s/crd.yaml
$ helm uninstall anjuna-cc --wait
$ kubectl delete runtimeclass anjuna-remote --ignore-not-found

This operation might take about a minute to fully complete.

To delete cert-manager, run the following command:

$ kubectl delete -f https://github.com/jetstack/cert-manager/releases/download/v1.9.1/cert-manager.yaml

Refer to the next section, Cleaning up resources, to see how to destroy the resources created by Terraform in your Azure subscription.