Notes on Pod upgrades

In this section, both examples used a Kubernetes Deployment resource. This kind of resource allows you to preemptively manage the lifecycle of your Pod with regards to:

  1. What should happen when the Pod fails?

  2. What should happen when the Pod exits successfully?

  3. How many Pod replicas should there be?

  4. What liveness conditions should be guaranteed when upgrading to a new version?

Anjuna recommends always wrapping your Anjuna Confidential Pod manifest within higher level manifest kinds, also called workload resources, such as Deployments and StatefulSets.

The Anjuna Kubernetes Toolset makes sure that signals are propagated correctly to the Kubernetes control plane, so that workload resources behave as expected.

Simple Pod manifest

A user might encounter situations where a simple Pod manifest is desired, and in this case upgrades should be handled more carefully. The Anjuna Kubernetes Toolset enforces immutability of the Anjuna Confidential Pod, i.e., the application cannot change after it has been launched.

For that reason, reapplying the simple Pod manifest with a modified image field will cause the Pod to fail with the following error:

$ kubectl get pod nginx
NAME    READY   STATUS              RESTARTS     AGE
nginx   0/1     RunContainerError   3 (2s ago)   3m1s

$ kubectl describe pod nginx | grep Error:
Warning  Failed     7m10s (x3 over 4m52s)   kubelet            Error:
failed to create containerd task: failed to create shim task: cannot create more than one workload container in a CVM: unknown

You can also check the Pod’s CVM logs to find the following error message:

$ cd ${WORKSPACE}/anjuna-k8s-cli
$ ./get-pod-cvm-logs.sh nginx
$ tail nginx-cvm.logs
...
{"msg":"attempted to start more than one workload container","level":"ERRO",...}

Changes to the annotation io.anjuna.sev.image are not considered trigger events. This means Kubernetes will not try to upgrade the Pod if only the annotation is changed.

For that reason, to upgrade a simple Pod manifest, Anjuna recommends the following procedure:

  1. Modify both io.anjuna.sev.image annotation and image field with the new image data

  2. Terminate the old Pod with kubectl delete -f <manifest>

  3. Apply the new manifest with kubectl apply -f <manifest>

Workload Resource manifest upgrades

Workload Resource manifests control the lifecycle of the Pod as a whole. For example, during upgrades, a Deployment will launch a new Pod by default, and only when the new Pod is ready will the old Pod be terminated.

This conforms with Anjuna’s Confidential Pod immutability feature: the Pods are not changing, but upgrades can be achieved nonetheless.

It is worth noting that deployments do consider changes to the io.anjuna.sev.image annotation as a trigger event for upgrades if they are located as part of the Pod spec (i.e., field .spec.template.spec.metadata.annotations).

As an example, assume the Nginx Pod was defined within a Deployment manifest, and needs to have its enclave configuration updated with a new secret. In this case, in order to upgrade it to the new version, the following simpler procedure is sufficient:

  1. Change the io.anjuna.sev.image annotation to the new image version

  2. Reapply the Deployment manifest with kubectl apply -f <manifest>

The change to the annotation alone will trigger an upgrade. You should see something like the following in this case:

$ kubectl apply -f nginx.yaml
deployment.apps/nginx configured
$ kubectl get pods
NAME                  READY STATUS            RESTARTS AGE
nginx-9db88ffc8-rzj92 0/1   ContainerCreating 0        10s
nginx-9db88ffc8-dfzvs 1/1   Running           0        3m1s

# after a while
$ kubectl get pods
NAME                  READY STATUS      RESTARTS AGE
nginx-9db88ffc8-rzj92 1/1   Running     0        10s
nginx-9db88ffc8-dfzvs 0/1   Terminating 0        4m1s