Volumes

In this section, you will learn how to leverage Kubernetes volumes in order to mount files and directories inside a Confidential Pod.

Using persistent storage in the context of Confidential Pods might present a security risk to the application, as the Anjuna Nitro Runtime creates shared storage between the AWS Nitro Enclave and the untrusted parent instance. Refer to the Persistent storage security section before proceeding.

As mentioned on the Providing an EIF for the Pod’s Enclave page, there are two ways to create the Enclave Image File (EIF) in order to run it in the Confidential Pod: build on the fly, or build in advance. While it is recommended to build the EIF in advance, for ease of explanation, some of the examples on this page use the option of building the EIF on the fly.

Mounting a ConfigMap

A common use case for Kubernetes volumes is mounting configuration files inside of Pods. In the example below, the Anjuna Kubernetes Toolset for AWS EKS will automatically create an Enclave Image File from the Pod spec. This includes correctly mapping the specified volumeMounts to the enclave so that the application running inside the Confidential Pod can access the ConfigMap contents.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-map-name
data:
  data.txt: "Welcome to Anjuna!"
---
apiVersion: v1
kind: Pod
metadata:
  name: nitro-busybox-pod
  labels:
    nitro.k8s.anjuna.io/managed: "yes"
spec:
  containers:
    - name: nitro-busybox-pod
      command:
        - "sh"
        - "-ec"
        - |
          cat /tmp/data/data.txt && echo
          sleep infinity
      image: busybox
      resources:
        limits:
          memory: "2048Mi"
          cpu: "2"
      volumeMounts:
        - name: config-volume
          mountPath: /tmp/data
  volumes:
    - name: config-volume
      configMap:
        name: config-map-name

After its startup, the enclave will display the contents of /tmp/data/data.txt and sleep indefinitely. You can create this Pod by saving the Pod spec above to example-configmap.yaml and then running:

$ kubectl create -f example-configmap.yaml

Check the Pod logs to confirm that the enclave can access the mounted file:

$ kubectl logs nitro-busybox-pod -f

The output will include many logs, including the steps the Anjuna Kubernetes Toolset for AWS EKS takes to create an Enclave Image File from your Pod spec. Then, the application (as specified in the command field) will run. You should see the following toward the end of the output:

ANJ-ENCLAVE: Service anjunafs started with original pid=465
ANJ-ENCLAVE: Mounting volume with mountPath '/tmp/data'
ANJ-ENCLAVE: Mounting volume with mountPath '/var/run/secrets/kubernetes.io/serviceaccount'
ANJ-ENCLAVE: Launched "/bin/sh" with pid=470
Welcome to Anjuna!

Using `PersistentVolume` and `PersistentVolumeClaim`

Another common use case for volumes is providing persistent storage to Pods, which have ephemeral filesystems. Applications running inside enclaves can benefit from the persistent volumes in Kubernetes. In the example below, you will mount /tmp/data, backed by a 1Gi Persistent Volume that is dynamically provisioned by the Storage Class gp2 (the default for EKS). Note that manually provisioned volumes should work in the same way.

The application will write a file to the persisted directory (see the command field of the Pod spec).

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-pvc
spec:
  storageClassName: gp2
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: nitro-busybox-pod
  labels:
    nitro.k8s.anjuna.io/managed: "yes"
spec:
  containers:
    - name: nitro-busybox-pod
      command:
        - "sh"
        - "-ec"
        - |
          echo "I am a persisted file" | tee /tmp/data/file.txt
          sleep infinity
      image: busybox
      resources:
        limits:
          memory: "2048Mi"
          cpu: "2"
      volumeMounts:
        - name: data
          mountPath: /tmp/data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-pvc

To create this example, save the manifests above to example-pvc.yaml and run:

kubectl create -f example-pvc.yaml

File ownership on AnjunaFS mounts

Files on AnjunaFS mounts keep the uid and gid of the files on the host, which anjuna-fs-proxy creates as the Anjuna Nitro Launcher Pod’s user. Files created by any enclave user are therefore owned by the Launcher uid (10003 by default) on the host, and the enclave sees that same ownership when it reads the files back.

For an enclave workload that runs as root, this is transparent. A non-root enclave workload with a uid that differs from the Launcher’s cannot write to AnjunaFS mounts by default.

To give a non-root enclave workload read/write access to its volumes, align the Launcher uid with the workload uid by setting .spec.securityContext in the Pod specification:

apiVersion: v1
kind: Pod
metadata:
  name: nitro-nonroot-pod
  labels:
    nitro.k8s.anjuna.io/managed: "yes"
spec:
  securityContext:
    # The workload inside the enclave runs with user ID 5000.
    # NOTE: For Confidential Pods, the uid must be set either in the container
    # image (e.g. the `USER` directive) or as part of the Anjuna Nitro Enclave
    # configuration file (the `username` field).
    runAsUser: 5000
    fsGroup: 5000
<snip>...

The fsGroup field is required for volumes provisioned by CSI drivers (such as PersistentVolumeClaim volumes): it makes the kubelet re-own the volume root as group-writable when the volume is attached. Without it, the volume root stays owned by root, and writes are still denied.

The Webhook applies the Anjuna Kubernetes Toolset for AWS EKS security context on top of these fields rather than replacing them, so both reach the mutated Launcher Pod. The same result can be achieved with the nitro.k8s.anjuna.io/launcherPodSecurityContext annotation, which is the only option on Anjuna Kubernetes Toolset for AWS EKS versions before 1.59:

    nitro.k8s.anjuna.io/launcherPodSecurityContext: '{"runAsUser": 5000, "fsGroup": 5000}'

Refer to Configuring a security context for a particular Anjuna Nitro Launcher Pod for the merge rules that apply to both forms.

Volume naming rules

Two naming rules apply to the volumes of a Confidential Pod: the anjuna-system- prefix, described in Providing an EIF for the Pod’s Enclave and Using a local Encrypted Configuration, and the following names reserved by the Anjuna Kubernetes Toolset for AWS EKS.

The Anjuna Kubernetes Toolset for AWS EKS adds its own volumes to the Launcher Pod for the run-time directories the Launcher writes to, which the Launcher container needs because it runs with a read-only root filesystem. These volumes are emptyDir volumes created per Pod, and are never forwarded to the enclave.

The following volume names are reserved, and your Pod spec must not declare them:

anjuna-launcher-ipc: Enclave sockets and the port files of the Launcher’s proxy services.
anjuna-launcher-scratch: The generated enclave configuration, the EIF built or downloaded at launch, and the Launcher’s HOME and XDG_RUNTIME_DIR.

A Pod that declares either name is rejected by the Kubernetes API server for a duplicate volume name.

Using volumes with pre-built EIFs

In the examples above, the Anjuna Kubernetes Toolset for AWS EKS automatically created an EIF from your Pod spec. This will generate enclave configuration files dynamically, which will map the volume mounts inside the enclave.

To use Kubernetes Volumes with pre-built EIFs, you must explicitly configure basic volume mounts in your enclave configuration file. For more information about the basic volume mounts, read about Persistent storage and Basic mounts.

Volume names starting with the anjuna-system- prefix are reserved for internal use and will not be mounted inside the enclave. Avoid using this prefix for custom volumes to prevent unexpected behavior.

As an example, you can create or extend your existing enclave configuration file with a new basic mount for the data volume, as shown below. Note that the name of the mount must match the volume name in the Pod spec, and the mountPath of the mount must match the mountPath in the Pod spec:

version: 1.8
# ... other fields of your enclave config file ...
mounts:
  - type: basic
    name: data # must match the volume name in the Pod spec
    mountPath: /tmp/data # must match the mountPath in the Pod spec

After building and pushing your new EIF, you can transform the previous Pod spec with a PersistentVolumeClaim to use a pre-built EIF saved to s3://<bucket>/busybox.eif by adding the nitro.k8s.anjuna.io/imageLocation annotation:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-pvc
spec:
storageClassName: gp2
accessModes:
  - ReadWriteOnce
resources:
  requests:
    storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: nitro-busybox-pod
labels:
  nitro.k8s.anjuna.io/managed: "yes"
annotations:
  nitro.k8s.anjuna.io/imageLocation: "s3://<bucket>/busybox.eif"
spec:
  containers:
    - name: nitro-busybox-pod
      image: busybox
      resources:
        limits:
          memory: "2048Mi"
          cpu: "2"
      volumeMounts:
        - name: data
          mountPath: /tmp/data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-pvc