Kubernetes Pod specification

This section describes the changes you need to make to a Kubernetes Pod specification in order to instruct the Anjuna Kubernetes Toolset for AWS EKS to run the specified application in an AWS Nitro Enclave.

On this page, some code blocks are shortened to emphasize only the relevant configuration. A line with <snip>…​ indicates that some lines have been removed from the full configuration.

Enabling running a Pod in an AWS Nitro Enclave

To indicate that a Pod should be running in an AWS Nitro Enclave, set the label nitro.k8s.anjuna.io/managed to yes in the Pod specification:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    name: nginx-pod
    nitro.k8s.anjuna.io/managed: "yes"
<snip>...

Setting this label to yes instructs the Anjuna Nitro Webhook to intercept the creation of that Pod, and automatically convert it into an AWS Nitro Enclave.

Other Kubernetes workload objects like Deployments

Kubernetes workload objects like Deployments are translated into Pods using the pod template field. So, add nitro.k8s.anjuna.io/managed: "yes" to the template of the Deployment. For example, if you have a Deployment with the following specification, line 16 defines the Anjuna label, and this label will be passed onto the created Pods:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        nitro.k8s.anjuna.io/managed: "yes"
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

Controlling the resources allocated to the AWS Nitro Enclave

Use the standard Kubernetes Pod specification attributes to control the vCPUs and memory reserved for the enclave:

  • spec.containers[].resources.limits.cpu

  • spec.containers[].resources.limits.memory

spec.containers[].resources.limits.cpu MUST be an integer when used in the context of an AWS Nitro Enclave (a regular Kubernetes Pod supports fractional vCPU values). The number of vCPU cores must be an even number due to hyperthreading.

This is an example Pod configuration that reserves 2GB of memory and 2 vCPUs for an AWS Nitro Enclave (see lines 14-16):

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    name: nginx-pod
    nitro.k8s.anjuna.io/managed: "yes"
spec:
  containers:
  - name: nginx-pod
    image: nginx:latest
    imagePullPolicy: Always
    resources:
      limits:
        memory: "2048Mi"
        cpu: "2"
    ports:
      - containerPort: 80
<snip>...

Configuring a security context for a particular Anjuna Nitro Launcher Pod

The security context field defines privileges and access control settings for a Pod.

You can configure a default security context for all Anjuna Nitro Launcher Pods in the Toolset configuration via the Anjuna Toolset Helm chart, which is the recommended way of configuring the Launcher security context. Refer to the Configuring and deploying the Anjuna Kubernetes Toolset section for more information.

Setting a security context in the Pod specification

The Pod specification keeps the security context fields it sets itself. The Webhook applies the Toolset configuration on top of .spec.securityContext and .spec.containers[].securityContext rather than replacing them. Fields that are required for the Launcher are enforced, while the fields the Launcher does not depend on keep the value the Pod specification gives them:

apiVersion: v1
kind: Pod
metadata:
  name: nitro-nonroot-pod
  labels:
    nitro.k8s.anjuna.io/managed: "yes"
spec:
  securityContext:
    # The workload inside the enclave runs with user ID 5000.
    # NOTE: For Confidential Pods, the uid must be set either in the container
    # image (e.g. the `USER` directive) or as part of the Anjuna Nitro Enclave
    # configuration file (the `username` field).
    runAsUser: 5000
    fsGroup: 5000
    supplementalGroups: [5000]
<snip>...

In the example above, the mutated Launcher Pod keeps runAsUser, fsGroup, and the requested supplemental group. The AWS Nitro Enclave device gid is added to supplementalGroups alongside the group the Pod asked for. Settings the Launcher requires still override the Pod specification, and the optional hardening defaults still fill in the fields the Pod leaves unset. Refer to the Configuring and deploying the Anjuna Kubernetes Toolset section for the order in which the layers are applied.

Anjuna Kubernetes Toolset for AWS EKS versions before 1.59 removed both security contexts from the Pod specification before injecting the Launcher security context. Pods that set a security context and relied on it being dropped now carry those fields through to the Launcher.

Overriding the security context with annotations

If you need to override a setting the Anjuna Kubernetes Toolset for AWS EKS enforces, you can set the security context of a particular Anjuna Nitro Launcher Pod. Two annotations are available, each corresponding to one of the Helm chart values:

nitro.k8s.anjuna.io/launcherPodSecurityContext

Sets .spec.securityContext on the Launcher Pod. The value must be a JSON string that defines a valid v1.PodSecurityContext Kubernetes field.

nitro.k8s.anjuna.io/launcherContainerSecurityContext

Sets .spec.containers[].securityContext on the Launcher container. The value must be a JSON string that defines a valid v1.SecurityContext Kubernetes field.

The Pod creation will fail if a provided value is not a valid JSON string or it does not specify a valid object of the corresponding type.

Each annotation is the last layer applied to the matching security context, merged onto the security context fields (that were resolved from the Pod specification and from the Toolset configuration) as a JSON Merge Patch (RFC 7386):

  • Fields set in the annotation override the resolved value, including the settings the Anjuna Kubernetes Toolset for AWS EKS enforces.

  • Fields omitted from the annotation keep their resolved value.

  • A field set to null is removed from the result.

  • List fields (such as supplementalGroups or capabilities.add) are replaced wholesale.

Because list fields are replaced rather than merged, an annotation that sets supplementalGroups must include the nitroDeviceGID gid of the group that owns the AWS Nitro Enclave device files (75 by default), along with any supplemental group the Pod specification asks for. For example, to add a supplemental group 9999 while keeping the default device gid, use '{"supplementalGroups": [75, 9999]}'.

Setting the Launcher Pod security context

Example:

apiVersion: v1
kind: Pod
metadata:
  name: custom-device-gid-example
  labels:
    name: custom-device-gid-example
    nitro.k8s.anjuna.io/managed: "yes"
  annotations:
    nitro.k8s.anjuna.io/launcherPodSecurityContext: '{"supplementalGroups": [1001]}'
    <snip>...

The example above replaces the gid of the group that owns the AWS Nitro Enclave device files, for example for heterogeneous Node groups. Because supplementalGroups is replaced wholesale, the annotation also drops the nitroDeviceGID gid and any group the Pod specification asked for. Restate every gid the Launcher needs in the annotation.

To align the Launcher uid with a non-root enclave workload’s uid, set .spec.securityContext in the Pod specification rather than using this annotation. Refer to the File ownership on AnjunaFS mounts section for more information about non-root enclave workloads and volumes.

Configure the Seccomp profile of the Launcher through the container security context (.spec.containers[].securityContext), rather than the Pod security context (.spec.securityContext). Refer to Configuring Launcher Seccomp for examples on how to install and use custom Seccomp profiles, or disable Seccomp altogether.

Setting the Launcher container security context

The nitro.k8s.anjuna.io/launcherContainerSecurityContext annotation adjusts the security context of the Launcher container itself, which is where the Anjuna Kubernetes Toolset for AWS EKS applies its hardening. That hardening comes in two tiers: allowPrivilegeEscalation, the capability set, and the seccompProfile are enforced, while runAsNonRoot and readOnlyRootFilesystem are defaults that the Pod specification can override on its own. Since the annotation is applied after both tiers, it can override the enforced settings as well. Refer to When the Launcher needs privilege escalation for the cases in which you can disable allowPrivilegeEscalation.

Since capabilities.add is replaced wholesale rather than merged (see above), an annotation that sets it drops any capabilities that it does not restate. If the capabilities relevant to anjuna-fs-proxy (CHOWN, MKNOD, FOWNER, DAC_OVERRIDE) are dropped, the Launcher still starts and AnjunaFS mounts are still served, but file operations that need a dropped capability will fail. Refer to Launcher privilege escalation details for which operations need which capability.

A seccompProfile set through this annotation must allow vsock operations, as described in Configuring Seccomp profiles for the Anjuna Nitro Launcher.

 
# Grant one additional capability, restating the four AnjunaFS needs:
nitro.k8s.anjuna.io/launcherContainerSecurityContext: '{"capabilities": {"add": ["CHOWN", "MKNOD", "FOWNER", "DAC_OVERRIDE", "SYS_TIME"]}}'

# Replace the Unconfined Seccomp profile with a custom one that allows vsock operations:
nitro.k8s.anjuna.io/launcherContainerSecurityContext: '{"seccompProfile": {"type": "Localhost", "localhostProfile": "profiles/anjuna.json"}}'

Passing parameters to the Anjuna Nitro Runtime

When a Pod is launched in an AWS Nitro Enclave, the Anjuna Nitro Runtime is used to build, configure, and run the AWS Nitro Enclave.

The following environment variable is used to control how the Anjuna Nitro Runtime behaves:

  • ANJ_ENCLAVE_DEBUG_MODE: Set this variable to “yes” to create a debug enclave. If not defined or set, the Anjuna Kubernetes Toolset for AWS EKS will start the AWS Nitro Enclave in production mode.

Downloading an EIF instead of building it on the fly

The Anjuna Kubernetes Toolset for AWS EKS tools can create the Enclave Image File automatically from the Pod specification. However, you also have the option to pre-create an EIF and instruct the Anjuna Kubernetes Toolset for AWS EKS to download the EIF from an S3 bucket by using the nitro.k8s.anjuna.io/imageLocation annotation:

apiVersion: v1
kind: Pod
metadata:
  name: secure-eif-pod
  labels:
    name: secure-eif-pod
    nitro.k8s.anjuna.io/managed: "yes"
  annotations:
    nitro.k8s.anjuna.io/imageLocation: "s3://your-eif-bucket/your-eif-file"
    <snip>...

When using this option, the container image in the Pod specification is ignored, although the Kubernetes Pod specification requires a value. You can simply leave the container image used in the original Pod specification.

Providing an Encrypted Configuration to the Pod

The Anjuna Kubernetes Toolset for AWS EKS tools supports two methods to provide an Encrypted Configuration to a Confidential Container that runs in EKS.

One method supports using S3 Encrypted Configurations, which hard codes the location of the Encrypted Config in the EIF. This option does not require you to make any further changes to the Pod spec.

A second method allows you to specify a Local Encrypted Configuration, which should be passed to the Pod’s enclave on enclave start, allowing you to provide different secrets to different launches of the enclave. This method requires you to change the Pod spec to inform the launcher Pod of where the Encrypted Configuration is located, using the nitro.k8s.anjuna.io/encryptedConfigLocation annotation:

apiVersion: v1
kind: Pod
metadata:
  name: secret-enabled-pod
  labels:
    name: secret-enabled-pod
    nitro.k8s.anjuna.io/managed: "yes"
  annotations:
    nitro.k8s.anjuna.io/encryptedConfigLocation: "/path/on/the/pod/filesystem"
    <snip>...

This method requires using a prebuilt EIF, as described in the Downloading an EIF section. It assumes that the file containing the Encrypted Configuration exists in the Pod’s filesystem. For examples on how to expose it to the Pod, see Leveraging Remote Attestation to protect the Pod sensitive configuration.

Controlling the resources assigned to the Anjuna Nitro Pod

You have the ability to control the resources assigned to the Anjuna Nitro Pod by specifying the following annotations:

  • nitro.k8s.anjuna.io/launcherCPU: The limit for the number of vCPUs for the Pod.

  • nitro.k8s.anjuna.io/launcherMemory: The limit for the amount of memory for the Pod.

The values for these annotations are in the same format than the limits properties (memory/cpu) for a Pod specification:

<snip>
annotations:
  <snip>...
  nitro.k8s.anjuna.io/launcherMemory: "4Gi"
  nitro.k8s.anjuna.io/launcherCPU: "750m"
  <snip>...

If you do not specify these annotations, no resource limits are set on the launcher Pod. This is particularly useful when creating Enclave Image Files (EIF) automatically, which is a memory-intensive process. Once the EIF has been created, the resource requirements for the launcher Pod are minimal.

Licensing the Anjuna Nitro Runtime for an Anjuna Nitro Pod

A license is required to run an Anjuna Nitro Pod. See the Licensing page for instructions on how to download the license from the Anjuna Resource Center.

This license file must be mounted to each Pod as a Kubernetes secret.

The Kubernetes secret must be named anjuna-license. The key license.yaml must contain the contents of the license file downloaded earlier. Run the following command to create the secret:

$ kubectl create secret generic anjuna-license --from-file=license.yaml

The Anjuna Nitro Webhook will automatically mount the license file to your Pod at /opt/anjuna/license.yaml.

Example

The following Pod specification creates an AWS Nitro Enclave for the Nginx web server.

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
  labels:
    name: nginx-pod
    nitro.k8s.anjuna.io/managed: "yes"
spec:
  containers:
    - name: nginx-pod
      image: nginx:latest
      imagePullPolicy: Always
      env:
        - name: ANJ_ENCLAVE_DEBUG_MODE
          value: "yes"
      resources:
        limits:
          memory: "2048Mi"
          cpu: "2"
      ports:
        - containerPort: 80