The Confidential Pod Configuration File Reference

The Anjuna Confidential Pod Configuration file allows users to:

Enforce aspects of the Pod’s containers, to provide security to the execution of these containers, even in situations where the Kubernetes infrastructure or privileged Kubernetes users are compromised.
Change certain behaviors of the containers in a Pod, without needing to modify the original container.
Inject sensitive data into the containers, using remote attestation, to allow for secure release of protected data only to trusted software running in a Confidential Pod.

This configuration is mandatory for every disk build with the Anjuna Kubernetes CLI. It is written in YAML, and in structure it is similar to the Kubernetes Pod definition.

When building Anjuna Confidential Pod disk images, you must set the Anjuna Confidential Pod Configuration file using the command anjuna-k8s-cli build with a --cpod-config <config_file> parameter. For example, if your Anjuna Confidential Pod Configuration file is called config.yaml, use the following command:

AKS
OpenShift on GCP

$ anjuna-k8s-cli build azure  \
	--cpod-config=config.yaml

$ anjuna-k8s-cli build gcp  \
	--cpod-config=config.yaml

The two main sections of the Confidential Pod configuration file

A Confidential Pod configuration file is built of two main sections - pod and apm.

The pod section is a mandatory section, which defines the container used in the Pod. Anjuna is working on adding support to run multiple containers in the Confidential Pod (init and sidecar containers). If you are interested in that option, please contact support@anjuna.com for more details.

The apm section defines the connection details to the Anjuna Policy Manager (APM). It should be used if the Confidential Pod needs to inject secrets using the APM and remote attestation.

pod:
  containers:
...
apm:
...

The rest of this page details the content of these two sections.

The `pod` section

The pod section in the Confidential Pod configuration file is specified under the pod entry. The main sections of the pod entry is the containers subentry, which are lists of the containers intended to run inside of the pod. For example:

pod:
  containers:
  - ...

At the moment only one container is supported in this subentry. If you are interested in the option to use more than one container, please contact support@anjuna.io.

The container entry needs to specify the name of the container, and the URI of the container’s image. It may also specify the command to be used by the container, and the environment used by the container (variables, files, working directory, security context and so on).

An example of a container configuration:

  - name: hello
    image: hello-world
    command: [ 'sh', '-c', 'echo "Hello"']
    env:
    - name: MEASURED_ENV
      value: "Measured!"
    - name: SECRET_ENV
      valueFrom:
          apm:
            path: /some/secret/path
            enginePath: kv-engine
    files:
    - path: /some/path
      mode: 0644
      owner: root
      group: root
      value: |
        some file content
    - path: /some/other/path
      valueFrom:
      apm:
        path: /apm/path/to/file
        enginePath: kv-engine
      workingDir: /workdir
      securityContext:
        runAsUser: 1002
    untrusted:
      env:
        allow:
          - MEASURED_ENV
          - OTHER_ENV
      paths:
        allow:
          - /some/path

The Container Name - Mandatory

The name entry is a mandatory part of a container configuration. When creating a container inside of the Confidential Pod configuration file, the name of the container is used as the primary identifier. The container configuration specified in the Pod configuration is applied based on that name.

As such, the name of the container in the Anjuna Confidential Pod Configuration file needs to match the name provided in the Kubernetes Pod spec. For example:

pod:
  containers:
  - name: my-container
    ...

The Container Image - Mandatory

The image entry of a container is used at disk build time to decide what image to embed in the CVM disk image which is created for the Confidential Pod. It is then used at runtime, to decide which image to launch for each container. As such, this image should be provided as a full URI. For example:

pod:
  containers:
  - image: docker.io/library/nginx:1.27.3
    ...

The Container Command - Optional

The command entry of the container is used to override the command the container will use. The command provided by Kubernetes is ignored in favor of what was defined in the configuration. This command should be provided as a list. If this entry is empty, the command defined by the container image will be used. In any case, the command provided by Kubernetes is always ignored. For example:

---
pod:
  containers:
  - command: ["echo", "hello", "world"]
    ...
----

The Container Environment Variables - Optional

The env entry of the container is used to add environment variables (on top of the container image’s) that the container will run with. This is a list of environment entries. There are two ways of specifying values for these environment variables: either literally, or using the APM.

Each environment variable must define either a literal value or an APM sourced value, but never both.

Literal Environment Variables

When setting environment variables in this section, the Anjuna Runtime will set these environment variables with the values specified in the Container when it is started inside the Confidential Pod. This option allows setting the value for environment variables, without the need to change the original container. When providing the values literally, use the value subentry. For example:

pod:
  containers:
  - env:
    - name: my-public-env
      value: some-public-value
    ...

APM Sourced Environment Variables

When setting environment variables in this section, the Anjuna Runtime will use remote attestation to inject secrets into the container. These secrets are fetched from the APM based on the Confidential Pod identity (as presented in the attestation quote).

When providing the values through the APM, use the valueFrom entry with a nested apm entry. In the apm entry, specify the path entry (pointing to the path in APM where the value is stored), and the enginePath entry (pointing to the path of the secret engine where this value is stored). For example:

pod:
  containers:
  - env:
    - name: my-secret-value
      valueFrom:
        apm:
          path: path/to/secret
          enginePath: some-kv-engine
    ...

The Container Files - Optional

The files entry of the container is used to define files that will be created for the container. This is a list of file entries. In terms of metadata, each file entry can define path, mode, owner and group. There are two ways of specifying the content for these files, either literally, or using the APM.

Each file must define either a literal content or an APM sourced content, but never both.

Literal Files

When setting files in this section, the Anjuna Runtime will set these files with the content specified in the Container when it is started inside the Confidential Pod. This option allows setting files with the needed content, without the need to change the original container. When providing the values literally, use the value subentry. For example:

pod:
  containers:
   - files:
      - path: /etc/hosts
        value: |
          127.0.0.1 my.local.server
    ...

Values can be defined similarly to environment variables with literal value or the use of valueFrom to fetch the value from the APM. For example:

APM Sourced Files

When setting file contents from the APM, the Anjuna Runtime will use remote attestation to inject secrets into the container filesystem. These secrets are fetched from the APM based on the Confidential Pod identity (as presented in the attestation quote).

pod:
  containers:
  - files:
    - name: /home/someuser/.ssh/id_ecdsa
      owner: someuser
      group: someuser
      mode: 0600
      valueFrom:
        apm:
          path: path/to/ssh/key
          enginePath: some-kv-engine
    ...

The Container Working Directory - Optional

The workingDir entry of the container is used to define the working directory that the container main process will use. For example, if the application is expecting to run in the /work/dir directory, this entry allows specifying that. For example:

pod:
  containers:
  - workingDir: /work/dir
    ...

The Container Security Context - Optional

The securityContext entry of the container is used to define permissions associated with the container main process. For now, it only supports the runAsUser subentry, which can be used to define the user ID that the main process will run under. For example, you can specify the application to run under a specific user ID, like 1000:

pod:
  containers:
  - securityContext:
      runAsUser: 1000
    ...

The Container Untrusted Configuration

The Anjuna Confidential Pod Configuration provides the user a trusted and measured configuration that is attached to the disk at build time. However, there are cases where applications require data that cannot be known in advance, for example:

The IP address of a logging server
Values fetched from the Kubernetes Downward API
Values provided to the Anjuna Confidential Pod via Secrets or ConfigMaps.

The Anjuna Kubernetes Toolset provides a way to set this type of untrusted configuration via environment variables or volume mounts that can be allowed in the untrusted entry of the container.

Make sure to understand the consequences of allowing untrusted environment variables or volume mounts to be set from the Pod specification. Keep in mind that these environment variables or volume mounts can be manipulated by potential attackers to alter the behavior of your Anjuna Confidential Pod in undesirable ways.

Untrusted Environment variables

By default, all the environment variables in a Pod specification are ignored since the cluster itself is considered to be untrusted. For instance, a malicious cluster administrator could set arbitrary environment variables by editing your Pod specification on the fly, thus affecting the behavior of the Pod.

Only environment variables that are previously measured as part of the container image or as part of the Anjuna Confidential Pod Configuration file are considered trusted and are effectively applied to the Anjuna Confidential Pod in Kubernetes. Environment variables that are injected from Anjuna Policy Manager secrets are also considered trusted, as they are only injected to trustworthy Anjuna Confidential Pods.

To allow an untrusted environment variable to be applied from the Anjuna Confidential Pod specification, you must explicitly allow it in the Anjuna Confidential Pod Configuration file via the field untrusted.env.

For example:

pod:
  containers:
  - name: my-container
    untrusted:
      env:
        allow:
        - LOG_SERVER_ADDRESS
        - LOG_LEVEL
        - SERVICE_ADDRESS

Once you build your Anjuna Confidential Pod image with this configuration, the environment variables LOG_SERVER_ADDRESS, LOG_LEVEL, and SERVICE_ADDRESS will be applied if set in the Pod specification.

For instance, the Pod specification below would set the value of LOG_LEVEL directly in the Pod spec, the value of LOG_SERVER_ADDRESS from a ConfigMap, and fetch the value of SERVICE_ADDRESS from the Kubernetes Downward API.

apiVersion: v1
kind: Pod
# <snip>...
metadata:
  labels:
	anjuna.io/run-confidential: "yes"
spec:
  containers:
	# <snip>...
	env:
  	- name: LOG_LEVEL
    	value: "debug"
  	- name: LOG_SERVER_ADDRESS
    	valueFrom:
      	configMapKeyRef:
        	name: logging
        	key: server_address
  	- name: SERVICE_ADDRESS
    	valueFrom:
      	fieldRef:
        	fieldPath: status.podIP
  	# <snip>...

apiVersion: v1
kind: ConfigMap
metadata:
  name: logging
data:
  server_address: "my-log-server.anjuna.io"

If the Anjuna Confidential Pod Configuration file specified a trusted value for any of those variables through the container’s envs entry, they would have taken precedence over the untrusted value.

If you define the same environment variable in more than one place, the following describes the precedence used:

The env section in the Anjuna Confidential Pod Config file that is provided when building the enclave, is the highest in precedence.
The env’s in the `untrusted section is second in precedence, and is only applied if an environment variable is not already applied through the container' env.
Finally, the environment variables set by the build-time container image are lowest in precedence.

Untrusted Volume Mounts

Similar to environment variables, all volume mounts defined in the Pod specification are considered untrusted and therefore ignored by default, unless explicitly allowed through the untrusted.paths field.

For example:

pod:
  containers:
  - name: my-container
    untrusted:
      paths:
        allow:
        - /var/run/secrets/kubernetes.io/serviceaccount
        - /usr/share/nginx/html/index.html

The Anjuna Confidential Pod Configuration file above allows volume mounts for /var/run/secrets/kubernetes.io/serviceaccount (the Kubernetes service account token) and /usr/share/nginx/html/index.html.

For instance, the Pod specification below mounts a file at /usr/share/nginx/html/index.html from the ConfigMap custom-page. Kubernetes automatically mounts the service account token to all Pods by default.

apiVersion: v1
kind: Pod
# <snip>...
metadata:
  labels:
	anjuna.io/run-confidential: "yes"
spec:
  containers:
	# <snip>...
	volumeMounts:
  	- name: custom-page
    	mountPath: /usr/share/nginx/html/index.html
    	subPath: index.html
  	# <snip>...
  volumes:
	- name: custom-page
  	configMap:
    	name: custom-page

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-page
data:
  index.html: "This is my custom page!"

If the enclave configuration file specified a trusted file for any of the mount paths through the field files, they would have taken precedence over the untrusted volume mount from the Pod specification.

The `apm` section

The apm section is used to configure the communication with the Anjuna Policy Manager (APM) server. It has two configuration parameters: url, caCert.

`url`

This field points to the URL where the APM resides. This is a mandatory field, if using the apmConfig entry.

The field must start with https:// and be combined with the caCert field described below.

For example, if the APM sits on port 8200 of a server with the DNS name apm.anjuna.com, use the entry:

...
apm:
  url: https://apm.anjuna.com:8200
  caCert: |
	-----BEGIN CERTIFICATE-----
	... APM's certificate content ...
	-----END CERTIFICATE-----

`caCert`

This field points to the APM’s certificate. When this field is specified, the Anjuna Confidential Container instance will verify the APM’s identity using this certificate.

For example, if the APM sits on port 8200 of a server with the DNS name apm.anjuna.com, and it is using TLS connections, use the entry:

...
apm:
  url: https://apm.anjuna.com:8200
  caCert: |
	-----BEGIN CERTIFICATE-----
	... APM's certificate content ...
	-----END CERTIFICATE-----

Complete Example

Provided here is an example that utilizes every aspect of the configuration as described in this page.

pod:
  containers:
  - name: hello
    image: hello-world
    command: [ 'sh', '-c', 'echo "Hello"']
    env:
    - name: MEASURED_ENV
      value: "Measured!"
    - name: SECRET_ENV
      valueFrom:
          apm:
            path: /some/secret/path
            enginePath: kv-engine
    files:
    - path: /some/path
      mode: 0644
      owner: root
      group: root
      value: |
        some file content
    - path: /some/other/path
      valueFrom:
      apm:
        path: /apm/path/to/file
        enginePath: kv-engine
      workingDir: /workdir
      securityContext:
        runAsUser: 1002
    untrusted:
      env:
        allow:
          - MEASURED_ENV
          - OTHER_ENV
      paths:
        allow:
          - /some/path
apm:
  url: https://apm.remote.server:8000
  caCert: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----