Confidential Pod Configuration File reference

The Anjuna Confidential Pod Configuration file allows users to:

Enforce aspects of the Pod’s containers, to secure their execution, even if the Kubernetes infrastructure or privileged Kubernetes users are compromised.
Change certain behaviors of the containers in a Pod, without needing to modify the original container.
Inject sensitive data into the containers, using remote attestation, to allow for secure release of protected data only to trusted software running in a Confidential Pod.

This configuration is mandatory for every disk build with the Anjuna Kubernetes CLI. It is written in YAML, and in structure it is similar to the Kubernetes Pod definition.

When building Anjuna Confidential Pod disk images, you must set the Anjuna Confidential Pod Configuration file using the command anjuna-k8s-cli build with the --cpod-config=<config_file> parameter. For example, if your Anjuna Confidential Pod Configuration file is called config.yaml, use the following command:

AKS
OpenShift on Google Cloud

$ anjuna-k8s-cli build azure  \
	--cpod-config=config.yaml

$ anjuna-k8s-cli build gcp  \
	--cpod-config=config.yaml

Two main sections of the Confidential Pod configuration file

A Confidential Pod configuration file is built of two main sections — pod and apm.

The pod section is a mandatory section, which defines the containers that run in the Pod. It can define regular containers and sidecars, init containers, and ephemeral, secure empty directories that are shared among the Pod’s containers.

The apm section defines the connection details to the Anjuna Policy Manager (APM). It allows providing initial secrets to the containers in the Anjuna Confidential Pod, from the APM, leveraging the power of remote attestation.

For example, a Confidential Pod configuration file has the following structure:

pod:
  containers:
...
apm:
...

The rest of this page describes these two sections in detail.

`pod` section

The pod section in the Confidential Pod configuration file is specified under the pod entry. It supports the following subentries:

containers — a mandatory list of the regular containers, such as the application and sidecars, intended to run inside of the Pod.
initContainers — an optional list of init containers, which run before the regular containers start.
emptyDirs — an optional list of secure ephemeral empty directories that can be shared among the Pod’s containers.

For example:

pod:
  initContainers:
  - ...
  containers:
  - ...
  emptyDirs:
  - ...

Every container name in the pod section must be unique across both the containers and initContainers lists.

The containers and initContainers subentries share the same container schema, described below. Each container entry needs to specify the name of the container, and the URI of the container’s image. It may also specify the command to be used by the container, and the environment used by the container (variables, files, working directory, security context, and so on).

An example of a container configuration:

  - name: hello
    image: your-company.io/hello-world:v1
    command: [ 'sh', '-c', 'echo "Hello"']
    env:
    - name: MEASURED_ENV
      value: "Measured!"
    - name: SECRET_ENV
      valueFrom:
          apm:
            path: /some/secret/path
            enginePath: kv-engine
    files:
    - path: /some/path
      mode: 0644
      owner: root
      group: root
      value: |
        some file content
    - path: /some/other/path
      valueFrom:
        apm:
            path: /apm/path/to/file
            enginePath: kv-engine
      workingDir: /workdir
      securityContext:
        runAsUser: 1002
    untrusted:
      env:
        allow:
          - MEASURED_ENV
          - OTHER_ENV
      paths:
        allow:
          - /some/path
  - name: sidecar
    image: your-company.io/sidecar:v1

Container name - mandatory

The name entry is a mandatory part of a container configuration. When creating a container inside of the Confidential Pod configuration file, the name of the container is used as the primary identifier. The container configuration specified in the Pod configuration is applied based on that name.

Therefore, the name of the container in the Anjuna Confidential Pod Configuration file needs to match the name provided in the Kubernetes Pod spec. For example:

pod:
  containers:
  - name: my-container
    ...

Container image - mandatory

The image entry of a container is used at disk build time to decide what image to embed in the Anjuna Confidential Pod disk image. It is then used at runtime to decide which image to launch for each container.

This image must therefore be provided as a full URI. For example:

pod:
  containers:
  - image: docker.io/library/nginx:1.27.3
    ...

Container command and args - optional

The command and args entries of the container are used to specify the command and args that the container in the Anjuna Confidential Pod will run. These fields should be provided as lists. If any of the fields are omitted in the Anjuna Confidential Pod configuration file, their values are instead derived from the container image being measured:

command is derived from the image ENTRYPOINT
args is derived from the image CMD

This matches Kubernetes' default behavior when the Pod definition omits command and args. The combination of command and args provided by Kubernetes at deploy time must match the build-time command and args, otherwise the container will fail to start. To avoid this mismatch, mirror the expected command and args from the Pod definition when creating the Anjuna Confidential Pod configuration file.

Example:

pod:
  containers:
  - command: ["bash"]
    args: ["-c", "echo 'hello, world'"]
    ...

Container environment variables - optional

The env entry of the container is used to add environment variables that the container runs with, in addition to those already defined in the container image. This is a list of environment entries. There are two ways of specifying values for these environment variables: either literally or using the APM.

Each environment variable must define either a literal value or an APM-sourced value, but never both.

Literal environment variables

When setting environment variables in this section, the Anjuna Runtime will set these environment variables to the literal values specified in the Confidential Pod configuration. This option allows setting the values for environment variables without the need to change the original container. When providing the values literally, use the value subentry. For example:

pod:
  containers:
  - env:
    - name: my-public-env
      value: some-public-value
    ...

APM sourced environment variables

When setting environment variables in this section, the Anjuna Runtime will use remote attestation to inject secrets into the container. These secrets are fetched from the APM based on the Confidential Pod identity (as presented in the attestation quote).

When providing the values through the APM, use the valueFrom entry with a nested apm entry.

In the apm entry, specify:

path — the path in the APM where the value is stored
enginePath — the path of the secret engine where the value is stored

For example:

pod:
  containers:
  - env:
    - name: my-secret-value
      valueFrom:
        apm:
          path: path/to/secret
          enginePath: some-kv-engine
    ...

Container files - optional

The files entry of the container is used to define files that will be created for the container. This is a list of file entries. In terms of metadata, each file entry can define path, mode, owner, and group. There are two ways of specifying the content for these files: either literally or using the APM.

Each file must define either literal content or APM-sourced content, but never both.

Literal files

When setting files in this section, the Anjuna Runtime creates these files with the specified content inside the Container when it starts. This option allows setting files with the needed content, without the need to change the original container. When providing the values literally, use the value subentry. For example:

pod:
  containers:
   - files:
      - path: /etc/hosts
        value: |
          127.0.0.1 my.local.server
    ...

You can define values similarly to environment variables, using either a literal value or valueFrom to fetch the value from the APM, as illustrated below.

APM sourced files

When setting file contents from the APM, the Anjuna Runtime will use remote attestation to inject secrets into the container filesystem. These secrets are fetched from the APM based on the Confidential Pod identity (as presented in the attestation quote).

When providing the values through the APM, use the valueFrom entry with a nested apm entry. In the apm entry, specify:

path — the path in the APM where the value is stored
enginePath — the path of the secret engine where the value is stored

For example:

pod:
  containers:
  - files:
    - name: /home/someuser/.ssh/id_ecdsa
      owner: someuser
      group: someuser
      mode: 0600
      valueFrom:
        apm:
          path: path/to/ssh/key
          enginePath: some-kv-engine
    ...

Container working directory - optional

The workingDir entry of the container is used to define the working directory that the container’s main process will use. For example, if the application expects to run in the /work/dir directory, you can specify it using this entry:

pod:
  containers:
  - workingDir: /work/dir
    ...

Container security context - optional

The securityContext entry of the container is used to define permissions associated with the container’s main process. For now, it only supports the runAsUser subentry, which can be used to define the user ID that the main process will run under. For example, you can configure the application to run under a specific user ID, such as 1000:

pod:
  containers:
  - securityContext:
      runAsUser: 1000
    ...

Container untrusted configuration

The Anjuna Confidential Pod Configuration provides the user a trusted and measured configuration that is attached to the disk at build time. However, there are cases where applications require data that cannot be known in advance, for example:

The IP address of a logging server
Values fetched from the Kubernetes Downward API
Values provided to the Anjuna Confidential Pod via Secrets or ConfigMaps

Using the untrusted entry of the container, the Anjuna Kubernetes Toolset lets you allow this type of untrusted configuration through environment variables, volume mounts, and the container’s command and arguments.

Understand the consequences of allowing untrusted environment variables, volume mounts, or command and arguments to be set from the Pod specification. These values can be manipulated by potential attackers to alter the behavior of your Anjuna Confidential Pod in undesirable ways. Always verify in your application that the values and data provided through these sources are correct and valid before using them.

Untrusted environment variables

By default, all the environment variables in a Pod specification are ignored since the cluster itself is considered to be untrusted. For instance, a malicious cluster administrator could set arbitrary environment variables by editing your Pod specification on the fly, thus affecting the behavior of the Pod.

Only environment variables that are previously measured as part of the container image or as part of the Anjuna Confidential Pod Configuration file are considered trusted. These trusted environment variables are effectively applied to the Anjuna Confidential Pod in Kubernetes. Environment variables that are injected from APM secrets are also considered trusted, because they are only injected into trustworthy Anjuna Confidential Pods.

To allow an untrusted environment variable to be applied from the Anjuna Confidential Pod specification, you must explicitly allow it in the Anjuna Confidential Pod Configuration file via the untrusted.env field.

For example:

pod:
  containers:
  - name: my-container
    untrusted:
      env:
        allow:
        - LOG_SERVER_ADDRESS
        - LOG_LEVEL
        - SERVICE_ADDRESS

Untrusted environment variables can be supplied by untrusted entities and are a potential attack vector. Ensure that your application does not use them blindly. Their values must be thoroughly validated before use.

Once you build your Anjuna Confidential Pod image with this configuration, the values of the environment variables LOG_SERVER_ADDRESS, LOG_LEVEL, and SERVICE_ADDRESS will be propagated if set in the Pod specification.

For instance, the Pod specification below:

Sets the value of LOG_LEVEL directly in the Pod spec
Sets the value of LOG_SERVER_ADDRESS from a ConfigMap
Fetches the value of SERVICE_ADDRESS from the Kubernetes Downward API

apiVersion: v1
kind: Pod
# <snip>...
metadata:
  labels:
    anjuna.io/run-confidential: "yes"
spec:
  containers:
    # <snip>...
    env:
    - name: LOG_LEVEL
      value: "debug"
    - name: LOG_SERVER_ADDRESS
      valueFrom:
        configMapKeyRef:
          name: logging
          key: server_address
    - name: SERVICE_ADDRESS
      valueFrom:
        fieldRef:
          fieldPath: status.podIP
    # <snip>...

This logging ConfigMap, referenced by the Pod specification above, is defined as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: logging
data:
  server_address: "my-log-server.anjuna.io"

If the Anjuna Confidential Pod Configuration file specifies a trusted value for any of those variables through the container’s env entry, it takes precedence over the untrusted value.

If you define the same environment variable in more than one place, the following describes the precedence used:

The env section you provide in the Anjuna Confidential Pod Configuration file at build time takes the highest precedence.
The allowed env entries in the untrusted section take second precedence; they apply only if the container’s env section does not already set the same environment variable.
Finally, the build-time container image sets environment variables with the lowest precedence.

Untrusted volume mounts

Similar to environment variables, all volume mounts defined in the Pod specification are considered untrusted and therefore ignored by default, unless explicitly allowed through the untrusted.paths field.

For example:

pod:
  containers:
  - name: my-container
    untrusted:
      paths:
        allow:
        - /var/run/secrets/kubernetes.io/serviceaccount
        - /usr/share/nginx/html/index.html

Untrusted mount paths can be supplied by untrusted entities and are a potential attack vector. Ensure that your application does not use mounted paths and files blindly. Their content must be thoroughly validated before use. Consider applying encryption or integrity-protection techniques to ensure your application uses the data it expects.

The Anjuna Confidential Pod Configuration file above allows volume mounts for /var/run/secrets/kubernetes.io/serviceaccount (the Kubernetes service account token) and /usr/share/nginx/html/index.html.

For instance, the Pod specification below mounts a file at /usr/share/nginx/html/index.html from the ConfigMap custom-page. Kubernetes automatically mounts the service account token to all Pods by default. The Pod specification is defined as follows:

apiVersion: v1
kind: Pod
# <snip>...
metadata:
  labels:
    anjuna.io/run-confidential: "yes"
spec:
  containers:
  # <snip>...
  volumeMounts:
  - name: custom-page
    mountPath: /usr/share/nginx/html/index.html
    subPath: index.html
  # <snip>...
  volumes:
  - name: custom-page
    configMap:
      name: custom-page

This custom-page ConfigMap, referenced by the Pod specification above, is defined as follows:

apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-page
data:
  index.html: "This is my custom page!"

If the Anjuna Confidential Pod Configuration file specifies a trusted file for any of the mount paths through the files field, it takes precedence over the untrusted volume mount from the Pod specification.

Untrusted command and arguments

By default, the command line (command and args) that a container runs with is fixed at build time. It comes either from the container image, or from the command and args entries of the container in the Anjuna Confidential Pod Configuration file (see Container command and args). If the cluster control plane requests a command line that does not match this fixed command line, the container fails to start.

Sometimes, parts of the command line cannot be known in advance — for example, a value that is only available at deployment time. For these cases, the untrusted.command.allow and untrusted.args.allow fields let you allow the control plane to supply the command and arguments at runtime, constrained to patterns that you define.

untrusted.command corresponds to the container’s command (the Docker ENTRYPOINT), and untrusted.args corresponds to its arguments (the Docker CMD). Each is an allow list, where every element is a pattern that a single command-line token must match.

A given field can only be specified once, either in its trusted or its untrusted form, but never both. For the same container, you cannot specify both command and untrusted.command, and you cannot specify both args and untrusted.args.

Patterns

Each element of an untrusted.command or untrusted.args allow list is a pattern.

The * character is a wildcard that matches a sequence of word characters, dots (.), hyphens (-), and spaces. To match a literal * character, escape it with a backslash as \*. A literal backslash is written as \\.

For example, the pattern FOO=* matches FOO=bar and FOO=baz, while the pattern \* matches only the literal string *.

Matching the requested command line

The command and arguments are matched together, token by token, against the combined pattern built from the trusted and untrusted configuration. A token can be defined in either the trusted section (command, args) or the untrusted section (untrusted.command, untrusted.args). Regardless of where it is defined, the resulting combination must match the command line that the cluster control plane requests to run. If the requested command line does not match, the container will fail to start.

Example

The following container allows the control plane to supply a command of /bin/bash -c, and arguments consisting of echo followed by a token of the form FOO=<value>:

pod:
  containers:
  - name: my-container
    image: docker.io/library/ubuntu:24.04
    command:
      - /bin/bash
      - -c
    untrusted:
      args:
        allow:
          - echo
          - FOO=*

Patterns are especially useful for applications whose arguments are injected by the cluster at deployment time. For example, the following configuration allows a sidecar to be launched with arguments that vary between deployments:

pod:
  containers:
  - name: my-proxy
    image: docker.io/my-company/my-proxy:v1.0
    untrusted:
      args:
        allow:
          - proxy
          - sidecar
          - --domain
          - '*.svc.cluster.local'
          - --proxyLogLevel=warning
          - --log_output_level=default:info

Init containers - optional

The initContainers entry of the pod section defines containers that run before the regular containers start. Init containers are commonly used to perform setup tasks, such as:

Preparing data in a shared empty directory (see Empty directories - optional) for the regular containers to consume
Running as sidecars, when configured with restartPolicy: Always

Init containers use the same schema as regular containers, so all of the container fields described above apply to them as well. Container names must be unique across both the initContainers and containers lists.

For example, the following configuration defines an init container that writes a file before the regular container runs:

pod:
  emptyDirs:
  - name: shared
  initContainers:
  - name: init
    image: docker.io/library/ubuntu:24.04
    command: [ 'bash', '-c', 'echo "ready" > /shared/status' ]
    emptyDirMounts:
    - name: shared
      mountPath: /shared
  containers:
  - name: my-container
    image: docker.io/library/ubuntu:24.04
    emptyDirMounts:
    - name: shared
      mountPath: /shared

Empty directories - optional

An empty directory is an ephemeral, confidential directory located inside the CVM. Its contents are not shared with, or accessible by, the worker Node or any entity outside the CVM. The same empty directory can be mounted into multiple containers, which makes it a convenient mechanism for sharing data between the containers of an Anjuna Confidential Pod.

Configuring an empty directory involves two entries:

emptyDirs, at the pod level, declares the available empty directories by name.
emptyDirMounts, at the container level, mounts a declared empty directory into a container.

Declaring empty directories

The emptyDirs entry of the pod section is a list of empty directories. Each entry defines a single name, which must be unique within the Pod.

pod:
  emptyDirs:
  - name: shared

Mounting empty directories

The emptyDirMounts entry of a container is a list of mounts. Each mount specifies:

name — the name of an empty directory declared in the pod’s `emptyDirs list.
mountPath — the absolute path at which the empty directory is mounted inside the container.

A container cannot mount more than one empty directory at the same mountPath. Additionally, a mountPath used by an emptyDirMount cannot also be listed as an allowed untrusted path (see Container untrusted configuration) for the same container.

The following example shows two containers sharing the same empty directory, so that data written by one is visible to the other:

pod:
  emptyDirs:
  - name: shared
  initContainers:
  - name: producer
    image: docker.io/library/ubuntu:24.04
    command: [ 'bash', '-c', 'echo "hello" > /shared/message' ]
    emptyDirMounts:
    - name: shared
      mountPath: /shared
  containers:
  - name: consumer
    image: docker.io/library/ubuntu:24.04
    command: [ 'bash', '-c', 'cat /shared/message' ]
    emptyDirMounts:
    - name: shared
      mountPath: /shared

Each empty directory declared in the Anjuna Confidential Pod Configuration file must also be requested by the cluster control plane through the Pod specification. This request must take the form of a volume mount backed by an emptyDir volume that uses the Memory medium. If a matching, correctly-configured emptyDir volume is not requested in the Pod specification, the affected container will fail to start.

For example, the Pod specification below provides the shared empty directory used in the configuration above:

apiVersion: v1
kind: Pod
# <snip>...
metadata:
  labels:
    anjuna.io/run-confidential: "yes"
spec:
  containers:
    # <snip>...
    volumeMounts:
    - name: shared
      mountPath: /shared
  volumes:
  - name: shared
    emptyDir:
      medium: Memory

If the Anjuna Confidential Pod Configuration file does not specify any emptyDirs, but the cluster control plane requests one via the Pod specification, you can only allow this mount path via untrusted.paths.allow. Note that in this case, the data will reside on the Node’s filesystem, accessible to any cluster admin or privileged Kubernetes users, not inside the Anjuna Confidential Pod.

`apm` section

The apm section configures communication with the APM server. It has two configuration parameters:

url — the URL where the APM resides.
caCert — the APM’s certificate, used to verify the APM’s identity.

The following sections describe each parameter in detail.

`url`

This field points to the URL where the APM resides. This is a mandatory field if you use the apmConfig entry.

The field must start with either https:// or http://. If you use https://, you must also define the caCert field described below.

For example, if the APM sits on port 8200 of a server with the DNS name apm.mydomain.com, use the entry:

...
apm:
  url: https://apm.mydomain.com:8200
  caCert: |
	-----BEGIN CERTIFICATE-----
	... APM's certificate content ...
	-----END CERTIFICATE-----

`caCert`

This field points to the APM’s certificate. When you specify this field, the Anjuna Confidential Container instance verifies the APM’s identity using this certificate.

For example, if the APM sits on port 8200 of a server with the DNS name apm.mydomain.com, and it is using TLS connections, use the entry:

...
apm:
  url: https://apm.mydomain.com:8200
  caCert: |
	-----BEGIN CERTIFICATE-----
	... APM's certificate content ...
	-----END CERTIFICATE-----

Complete example

This example uses every aspect of the configuration described on this page.

pod:
  initContainers:
  - name: init
    image: hello-world
    command: [ 'sh', '-c', 'echo "ready" > /shared/status' ]
    emptyDirMounts:
    - name: shared
      mountPath: /shared
  containers:
  - name: hello
    image: hello-world
    command: [ 'sh', '-c', 'echo "Hello"']
    env:
    - name: MEASURED_ENV
      value: "Measured!"
    - name: SECRET_ENV
      valueFrom:
          apm:
            path: /some/secret/path
            enginePath: kv-engine
    files:
    - path: /some/path
      mode: 0644
      owner: root
      group: root
      value: |
        some file content
    - path: /some/other/path
      valueFrom:
        apm:
            path: /apm/path/to/file
        enginePath: kv-engine
      workingDir: /workdir
      securityContext:
        runAsUser: 1002
    untrusted:
      env:
        allow:
          - MEASURED_ENV
          - OTHER_ENV
      paths:
        allow:
          - /some/path
    emptyDirMounts:
    - name: shared
      mountPath: /shared
  emptyDirs:
  - name: shared
apm:
  url: https://apm.remote.server:8000
  caCert: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

Matching the Anjuna Confidential Pod Configuration file and the Pod specification

Most of the Anjuna Confidential Pod configuration file items mirror the Kubernetes Pod definition, which simplifies creating Anjuna Confidential Pod configuration files.

This mirroring also makes it easy to compare the trusted configuration set at build time against what the Kubernetes cluster actually supplies at deploy time:

At build time, the runtime measures and trusts the values you specify in the Anjuna Confidential Pod Configuration file.
At deploy time, the Kubernetes cluster supplies its own values, which privileged third-parties, such as cluster admins, can see and alter.

Behavior on mismatch

When there is a mismatch between the trusted/measured value and the Pod definition submitted by the cluster, the Anjuna Confidential Pod behaves as follows:

Anjuna Confidential Pod container configuration Kubernetes equivalent Behavior on mismatch

Anjuna Confidential Pod container configuration	Kubernetes equivalent	Behavior on mismatch
`name`	`containers[].name`	Fail — the container name in the Pod definition must match the name in the Anjuna Confidential Pod configuration.
`image`	`containers[].image`	Warn — proceed with the image defined in the Anjuna Confidential Pod configuration.
`command` + `args` (or `untrusted.command` and `untrusted.args` allow lists)	`containers[].command` and `containers[].args`	Fail — the container’s combined command line (`command` + `args`) in the Pod definition must match the combined command line in the Anjuna Confidential Pod configuration. If `command` or `args` is omitted from the Anjuna Confidential Pod configuration file, the container image’s command line (`ENTRYPOINT`, `CMD`) is used instead.
`workingDir`	`containers[].workingDir`	Warn — proceed with the working directory defined in the Anjuna Confidential Pod configuration (or in the image’s `WORKDIR` if missing).
`securityContext.runAsUser`	`containers[].securityContext.runAsUser`	Warn — proceed with the user ID defined in the Anjuna Confidential Pod configuration (or in the image’s `USER` if missing).
`env`	`containers[].env`	Proceed with the environment variables defined in the Anjuna Confidential Pod (configuration + image). Propagate environment variables from the Kubernetes Pod definition only if explicitly allowed in `untrusted.env.allow`; otherwise, ignore them.
`files`	`containers[].volumeMounts` (with `volumes`)	Proceed with the files defined in the Anjuna Confidential Pod configuration. Propagate mount paths from the Kubernetes Pod definition only if explicitly allowed in `untrusted.paths.allow`; otherwise, ignore them.

name

containers[].name

Fail — the container name in the Pod definition must match the name in the Anjuna Confidential Pod configuration.

image

containers[].image

Warn — proceed with the image defined in the Anjuna Confidential Pod configuration.

command + args (or untrusted.command and untrusted.args allow lists)

containers[].command and containers[].args

Fail — the container’s combined command line (command + args) in the Pod definition must match the combined command line in the Anjuna Confidential Pod configuration. If command or args is omitted from the Anjuna Confidential Pod configuration file, the container image’s command line (ENTRYPOINT, CMD) is used instead.

workingDir

containers[].workingDir

Warn — proceed with the working directory defined in the Anjuna Confidential Pod configuration (or in the image’s WORKDIR if missing).

securityContext.runAsUser

containers[].securityContext.runAsUser

Warn — proceed with the user ID defined in the Anjuna Confidential Pod configuration (or in the image’s USER if missing).

env

containers[].env

Proceed with the environment variables defined in the Anjuna Confidential Pod (configuration + image). Propagate environment variables from the Kubernetes Pod definition only if explicitly allowed in untrusted.env.allow; otherwise, ignore them.

files

containers[].volumeMounts (with volumes)

Proceed with the files defined in the Anjuna Confidential Pod configuration. Propagate mount paths from the Kubernetes Pod definition only if explicitly allowed in untrusted.paths.allow; otherwise, ignore them.

Example - Start from Kubernetes specification

Build an Anjuna Confidential Pod Configuration file for the Deployment manifest below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: docker.io/library/nginx:1.14.2
        ports:
        - containerPort: 80

Since this is a Deployment manifest, use the Pod template (.spec.template) as the basis for the Anjuna Confidential Pod. Start by specifying the container with its name and image:

pod:
  containers:
  - name: nginx
    image: docker.io/library/nginx:1.14.2

This provides the baseline for building the Anjuna Confidential Pod image for Nginx. Note that the command and args fields are not specified. This is not an issue, because the Deployment manifest also does not set command and args: the cluster runs the container image’s default command line. This default command line is also the one measured in the Anjuna Confidential Pod.

Now suppose the Deployment configures a different command line:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: docker.io/library/nginx:1.14.2
        ports:
        - containerPort: 80
        command:
        - nginx
        args:
        - -g
        - "worker_processes 2;"

The cluster will try to run nginx -g "worker_processes 2;", which does not match the container image default command line.

In this case, you will need to update the Anjuna Confidential Pod configuration to specify the same command; otherwise, the Anjuna Confidential Pod will fail to start.

pod:
  containers:
  - name: nginx
    image: docker.io/library/nginx:1.14.2
    command:
    - nginx
    - -g
    - "worker_processes 2;"

The Anjuna Confidential Pod validates the combined command line (command and args) against the values specified by Kubernetes in the Pod specification. Because of this, the following configuration produces identical behavior:

pod:
  containers:
  - name: nginx
    image: docker.io/library/nginx:1.14.2
    command:
    - nginx
    args:
    - -g
    - "worker_processes 2;"

You can continue to build on top of this configuration by leveraging other fields of the Anjuna Confidential Pod Configuration file, for example:

Define environment variables for the Nginx container via env.
Specify the Nginx configuration file via files.
Configure TLS certificates that are injected by the APM after completing remote attestation (via files.valueFrom.apm).
Relax the allowed command and args values by specifying a pattern to match against what the cluster attempts to run (via untrusted.command and untrusted.args).

Refer to the Quickstart guides for end-to-end examples on how to build and deploy Anjuna Confidential Pods.

Example - Start from a running Pod

Another way to craft your Anjuna Confidential Pod configuration file is to use an existing running Pod as your starting point. This approach is often preferred over starting from the static Kubernetes Pod definition, because the cluster control plane can change that definition at deploy time. For example, the control plane might inject additional containers and volume mounts during admission. These additions are not visible in the static Pod definition, but the Anjuna Confidential Pod still needs them to work as expected.

In this example, you will create an Anjuna Confidential Pod for an API microservice. Start by looking at its live cluster specification:

$ kubectl get pods billing-api -o yaml

This Pod carries annotations (vault.hashicorp.com/agent-inject) that prompt the HashiCorp Vault Operator to inject init containers and sidecars during Pod creation:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/agent-inject-secret-db: secret/data/billing/db
    vault.hashicorp.com/role: billing-api
  name: billing-api
spec:
  initContainers:
  - name: vault-agent-init
    image: docker.io/hashicorp/vault:1.15
    args:
    - agent
    - -config=/vault/config/init-config.hcl
    env:
    - name: VAULT_ADDR
      value: https://vault.vault.svc:8200
    volumeMounts:
    - mountPath: /vault/secrets
      name: vault-secrets
    - mountPath: /vault/config
      name: vault-config
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-zv6cb
  containers:
  - name: billing-api
    image: mycorp.io/billing-api:2.4.1
    volumeMounts:
    - mountPath: /vault/secrets
      name: vault-secrets
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-zv6cb
  - name: vault-agent
    image: docker.io/hashicorp/vault:1.15
    args:
    - agent
    - -config=/vault/config/config.hcl
    env:
    - name: VAULT_ADDR
      value: https://vault.vault.svc:8200
    volumeMounts:
    - mountPath: /vault/secrets
      name: vault-secrets
    - mountPath: /vault/config
      name: vault-config
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-zv6cb
      readOnly: true
  volumes:
  - name: vault-secrets
    emptyDir:
      medium: Memory
  - name: vault-config
    emptyDir:
      medium: Memory
  - name: kube-api-access-zv6cb
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

The running Pod is composed of three containers:

One init container called vault-agent-init. The Vault Operator injected this init container at deploy time.
The main application container called billing-api.
One sidecar named vault-agent. The Vault Operator injected this container at deploy time.

These three containers form the starting point for the Anjuna Confidential Pod configuration file:

pod:
  initContainers:
  - name: vault-agent-init
    image: docker.io/hashicorp/vault:1.15
    args:
    - agent
    - -config=/vault/config/init-config.hcl
  containers:
  - name: billing-api
    image: mycorp.io/billing-api:2.4.1
  - name: vault-agent
    image: docker.io/hashicorp/vault:1.15

The next step is to lock down what each container runs, by mirroring their command and args fields:

pod:
  initContainers:
  - name: vault-agent-init
    image: docker.io/hashicorp/vault:1.15
  containers:
  - name: billing-api
    image: mycorp.io/billing-api:2.4.1
  - name: vault-agent
    image: docker.io/hashicorp/vault:1.15
    args:
    - agent
    - -config=/vault/config/config.hcl

Note that the billing-api container in the Pod does not specify any command or args, which means that the default ENTRYPOINT and CMD of the container image are being used. In this case, you can omit the fields in the Anjuna Confidential Pod configuration as well.

Now, lock down the environment of each container based on the Pod specification:

pod:
  initContainers:
  - name: vault-agent-init
    image: docker.io/hashicorp/vault:1.15
    env:
    - name: VAULT_ADDR
      value: https://vault.vault.svc:8200
  containers:
  - name: billing-api
    image: mycorp.io/billing-api:2.4.1
  - name: vault-agent
    image: docker.io/hashicorp/vault:1.15
    args:
    - agent
    - -config=/vault/config/config.hcl
    env:
    - name: VAULT_ADDR
      value: https://vault.vault.svc:8200

Finally, you can lock down the volume mounts as well. The running Pod specification defines three volumes and corresponding volumeMounts, all injected at deploy time by the cluster control plane and operators:

An emptyDir volume named vault-secrets, mounted at /vault/secrets.
An emptyDir volume named vault-config, mounted at /vault/config.
A projected volume named kube-api-access-zv6cb, which contains the Kubernetes API service account token.

The two emptyDir volumes just define a scratch space where containers can share data. This data, however, lives on the Node’s filesystem and can be inspected/altered by privileged third-parties. This is even more problematic given that the containers will share Vault secrets via these mounts.

For the Anjuna Confidential Pod, you can specify an emptyDir entry that ensures that the shared mount will only exist inside the Anjuna Confidential Pod, which is encrypted and cannot be accessed by anyone:

pod:
  emptyDirs:
  - name: vault-secrets
  - name: vault-config
  initContainers:
  - name: vault-agent-init
    image: docker.io/hashicorp/vault:1.15
    env:
    - name: VAULT_ADDR
      value: https://vault.vault.svc:8200
    emptyDirMounts:
    - name: vault-config
      mountPath: /vault/config
    - name: vault-secrets
      mountPath: /vault/secrets
  containers:
  - name: billing-api
    image: mycorp.io/billing-api:2.4.1
    emptyDirMounts:
    - name: vault-secrets
      mountPath: /vault/secrets
  - name: vault-agent
    image: docker.io/hashicorp/vault:1.15
    args:
    - agent
    - -config=/vault/config/config.hcl
    env:
    - name: VAULT_ADDR
      value: https://vault.vault.svc:8200
    emptyDirMounts:
    - name: vault-config
      mountPath: /vault/config
    - name: vault-secrets
      mountPath: /vault/secrets

For the third mount, which is the Kubernetes Service Account token, you must explicitly allow it in the Anjuna Confidential Pod configuration, because it comes from an untrusted source (the cluster).

Although the token is mounted to all containers in the Pod by default, assume for this example that only the Billing API container actually requires the Kubernetes token. In this case, you can allow it explicitly for that container via untrusted.paths.allow:

pod:
  emptyDirs:
  - name: vault-secrets
  - name: vault-config
  initContainers:
  - name: vault-agent-init
    image: docker.io/hashicorp/vault:1.15
    env:
    - name: VAULT_ADDR
      value: https://vault.vault.svc:8200
    emptyDirMounts:
    - name: vault-config
      mountPath: /vault/config
    - name: vault-secrets
      mountPath: /vault/secrets
  containers:
  - name: billing-api
    image: mycorp.io/billing-api:2.4.1
    emptyDirMounts:
    - name: vault-secrets
      mountPath: /vault/secrets
    untrusted:
      paths:
        allow:
        - /var/run/secrets/kubernetes.io/serviceaccount
  - name: vault-agent
    image: docker.io/hashicorp/vault:1.15
    args:
    - agent
    - -config=/vault/config/config.hcl
    env:
    - name: VAULT_ADDR
      value: https://vault.vault.svc:8200
    emptyDirMounts:
    - name: vault-config
      mountPath: /vault/config
    - name: vault-secrets
      mountPath: /vault/secrets

The Anjuna Confidential Pod configuration above matches the billing-api Pod’s specification you examined earlier in this example. It builds and runs an Anjuna Confidential Pod that achieves the same behavior, but with the following protections:

The container images cannot be altered or tampered with by third-parties.
The command and args executed by each one of the containers cannot be altered.
The environment variables for each container cannot be altered or tampered with.
The secrets and configurations that are shared in emptyDirs only exist in the Anjuna Confidential Pod, not on the shared Node’s filesystem.
No one can introduce new volume mounts without having them be explicitly allowed in the Anjuna Confidential Pod configuration.

Furthermore, secrets can be delivered via the APM after ensuring that the measurements of your Anjuna Confidential Pod match the expected value.

Confidential Pod Configuration File reference

Two main sections of the Confidential Pod configuration file

pod section

Container name - mandatory

Container image - mandatory

Container command and args - optional

Container environment variables - optional

Literal environment variables

APM sourced environment variables

Container files - optional

Literal files

APM sourced files

Container working directory - optional

Container security context - optional

Container untrusted configuration

Untrusted environment variables

Untrusted volume mounts

Untrusted command and arguments

Patterns

Matching the requested command line

Example

Init containers - optional

Empty directories - optional

Declaring empty directories

Mounting empty directories

apm section

url

caCert

Complete example

Matching the Anjuna Confidential Pod Configuration file and the Pod specification

Behavior on mismatch

Example - Start from Kubernetes specification

Example - Start from a running Pod

`pod` section

`apm` section

`url`

`caCert`