Troubleshooting

This section addresses possible warnings or errors you might encounter while using the Anjuna Nitro Runtime.

For issues with deploying Pods with the Anjuna Nitro Kubernetes Toolset, the Anjuna Nitro Kubernetes Toolset troubleshooting guide may also be helpful.

Loading out-of-tree module taints kernel

You may see enclave log messages mentioning a tainted kernel like this:

[ 3.431418] nsm: loading out-of-tree module taints kernel.
[ 3.431993] nsm: module verification failed: signature and/or required key missing - tainting kernel

This is an informational message for Linux kernel developers and does not indicate an issue with your host. It is printed when the system loads the Nitro Secure Module kernel module, which is software that is developed by AWS to implement various operations for AWS Nitro Enclaves, including attestation.

The taint message informs kernel developers that a kernel module has been installed and is not included in the mainline Linux kernel, which may be helpful to know when debugging kernel issues. See the Linux kernel documentation on Tainted kernels for more information.

The enclave cannot start because full CPU cores have not been set

This error indicates that you are trying to run anjuna-nitro-cli run-enclave with an odd-numbered --cpu-count. AWS Nitro Enclaves on Intel and AMD-based instance types require an even number of CPUs due to hyperthreading.

To resolve the error, update --cpu-count to use an even number of CPUs.

Enclave services configuration error

You may see error messages similar to the following: Enclave services configuration error. For example, this type of error appears when the CLI fails to parse Anjuna enclave service configuration for anjuna-nitro-netd-parent.

This error indicates that anjuna-nitro-netd-parent for the given --enclave-name cannot be found. The --enclave-name parameter must match for the anjuna-nitro-cli run-enclave and anjuna-nitro-netd-parent commands. Ensure that anjuna-nitro-netd-parent is running with the correct --enclave-name.

Invalid enclave name

You may see error messages like Invalid enclave name 'example@enclave'.

Enclave names can only contain alphanumeric characters, - (dash), _ (underscore), and . (period). Enclave names have a maximum length of 128 characters. Ensure that your --enclave-name parameter meets these requirements.

Enclave already exists

You may see error messages like Enclave 'example' already exists. Please terminate the existing enclave, or use a different name.

This error indicates that anjuna-nitro-cli run-enclave was called with an --enclave-name that is already used by a running enclave. Each running enclave must have a unique enclave name. The enclave name is used to set up Anjuna Nitro Runtime services for each running enclave.

You can view information about currently-running enclaves using anjuna-nitro-cli describe-enclaves. If you want to run a new enclave without stopping the running one, choose a different --enclave-name for anjuna-nitro-cli run-enclave. If you want to replace the currently-running enclave, stop it with anjuna-nitro-cli terminate-enclave --enclave-name <name> and start the new one with anjuna-nitro-cli run-enclave with the same enclave name.

EIF was built with an unknown version

You may see an error message stating that your EIF was built with an unknown version.

Your EIF version must be the same as your Anjuna Nitro Runtime version. If your versions do not match, and if the EIF was built using Anjuna Nitro Runtime 1.35 or earlier, an error message will state that your EIF was built with an unknown version. You need to rebuild your EIF using your current Anjuna Nitro Runtime version.

`anjuna-nitro-cli build-enclave`: error writing outputs

When running the anjuna-nitro-cli build-enclave command, you may see an error message like Error writing outputs: Error writing kernel+initrd output: write <filename>: no space left on device. This error occurs when there is not enough disk space to create the Enclave Image File (EIF).

You can check the filesystem with a command like df -h and free up disk space, or you can use an instance with more disk space.

Invalid config file YAML

You may see error messages like Invalid config file "config.yaml": yaml: line 8: found unexpected end of stream. This occurs when the --enclave-config-file for anjuna-nitro-cli build-enclave is not a valid YAML file.

See Anjuna Nitro Enclave configuration for more information on the config file syntax. You may find it useful to use a YAML validation tool with your configuration file, and in some scenarios, use YAML multi-line strings.

New configuration version available

You may see informational messages like Found config version 1.6. For the latest, use: "version: 1.8". This occurs when you are using an --enclave-config-file with a version that is not the latest version.

The version of the Anjuna Nitro Enclave configuration may be updated to add new fields or fix bugs in enclave behavior. Using the latest config version for your Anjuna Nitro Runtime version is recommended.

`anjuna-nitro-netd-parent` bind: permission denied

You may see error messages like Startup failure: cannot add network services: listen tcp :80: bind: permission denied in the log of anjuna-nitro-netd-parent. This error occurs when you are trying to expose a port numbered less than 1024, but the anjuna-nitro-netd-parent does not have permission to bind on privileged ports.

Following the instructions in Install the Anjuna Nitro Runtime, run the following command to give anjuna-nitro-netd-parent the ability to bind on privileged ports. The command assumes you have installed the Anjuna Nitro Runtime at the default location /opt/anjuna/nitro.

$ sudo setcap cap_net_bind_service=+ep /opt/anjuna/nitro/bin/anjuna-nitro-netd-parent

Using setcap is preferred over running anjuna-nitro-netd-parent with sudo. setcap is a one-time setup operation and grants only the needed port binding capability.

Failed to get EC2 region and other IMDS connectivity issues

You may encounter errors when retrieving instance metadata from AWS IMDS endpoints, with messages such as error getting EC2 region while getting encrypted config: context deadline exceeded when using the default instance metadata configuration.

Applications running in containers may experience an additional hop in their network route to the IMDS endpoint. The default IMDS hop limit is not sufficient to reach it. Ensure that your launch templates configure an appropriate hop limit:

metadata_options {
  http_put_response_hop_limit = 2
}

Refer to the AWS User Guide to learn more about considerations associated with using instance metadata services.

Failed to validate KMS CMK ARN

You may see error messages similar to the following: Failed to validate KMS CMK ARN: NoCredentialProviders: no valid providers in chain. Deprecated. Failed to validate KMS CMK ARN: NoCredentialProviders: no valid providers in chain. Deprecated.

Following are possible causes for this error:

Not authenticated - check using aws sts get-caller-identity
Using AWS SSO - this is a limitation of the AWS SDK. Use a tool like jaxxstorm/aws-sso-creds (Github) or follow the steps for Manual credential refresh from Getting IAM Identity Center user credentials for the AWS CLI or AWS SDKs.

The maximum number of memory regions per enclave has been reached

This error occurs when the number of huge memory pages per enclave (4096) has been exhausted, and typically occurs when a small hugepage size is used. For example, when using a 2MiB hugepage size, you will only be able to create an enclave with a maximum memory allocation of 8GiB. This limit is imposed by AWS and may change in the future.

To resolve the issue, increase the system hugepage size to 1GiB. One way to do this is to increase memory_mib in the AWS Nitro allocator configuration, following the instructions in Configure AWS Nitro Enclave resource allocation.

An attempt to communicate over shared memory failed

When an application attempts to use shared memory inside an enclave, you may see an error like one of the following:

/dev/shm (No such file or directory)
could not create shared memory segment
failed to open shm
OMP: Error #179: Function Can’t open SHM2 failed

See Using shared memory in the How-to guides section for information on these errors.

TCP connection issues

You may experience connection issues for one of the following two reasons:

The enclave is unreachable via IPv4, even though anjuna-nitro-netd-parent is running with the correct ports.

The enclave may fail to detect the anjuna-nitro-netd-parent if you run run-enclave immediately after running anjuna-nitro-netd-parent due to a known race condition. For now, you can add a three-second delay between the two commands using sleep 3. This behavior will be fixed in a future release.

UDP connection issues

You may see error messages like Can’t proxy a datagram to udp: write udp 127.0.0.1:39187→127.0.0.1:10514: write: connection refused in the log of anjuna-nitro-netd-parent.

The Anjuna Nitro Runtime has limited support for UDP connections. The enclave can send outbound message with UDP, like DNS requests, and responses to those messages can be received. But if the connection is inactive for a long time, it will be closed. This means that if an enclave sends out a UDP packet and a long time passes before the response is sent back, the enclave may not be able to receive the response.

At this time, the Anjuna Nitro Runtime does not support an enclave listening for UDP requests. The enclave will not be able to expose a UDP listener via anjuna-nitro-netd-parent. If you are interested in UDP listeners, contact support@anjuna.io.

AWS Nitro CLI error codes like [E01]

You may see error codes like [E01] or [E59]. These errors are documented on the AWS Error codes page.

No identity-based policy allows the kms:Decrypt action

When an encrypted config is assigned to an enclave in its configuration file, occasionally that enclave can fail to launch with a log message similar to the following:

FATAL: Failure adding encrypted config: failed to get encrypted config from s3: error retrieving and decrypting encrypted config
...
is not authorized to perform: kms:Decrypt on resource... because no identity-based policy allows the kms:Decrypt action

This might indicate that the KMS key policy with which the encrypted config was encrypted is not up to date yet across AWS.

Make sure to validate whether the KMS key policy was updated to allow the enclave to perform the kms:Decrypt action.

If the key policy is up to date, then either wait a few minutes for the policy to propagate or configure the environment variable ANJ_ENCLAVE_KMS_DECRYPT_RETRY_TIMEOUT_SECONDS as detailed here.

Enclave fails to start: NoSuchKey: The specified key does not exist

An enclave fails to start at runtime when attempting to add the encrypted configuration file, with a message about NoSuchKey.

FATAL: Failure adding encrypted config: failed to get encrypted config from s3: error retrieving
and decrypting encrypted config: error retrieving and decrypting the s3 object
secret-bucket/encrypted-data.bin: NoSuchKey: The specified key does not exist.

This message occurs when you use anjuna-nitro-encrypt --output-file to create a local file, and then upload that file to an S3 bucket. The resulting file is in the wrong format for an S3-type encrypted configuration, causing the error.

S3 and local anjuna-nitro-encrypt results have different encryption formats.

When using an encryptedConfig with type: s3, you must use the S3 object created by anjuna-nitro-encrypt --bucket-name --bucket-key. Manually uploading a local-type encrypted file to S3 will not work.
When using an encryptedConfig with type: local, you must use the file created by anjuna-nitro-encrypt --output-file. Downloading an S3-type encrypted object and using it locally will not work.

To resolve the error, change the enclave configuration to use a local-type encryptedConfig. Or alternatively, create the S3 object using anjuna-nitro-encrypt --bucket-name --bucket-key.

Other errors

If you do not see your error documented on this page, try searching the documentation using the search bar at the upper-right corner of the page.

Otherwise, contact support@anjuna.io with the error message, Anjuna Nitro Runtime version number, and relevant context about the action you were trying to perform.