Troubleshooting

You may see enclave log messages mentioning a tainted kernel like this:

This is an informational message for Linux kernel developers and does not indicate an issue with your host. It is printed when the system loads the Nitro Secure Module kernel module, which is software that is developed by AWS to implement various operations for AWS Nitro Enclaves, including attestation.

The taint message informs kernel developers that a kernel module has been installed and is not included in the mainline Linux kernel, which may be helpful to know when debugging kernel issues. See the Linux kernel documentation on Tainted kernels for more information.

This error indicates that you are trying to run anjuna-nitro-cli run-enclave with an odd-numbered --cpu-count. AWS Nitro Enclaves on Intel and AMD-based instance types require an even number of CPUs due to hyperthreading.

To resolve the error, update --cpu-count to use an even number of CPUs.

You may see error messages similar to the following: Enclave services configuration error. For example, this type of error appears when the CLI fails to parse Anjuna enclave service configuration for anjuna-nitro-netd-parent.

This error indicates that anjuna-nitro-netd-parent for the given --enclave-name cannot be found. The --enclave-name parameter must match for the anjuna-nitro-cli run-enclave and anjuna-nitro-netd-parent commands. Ensure that anjuna-nitro-netd-parent is running with the correct --enclave-name.

You may see error messages like Invalid enclave name 'example@enclave'.

Enclave names can only contain alphanumeric characters, - (dash), _ (underscore), and . (period). Enclave names have a maximum length of 128 characters. Ensure that your --enclave-name parameter meets these requirements.

You may see error messages like Enclave 'example' already exists. Please terminate the existing enclave, or use a different name.

This error indicates that anjuna-nitro-cli run-enclave was called with an --enclave-name that is already used by a running enclave. Each running enclave must have a unique enclave name. The enclave name is used to set up Anjuna Nitro Runtime services for each running enclave.

You can view information about currently-running enclaves using anjuna-nitro-cli describe-enclaves. If you want to run a new enclave without stopping the running one, choose a different --enclave-name for anjuna-nitro-cli run-enclave. If you want to replace the currently-running enclave, stop it with anjuna-nitro-cli terminate-enclave --enclave-name <name> and start the new one with anjuna-nitro-cli run-enclave with the same enclave name.

You may see an error message stating that your EIF was built with an unknown version.

Your EIF version must be the same as your Anjuna Nitro Runtime version. If your versions do not match, and if the EIF was built using Anjuna Nitro Runtime 1.35 or earlier, an error message will state that your EIF was built with an unknown version. You need to rebuild your EIF using your current Anjuna Nitro Runtime version.

When running the anjuna-nitro-cli build-enclave command, you may see an error message like Error writing outputs: Error writing kernel+initrd output: write <filename>: no space left on device. This error occurs when there is not enough disk space to create the Enclave Image File (EIF).

You can check the filesystem with a command like df -h and free up disk space, or you can use an instance with more disk space.

You may see error messages like Invalid config file "config.yaml": yaml: line 8: found unexpected end of stream. This occurs when the --enclave-config-file for anjuna-nitro-cli build-enclave is not a valid YAML file.

See Anjuna Nitro Enclave configuration for more information on the config file syntax. You may find it useful to use a YAML validation tool with your configuration file, and in some scenarios, use YAML multi-line strings.

You may see informational messages like Found config version 1.6. For the latest, use: "version: 1.8". This occurs when you are using an --enclave-config-file with a version that is not the latest version.

The version of the Anjuna Nitro Enclave configuration may be updated to add new fields or fix bugs in enclave behavior. Using the latest config version for your Anjuna Nitro Runtime version is recommended.

You may see error messages like Startup failure: cannot add network services: listen tcp :80: bind: permission denied in the log of anjuna-nitro-netd-parent. This error occurs when you are trying to expose a port numbered less than 1024, but the anjuna-nitro-netd-parent does not have permission to bind on privileged ports.

Following the instructions in Install the Anjuna Nitro Runtime, run the following command to give anjuna-nitro-netd-parent the ability to bind on privileged ports. The command assumes you have installed the Anjuna Nitro Runtime at the default location /opt/anjuna/nitro.

Using setcap is preferred over running anjuna-nitro-netd-parent with sudo. setcap is a one-time setup operation and grants only the needed port binding capability.

You may encounter errors when retrieving instance metadata from AWS IMDS endpoints, with messages such as error getting EC2 region while getting encrypted config: context deadline exceeded when using the default instance metadata configuration.

Applications running in containers may experience an additional hop in their network route to the IMDS endpoint. The default IMDS hop limit is not sufficient to reach it. Ensure that your launch templates configure an appropriate hop limit:

Refer to AWS User Guide to learn more about considerations associated with using instance metadata services.

You may see error messages similar to the following: Failed to validate KMS CMK ARN: NoCredentialProviders: no valid providers in chain. Deprecated. Failed to validate KMS CMK ARN: NoCredentialProviders: no valid providers in chain. Deprecated.

Following are possible causes for this error:

This error occurs when the number of huge memory pages per enclave (4096) has been exhausted, and typically occurs when a small hugepage size is used. For example, when using a 2MiB hugepage size, you will only be able to create an enclave with a maximum memory allocation of 8GiB. This limit is imposed by AWS and may change in the future.

To resolve the issue, increase the system hugepage size to 1GiB. One way to do this is to increase memory_mib in the AWS Nitro allocator configuration, following the instructions in Configure AWS Nitro Enclave resource allocation.

When an application attempts to use shared memory inside an enclave, you may see an error like one of the following:

See Using shared memory in the How-to guides section for information on these errors.

You may experience connection issues for one of the following two reasons:

The enclave is unreachable via IPv4, even though anjuna-nitro-netd-parent is running with the correct ports.

The enclave may fail to detect the anjuna-nitro-netd-parent if you run run-enclave immediately after running anjuna-nitro-netd-parent due to a known race condition. For now, you can add a three-second delay between the two commands using sleep 3. This behavior will be fixed in a future release.

You may see error messages like Can’t proxy a datagram to udp: write udp 127.0.0.1:39187→127.0.0.1:10514: write: connection refused in the log of anjuna-nitro-netd-parent.

The Anjuna Nitro Runtime has limited support for UDP connections. The enclave can send outbound message with UDP, like DNS requests, and responses to those messages can be received. But if the connection is inactive for a long time, it will be closed. This means that if an enclave sends out a UDP packet and a long time passes before the response is sent back, the enclave may not be able to receive the response.

At this time, the Anjuna Nitro Runtime does not support an enclave listening for UDP requests. The enclave will not be able to expose a UDP listener via anjuna-nitro-netd-parent. If you are interested in UDP listeners, please contact support@anjuna.io.

You may see error codes like [E01] or [E59]. These errors are documented on the AWS Error codes page.

When an encrypted config is assigned to an enclave in its configuration file, occasionally that enclave can fail to launch with a log message similar to the following:

This might indicate that the KMS key policy with which the encrypted config was encrypted is not up to date yet across AWS.

Make sure to validate whether the KMS key policy was updated to allow the enclave to perform the kms:Decrypt action.

If the key policy is up to date, then either wait a few minutes for the policy to propagate or configure the environment variable ANJ_ENCLAVE_KMS_DECRYPT_RETRY_TIMEOUT_SECONDS as detailed here.

If you do not see your error documented on this page, try searching the documentation using the search bar at the upper-right corner of the page.

Otherwise, contact support@anjuna.io with the error message, Anjuna Nitro Runtime version number, and relevant context about the action you were trying to perform.