Introduction
Problem
Configuring the Vault Lambda Extension requires multiple configuration steps between: the Vault AWS auth method, token policies, AWS IAM policies, and the lambda function itself. This guide will attempt to document common errors and how to resolve them.
Prerequisites (if applicable)
This article assumes you have already installed the Vault Lambda Extension, have your auth method and secrets configured, and are encountering unexpected errors executing your Lambda function. For installation instructions see this tutorial.
Error
Response
{
"errorMessage": "2023-01-26T21:05:37.348Z 1b6bda44-9df3-45ce-9b32-1a37f349e03d Task timed out after 3.02 seconds"
}
Possible Solutions
- Check that either the VLE_VAULT_ADDR or VAULT_ADDR environment variables for the lambda function are valid and pointed to the correct port for Vault.
- Check that the security rules for the vault server allow for TCP connections from the lambda function.
For this next error, it is important to check the operational logs for the lambda function to determine the cause of the exit status 1.
Response
{
"errorType": "Extension.Crash",
"errorMessage": "RequestId: 9a239fb7-cd76-4584-95f6-433257d2a15c Error: exit status 1"
}
Some possibilities include:
Error
2023-01-26T21:09:58.266Z [ERROR] Fatal error, exiting: error="missing VLE_VAULT_ADDR, VAULT_ADDR, VAULT_AUTH_PROVIDER or VAULT_AUTH_ROLE environment variables"
Possible Solution
- Check that the required environment variables are added to the lambda function:
VLE_VAULT_ADDR or VAULT_ADDR, VAULT_AUTH_PROVIDER, and VAULT_AUTH_ROLE.
Error
error=
| error logging in to Vault: Error making API request.
|
| URL: PUT http://ec2-00-000-000-000.us-east-1.compute.amazonaws.com:8200/v1/auth/aws/login
| Code: 403. Errors:
|
| * permission denied
Possible Solution
- Check that the Vault AWS auth method is enabled.
- Check that the VAULT_AUTH_PROVIDER environment variable for the lambda function is pointed to the path for the Vault AWS auth method and not the type.
- Check whether the Vault AWS auth method exists in a namespace and whether the VAULT_NAMESPACE environment variable is correctly set for the lambda function.
Error
error=
| error logging in to Vault: Error making API request.
|
| URL: PUT http://ec2-00-000-000-000.us-east-1.compute.amazonaws.com:8200/v1/auth/my-aws-path/login
| Code: 400. Errors:
|
| * entry for role lambda not found
Possible Solution
- Check that the VAULT_ROLE environment variable is correctly set for the lambda function to the name of the Vault role associated with the AWS auth method (not the AWS arn).
- Check that the Vault role exists for the configured AWS auth method. In Vault, try running:
vault list auth/$PATH_TO_AWS_AUTH_METHOD/roles
Error
error=
| error logging in to Vault: Error making API request.
|
| URL: PUT http://ec2-00-000-000-000.us-east-1.compute.amazonaws.com:8200/v1/auth/aws/login
| Code: 400. Errors:
|
| * IAM Principal "arn:aws:sts::000000000000:assumed-role/vault-aws-auth-admin-role/vault-example-1" does not belong to the role "lambda"
Possible Solution
- Check that the bound_arn for the Vault AWS auth method role is set correctly. In Vault, try running:
vault read auth/$PATH_TO_AWS_AUTH_METHOD/role/$ROLE_NAME
- Check that the lambda function has the correct IAM role assigned to it.
Error
| error reading secret: Error making API request.
|
| URL: GET http://ec2-34-219-194-106.us-west-2.compute.amazonaws.com:8200/v1/kv/my-secret
| Code: 403. Errors:
|
| * 1 error occurred:
| * permission denied
Possible Solution:
- Note: that the previous permission denied error in this article has a url pointed to v1/auth/aws/login whereas this error has a url pointed to v1/kv/my-secret. This indicates that the lambda successfully authenticated with Vault but is unable to read the secret path.
- Check that the VAULT_SECRET_PATH or VAULT_SECRET_PATH_FOO environment variables are correctly defined for the lambda function.
- Check that the token_policies associated with the Vault AWS role grant access to the path
Outcome
By default, if the lambda extension is able to read the secret from vault, it will write it to a temporary file at /tmp/vault/secret.json. In testing, you can log the file to output for the lambda function to verify its contents (note: This should only be used for testing:
import fs from 'fs';
export const handler = async(event) => {
let secret = fs.readFileSync('/tmp/vault/secret.json', {encoding:'utf8', flag:'r'})
return JSON.parse(secret);
};