A lot of organizations use Ansible Automation Platform (AAP) to orchestrate their infrastructure and Hashicorp Vault to manage their secrets. But how do they work together? Sometimes we @PuzzleITC don’t work out things on our own but in cooperation with our customers. These insights resulted from a collaboration with Beni Keller who worked at Schwyzer Kantonalbank.
Both AAP and Hashicorp Vault are well-known and well-established tools in enterprise environments. AAP provides native integration for Hashicorp Vault. But which features already work out of the box and in which use-cases do we need to put in some extra effort?
Note: In the below text, we use the term „Controller“ as synonym of Ansible Automation Controller. The Controller is the part of the AAP that is actually running Ansible through Execution Environments and the direct successor or Ansible Tower. Read more about the topic in this previous blogpost (in german).
Ansible without Controller
Combining Hashicorp Vault secrets with Ansible is no rocket science. There is a well-tested collection community.hashi_vault that provides modules and plugins to access and interact with a Hashicorp Vault server. It’s straight forward to use these modules with just plain Ansible. The following example shows how we could print out the value of a Vault secret using the hashi_vault lookup plugin and a token:
- ansible.builtin.debug:
msg: "{{ lookup('community.hashi_vault.hashi_vault', \
'secret=secret/puzzle/prod/root:rootpw_crypted \
token=a95537b0-b1fe-3f22-8bc4-07df59a68aa5
url=https://vault.puzzle.ch:8200') }}"
This works, but since the token is sensitive information, we don’t want to write it unencrypted in our playbooks. The „hashi_vault lookup“ lookup plugin can make use of some environment and Ansible varibles. So if we set ANSIBLE_HASHI_VAULT_TOKEN and ANSIBLE_HASHI_VAULT_URL as environment variables or ansible_hashi_vault_token and ansible_hashi_vault_url as Ansible variables in place, we can even slim down our task from above to:
- ansible.builtin.debug:
msg: "{{ lookup('community.hashi_vault.hashi_vault', \
'secret=secret/puzzle/prod/root:rootpw_crypted }}"
With proper environment or Ansible variables in place, this works just as well. If you prefer to use an approle rather than a token (in Hashi Vault), the environment variables ANSIBLE_HASHI_VAULT_SECRET_ID and ANSIBLE_HASHI_VAULT_ROLE_ID and their respective lowercase Ansible variable pendents are your friend.
Up to Ansible 2.5 the hashi_vault lookup plugin used the variable names VAULT_ADDR, VAULT_TOKEN, VAULT_ROLE_ID and VAULT_SECRET_ID without any equivalent Ansible variable. If you are still using Ansible 2.5 or below you have to set these variable. Make sure to set the new ones, too. Otherwise you’re bound to run into troubles upon upgrading Ansible.
What if we don’t use ansible-playbook or ansible-navigator on the command line though, but run our playbooks through Ansible Automation Platform? How can we let our Controller know about these environment variables?
Using the Controller
The AAP documentation shows that we can define credentials with the type HashiCorp Vault Secret Lookup. But beware! While this allows to retrieve for example an ssh key stored in a Hashicorp Vault in order to access a managed host and run a playbook, you cannot fetch secrets from within a playbook run in this way. Custom credential types, however, lets us inject authentication information from environment variables and Ansible variables into our Controller. So let us create a custom credentials type and use it to access our secrets in Hashicorp Vault!
Now for the tricky stuff! The VAULT_* variables in the hashi_vault lookup plugin were removed from Ansible versions above 2.5, so for a future-proof solution, we need to work out a different way. Alas, the new supported environment variables of the plugin are named as ANSIBLE_HASHI_VAULT_*, and interfere with the internal AAP/AWX namespace: AAP blocks external variables if they carry an ANSIBLE_* prefix. There’s an open debate whether this a good thing, but fortunately the folks behind the collection community.hashi_vault are very constructive and provided a workaround for the issue. This workaround not only sets the environment variables, but also similar extra vars that pass through AAP/AWX just fine.
Create our own custom credential
So let’s create our own custom credential type named Custom Hashi Vault Token or Custom Hashi Vault AppRole if you use approles:
We set the content of Input Field to:
--- fields: - id: vault_url type: string label: Vault URL - id: vault_token type: string label: Vault Token secret: true required: - vault_url - vault_token
Or in the case of approles:
--- fields: - id: vault_url type: string label: Vault URL - id: role_id type: string label: App Role ID - id: secret_id type: string label: App Role Secret ID secret: true required: - vault_url - role_id - secret_id"
The content of the field Injector Configuration has to look as follows. Note that with this content, we support the old style VAULT_* environments variables of the hashi_vault lookup plugin as well as the new style ansible_hashi_vault_* Ansible vars. This ensures, that we can use the credentials in the Controller no matter what version of the hashi_vault plugin we are using:
---
env:
VAULT_ADDR: '{{ vault_url }}'
VAULT_TOKEN: '{{ vault_token }}'
VAULT_AUTH_METHOD: token
extra_vars:
ansible_hashi_vault_url: '{{ vault_url }}'
ansible_hashi_vault_token: '{{ vault_token }}'
ansible_hashi_vault_auth_method: token
If you use the approle rather than the token method:
---
env:
VAULT_ADDR: '{{ vault_url }}'
VAULT_ROLE_ID: '{{ role_id }}'
VAULT_SECRET_ID: '{{ secret_id }}'
VAULT_AUTH_METHOD: approle
extra_vars:
ansible_hashi_vault_url: '{{ vault_url }}'
ansible_hashi_vault_role_id: '{{ role_id }}'
ansible_hashi_vault_secret_id: '{{ secret_id }}'
ansible_hashi_vault_auth_method: approle
The final step is to create our own Credential from the newly created credentials type Custom Hashi Vault AppRole (you would follow the same logic for one of the type Custom Hashi Vault Token):
In Type Details, we can now enter the values for our three variables defined in the Input Field of the custom credential. These values are passed to the environment variables VAULT_* and extra vars ansible_hashi_vault_* by the injector configuration. This means the hashi_vault plugin has all the information at its hands to access our Hashicorp Vault.
With everything set, we can now use tasks in our playbooks run from the Controller that directly access secrets in our HashiCorps Vault. Yeah!
- ansible.builtin.debug:
msg: "Confidential number of beers drunk at the last Puzzle party: \
{{ lookup('community.hashi_vault.hashi_vault', \
'secret=secret/puzzle/party/numbers_of:beers_crypted }}"
