As a coder or DevOps, you are probably familiar with the Zerotrust principles for securing your applications. This approach provides a rather demanding security framework for all users and machines, requiring them to be continuously authenticated/authorized and validated in order to access applications and data.
Here we go completely out of the traditional network framework to dive into a network with fuzzy edges, made of cloud, hybrid and traditional networks.
To meet this need for the management of secrets necessary for the operation of ZeroTrust procedures, Hashicorp has developed Vault. It is a secret management system responsible for protecting sensitive information.
What I call “Secret” here is something that you don’t want the public to have access to, but still need to be able to be accessed operationally through code or infrastructure. The most common secrets are database identifiers, encryption keys, API keys, etc.
Vault, which exists in free and open source versions, can be used both to store credentials, grant access to specific data or even authenticate users accessing their respective secrets.
Vault consists of a backend used for storing secrets, an API server used to process client requests and perform the various secret operations, and different “secret engines” for each type of secret supported.
This eliminates the need for your apps to store secrets, just query Vault or pass data to it. To summarize, Vault works with tokens and each token is linked to a “policy”, that is to say a set of rules limiting the actions and access of each client. Thus, the secrets stay warm in a database and all that is communicated to third-party applications are authorizations in the form of tokens.
Regarding security, Vault encrypts all data with a key before placing the secrets in its vault. This key is encrypted by the master key which is requested at startup to unlock Vault. Vault itself does not keep this master key and it is not present on the server (unless you accidentally copied it over).
Vault is undoubtedly the benchmark in this area and it would be a shame to deprive yourself of it. I am counting on you to put it in place in order to strengthen the security of their data and their access.
I therefore offer you a short tutorial to get you started and deploy it quickly on a server, using Docker and Docker-Compose, which I particularly like.
Vault + Consul on Docker
Using Vault to manage your secrets is obviously a much more secure solution than leaving them unencrypted in all of your config files.
I therefore strongly recommend that you adopt this solution, whether you are DevOps or a developer.
You can install Vault on a Windows or macOS computer, but for this tutorial, it will be on Linux. I’m going to assume you’re using a Debian or derivative distribution (Ubuntu…etc.) and we’re going to use Docker and Docker-Compose to do a clean and quick install of Vault.
I’m going to assume that your server has Docker-Compose, but if it doesn’t, please install it like this.
Update your server:
sudo apt update
sudo apt upgrade
Then, we will install Docker compose like this:
sudo apt-get install docker-compose
The next step is to create a docker-compose.yml file that will contain your Vault config. We can obviously do it by hand, but there is a repository on Github that includes everything you need to deploy Vault + Consul.
Consul is another Hashicorp tool. It is a distributed application management solution allowing to manage their configuration, their availability, but also to monitor them…etc. But above all, integrated with Vault, Consul can also store your secrets.
It’s more flexible than storing everything in a directory on the main server and it will allow you to operate in high availability on a distributed system.
To initialize the project, we are therefore going to do a
git clone https://github.com/testdrivenio/vault-consul-docker
The docker-compose.yml file is broken down into 3 services:
- Vault itself
- The Consul Server
- A Consul worker (a client what…)
version: '3.8' services: vault: build: context: ./vault dockerfile: Dockerfile ports: - 8200:8200 volumes: - ./vault/config:/vault/config - ./vault/policies:/vault/policies - ./vault/data:/vault/data - ./vault/logs:/vault/logs environment: - VAULT_ADDR=http://127.0.0.1:8200 - VAULT_API_ADDR=http://127.0.0.1:8200 command: server -config=/vault/config/vault-config.json cap_add: - IPC_LOCK depends_on: - consul consul: build: context: ./consul dockerfile: Dockerfile ports: - 8500:8500 command: agent -server -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect 1 -config-file=/consul/config/config.json volumes: - ./consul/config/consul-config.json:/consul/config/config.json - ./consul/data:/consul/data consul-worker: build: context: ./consul dockerfile: Dockerfile command: agent -server -join consul -config-file=/consul/config/config.json volumes: - ./consul/config/consul-config.json:/consul/config/config.json depends_on: - consul
Thanks to the IPC_LOCK parameter, the docker container will try to lock the memory to prevent your secrets from being swapped to the hard disk. It is therefore important to leave it.
This file is of course accompanied by the config files necessary for Consul and Vault, but also by various Dockerfiles necessary for deployment. I strongly urge you to look at the structure of these files to understand how they work.
Then do a:
Then start the docker like this:
docker-compose up -d --build
You can then access the services via the following URLs:
Vault can be initialized via the graphical interface http://IP:8200 or directly from the command line. I’m going to choose the command line option for more fun.
We will therefore launch a command line interface on the docker vault machine:
docker-compose exec vault bash
At the moment, Vault is not initialized. During initialization, a whole procedure for creating secure keys will be triggered. These keys are the “master” keys of your Vault. Better not to lose them. The master key is split into several keys that you can keep in different places or distribute among different people. This prevents a single person from being able to access the entire trunk.
Obviously, if you are alone at the controls, you will have all the keys. We will therefore enter the following command:
vault operator init
5 keys will be generated. Remember to keep them. As well as the Root Token which will be required to log in.
This is a step you can also perform through the GUI:
Well done, you have made good progress! Think well when you fill in your talent.io profile to stand out by highlighting your strengths and your differentiating elements so that recruiting companies stop on your profile. Why not talk about your skills in securing projects?
Then to unlock Vault you will need 3 keys out of the 5 keys (this is what we decided during initialization).
Via the UI it will be easy. Via the command line, you will have to enter the following command 3 times in a row:
vault operator unseal
Each time, Vault will ask us for a piece of the key. Then with the login command, we will identify ourselves using the root token:
Congratulations, your instance of Vault is unlocked and you are logged in.
Next, we will enable the logs:
vault audit enable file file_path=/vault/logs/audit.log
To see if everything is OK and that the log storage point has been taken into account, enter the command:
vault audit list
Creation of a first chest
In vault, you can create as many vaults as you want. We will create a first one whose very original name will be “secret” using the following command:
vault secrets enable -path=secret/ kv
In this chest, you will be able to store static secrets that do not expire unless you revoke them. It’s presented in a traditional key/value format. Or thanks to Consul dynamic secrets that are generated on demand and expire after a certain time.
To create a first key/value pair in our secret vault, you can do it through the GUI or like this in command line.
It is also possible via the Vault API if you wish, but remember to use a “root policy” on your vault to set read-only or read-write rights to applications that will call on this API. I invite you to refer to the documentation on this subject.
Add a static secret
vault kv put secret/wordpress korben=MonMotDePasse
And to read it:
vault kv get secret/wordpress
Note that you can enable version tracking on your vault like this:
vault kv enable-versioning secret/
You can save several values like this:
And retrieve the value you are interested in using the -version parameter:
vault kv get -version=1 secret/wordpress vault kv get -version=2 secret/wordpress
To delete one of these values:
vault kv delete -versions=1 secret/wordpress
And to restore the value (use versioning!):
vault kv undelete -versions=1 secret/wordpress
Afterwards, if you really want to delete it without the possibility of restoring it, you will have to use the destroy parameter:
vault kv destroy -versions=1 secret/wordpress
Add a dynamic secret
As I told you, dynamic secrets are generated on demand and can be limited by a role, but also by a lifetime.
For AWS. You will have to log in to your AWS account at Amazon, and go to the access management tool (IAM) to add a new user and thus obtain an access key and a secret key.
I also invite you to follow this Hashicorp procedure which will explain how to create a policy for your user and thus generate dynamic secrets linked to this user on demand.
Encryption with Vault
Vault also makes it possible to encrypt and decrypt data on the fly without having to store it. To do this, we will activate the “transit” service like this:
vault secrets enable transit
Then create an encryption key for each of your applications
vault write -f transit/keys/mykey
We can see it appear on the interface side:
Then to encrypt content you can call the following command, even if the most frequent will be to go through the Vault API to do exactly the same thing:
vault write transit/encrypt/mykey plaintext=$(base64 <<< "Ceci est un secret chuuut.")
And to decode it and retrieve the data in base64:
vault write transit/decrypt/mykey ciphertext=vault:v1:tN17ipGea3rv2tXzfHfNpvQbCkDSky5l/iFHN+I2/DJNSNqUh44w/1100vq4OqTXwxH3PZsltA==
Then all that remains is to decode the base64 like this:
base64 -d <<< "Q2VjaSBlc3QgdW4gc2VjcmV0IGNodXV1dC4K"
As I told you at the beginning, Vault is a tool that can be used on the command line, through a graphical interface, but especially via its API . It is this element that is used the most to link applications that need secrets with Vault.