Our team is thinking about deploying Vault within our Aptible environment. We were curious if anyone had tried this and is willing to share their experience in setting this up.
Thanks,
Khiem
1 Like
I thought about using Vault in Aptible too but there are a few limitations you have to be aware of if you’re going to use it for “on-demand” or time-limited credentials for various resources such as databases. You will not have command-line access to the containers running Aptible’s component databases (Postgres, Redis, Elastic Search, etc). So you’ll have to rely on the credentials and connection string that Aptible provides when you launch your database container in order to connect and create users for various needs through the db driver, API or CLI to the target database. For instance, in postgres, you won’t be able to modify the port or directly access the pg_hba or postgresql.conf files. Aptible’s support team may be able to provide additional feedback.
If you’re trying to access a database from outside your Aptible environment, you probably don’t want to enable external access directly. Instead, if you try the tunnel through Aptible’s CLI, then you’ll notice that they create a unique URI including a distinct password and port every time you set up a new tunnel. So in this respect, Aptible provides unique, time-sensitive credentials to certain resources for you or your devops team.
Within the production environment itself, my implementation does not expect to generate dynamic credentials for resources that are in the same VPC as our web, API, and worker containers.
I think where Vault shines is in controlling access by real users (admin, qa, developers) to sensitive resources to avoid credentials getting exposed to unauthorized personnel or to avoid having active, long TTL credentials available to former employees or hackers. Hope that helps.
First of all im sorry that @khiem i have not answerred this.
Yep. We saw your post about week ago, but we was still in “implementing phase”, but now it’s Done. It was not the easiest task. There are multiple problems you are going to encounter. I have no idea why this one guy ((@normkatz1) is talking about “database tunneling”. I’m pretty sure he has no idea what he is saying. Even if you got “live” access to your database - everything is encrypted. That’s the idea of Vault. So I see no point for that. Vault is used by API.
So let’s say that you want to deploy Hashicorp Vault to Aptible. That can be done. There is nothing special with that.
Challenge start when you want to make your Vault to work in HA-mode. Then you need to check what DB’s are supported by Aptible. Pretty soon you understand that there is only one real player and that is MySQL. If you want your Vault Storage-Backend to work in HA-mode you need to contact to Aptible. Even before that you are going to find “bug” what states, that Vault could not connect to MySQL because of TLS1.2 (that one goes to Vault not Aptible).
Let’s say that you are able to to get over that. Then you are thinking - Where could we store our data. To AWS you say. ( I won’t tell what DB you should use - easy to find ). Then you just configure your Vault and “up-and-running” it is. At this point you have one Vault-server and Storage out-side of Aptible. Working as planned.
The point of Vault is that it should be there all the time, so you need to think about the HA-mode. Now the problems starts. You need/should scale them to at least to two, so they are provisioned to physically different servers and maybe even continentals.
“When you scale horizontally, Enclave automatically deploys your Containers in a high-availability configuration, with Containers deployed in separate AWS Availability Zones.“ (https://www.aptible.com/documentation/enclave/reference/apps/scaling.html)
Because Vault-containers are after Aptible LB(nginx) and Vault needs to be in Master-Slave-mode you need to somehow force the aptible Load-Balancer to talk with your servers so that only one give them 200 (OK) as a master and rest of those give 429 (Slave) The other ones are going to be seen as “not working”. Only way to get Aptible LB to work as “normal load-balancer” is to enable STRICT_HEALTH_CHECKS. So you need to think about that on the way. Let me tell you little bit more about this. This is something you need. There are of course 307 (redirect) and CLUSTER_ADDR, but you don’t want to rely on those. So you need STRICT_HEALTH_CHECKS. Make some magic to get Aptible LB think that they are Ok., but then they are not… 
After that it will redirect traffic correctly.
The code says https://github.com/aptible/docker-nginx/blob/master/templates/etc/nginx/sites-enabled/server.conf.erb#L97 so the /heathcheck is “hardcoded” inside LB. We would like to make pull-request on that one, but Aptible are not interested 
But yeah. After you get over this little stuff you will get fully working HA-enabled Vault cluster inside Aptible. We are probably willing to help you if you ask nicely
vault status
Key Value
Seal Type shamir
Initialized true
Sealed false
Total Shares 2
Threshold 2
Version 0.11.5
Cluster Name vault-cluster-kshkssss
Cluster ID kahfoofofoffofofofofo
HA Enabled true
HA Cluster somethink-somethink-73633.aptible.in:8201
HA Mode active
Hi Khiem,
What is your specific use case and requirements for Vault?
Some Vault secrets engines simply store and read data - like encrypted Redis/Memcached - which could be use in lieu of the aptible config:set to provide variables/secrets to your Enclave Apps. Other secrets engines connect to other services and generate dynamic credentials on demand, which is what I believe @normkatz1 was referring to, and we agree is not terribly useful for Enclave databases given that Enclave already provides role-based access controls. Other secrets engines provide encryption as a service, time-based one-time password generation, certificate management, and more.
As for @numberseven’s advice, it’s true that Vault has a very specifically architected High Availability mode, which makes it almost more akin to a master/slave database than a highly available web app. As such Vault’s HA is not a “natural” fit for Enclave, but I believe it can work with some out-of-the-box thinking. Specifically, two Vault containers in HA mode expect to be able to communicate with each other directly, and have separate addresses - this is not how HA Apps (scaled to >1 container) work on Enclave. To work around this, you would need to deploy Vault as two separate Services or Apps on Enclave. Each service would need to be configured with a TLS Endpoint. This which would alleviate the issue that @numberseven worked around by forcing one container to “fail” runtime health checks, by giving them each a unique external hostname, rather than load balancing them behind a single Endpoint.
Again, depending on your use case, this type of HA configuration may not be a requirement to you how you plan on using Vault hosted in Enclave. We’re happy to help with whatever your specific use case may be!
– Alex Kubacki
Lead Support Engineer, Aptible