Hashicorp Vault has different Seal types, and a day we need to migrate between two seals.<\/p>\n\n\n\n
This is exactly what happened, and how I did it between Oracle Cloud KMS (ocikms) seal and Shamir Seal.<\/p>\n\n\n\n
I’ve my lab deployed on Oracle Cloud and I needed to deploy 1.12.0\/1.12.1 to get a bug fix deployed.<\/p>\n\n\n\n
You deployed it and Vault didn’t restart\u2026 \ud83d\ude31 So I tried to run it manually from the CLI and got the real error:<\/p>\n\n\n\n
\/usr\/bin\/vault server -config=\/etc\/vault.d\/vault.hcl\nError parsing Seal configuration: 'key_id' not found for OCI KMS seal configuration\n2022-10-13T04:07:07.570Z [INFO] proxy environment: http_proxy=\"\" https_proxy=\"\" no_proxy=\"\"<\/code><\/pre>\n\n\n\nSo no need to modify the configuration, Vault is broken with the OCI KMS. I reported it using GitHub: seal OCI KMS doesn’t find key_id<\/a> and someone already reported an additional error with OCI KMS: Oracle KMS seal: “did not find a proper configuration for private key”<\/a><\/p>\n\n\n\nBut investing a little more, it’s not the only KMS broken:<\/p>\n\n\n\n
\n- GCP KMS Autounseal Error bug<\/a><\/li>\n\n\n\n
- Can’t init 1.12.0 with awskms<\/a><\/li>\n<\/ul>\n\n\n\n
So Vault is broken for any deployment on AWS, GCP, and OCI (Oracle Cloud) if you plan to use their KMS.<\/p>\n\n\n\n
Migration<\/h2>\n\n\n\n
It needs to be done with a working version of Vault with the KMS so I rollbacked it to 1.11.4.<\/p>\n\n\n\n
On each node where Vault is running (hopefully, it wasn’t in a container), I added the little change in vault.hcl<\/code><\/p>\n\n\n\nOn each node, I add disabled = \"true\"<\/code> to the seal block in \/etc\/vault.d\/vault.hcl<\/code>:<\/p>\n\n\n\nseal \"ocikms\" {\n crypto_endpoint = \"https:\/\/xxxxxxxx-crypto.kms.sa-saopaulo-1.oraclecloud.com\"\n management_endpoint = \"https:\/\/xxxxxxxx-management.kms.sa-saopaulo-1.oraclecloud.com\"\n key_id = \"ocid1.key.oc1.sa-saopaulo-1.xxxxxxxx.yyyyyyyyyyyyyzzzzzzzzzzzzzz\"\n disabled = \"true\"\n}<\/code><\/pre>\n\n\n\nDon’t reboot at this time, the quorum is needed to complete this step. In addition, I check environment variables are ready to execute vault unseal<\/code> commands.<\/p>\n\n\n\nIdentify all standby nodes, the leader will be the last to be modified.<\/p>\n\n\n\n
Now, I restart Vault on one node\u2026 Yes, it’s one node at a time so in my case, 5 nodes so it’s very time-consuming to execute this migration.<\/p>\n\n\n\n
When Vault restarted, its status is sealed<\/em> so let’s go with vault unseal -migrate<\/code> for every key. When this node is unsealed, we do the same on the next standby, again and again.<\/p>\n\n\n\nNow, it’s time to do it on the active node, and good on this point.<\/p>\n\n\n\n
When every is done, don’t forget to comment\/remove the seal block in \/etc\/vault.d\/vault.hcl<\/code>.<\/p>\n\n\n\nTake care of a strange error<\/h3>\n\n\n\n
Something that I did was trying to add a 6th node and I saw the error: aead is not configured in the seal<\/code><\/p>\n\n\n\nTo resolve it, I needed to rotate the underlying encryption key:<\/p>\n\n\n\n
vault operator rotate<\/code><\/pre>\n\n\n\nIt’s done without downtime.<\/p>\n\n\n\n
Finally, I preferred generating a new set of unseal keys because they were generated when I installed Vault (a long time ago):<\/p>\n\n\n\n
vault operator rekey -init -key-shares=5 -key-threshold=3<\/code><\/pre>\n\n\n\nAnd rekey each unseal key with<\/p>\n\n\n\n
vault operator rekey<\/code><\/pre>\n\n\n\nConclusion<\/h2>\n\n\n\n
When you’re managing some Hashicorp Vault, you need to know those tasks and can execute them without creating a high downtime.<\/p>\n\n\n\n
This issue is still existing, so you have to choose between:<\/p>\n\n\n\n
\n- keep with 1.11.4<\/li>\n\n\n\n
- migrate and update to the latest version<\/li>\n<\/ul>\n\n\n\n
I don’t know what is happening at Hashicorp but they’re breaking the entire deployment due to bad dependency management or another bug introduced. In any case, we won’t sleep well due to this situation.<\/p>\n","protected":false},"excerpt":{"rendered":"
Hashicorp Vault has different Seal types, and a day we need to migrate between two seals. This is exactly what happened, and how I did it between Oracle Cloud KMS (ocikms) seal and Shamir Seal. Why am I doing this migration? Hashicorp broke Vault with a lot of Cloud KMS. I’ve my lab deployed on […]<\/p>\n","protected":false},"author":1,"featured_media":108,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"saved_in_kubio":false,"footnotes":""},"categories":[28],"tags":[19],"_links":{"self":[{"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/posts\/267"}],"collection":[{"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/comments?post=267"}],"version-history":[{"count":1,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/posts\/267\/revisions"}],"predecessor-version":[{"id":268,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/posts\/267\/revisions\/268"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/media\/108"}],"wp:attachment":[{"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/media?parent=267"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/categories?post=267"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/10.42.0.68:8080\/wp-json\/wp\/v2\/tags?post=267"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}