Reading time ~3 minutes
Migrating Terraform state from Terraform Cloud to S3
Yesterday, HashiCorp announced a new “free” tier for Terraform Cloud.
Today, HashiCorp #Terraform Cloud’s Free tier is adding new features including:— HashiCorp (@HashiCorp) May 16, 2023
🛤️ Sentinel & OPA
✅ Run Tasks
☁️ Cloud agents
We’re also making several changes to paid offerings and simplifying billing metrics. https://t.co/teum1G9oKl
Although this seems like a good improvement (free SSO and OPA!), in the FAQ you can see that it introduces a limit on the number of resources (
500) that you can manage.
Once you are over that limit, you get charged per resource.
This is less than ideal for current users with non-commercial projects in it.
Now, I (used to) use Terraform Cloud exclusively for storing multiple state files (as Terraform is run in CI/CD pipelines), roughly one per AWS account I manage.
On average, each of these workspaces contains more than
Failed Automated Migration
The first thing I did when I heard of this was try to migrate a test project to using S3 as remote storage.
However, when I tried to migrate to any other backend, I got the following error (the page linked from the error message is far from helpful in providing an actionable solution).
I was trying to avoid a manual migration as it could be error-prone, but also, looking in the HashiCorp community forum, it seemed that the only way was to pull the state down manually and re-initialise it in another backend (see this discussion for example).
In the meantime, the HashiCorp support team replied with a similar answer:
So, let’s migrate.
Create resources needed to manage the state in S3
You basically want a private S3 bucket with versioning enabled + a DynamoDB table to lock the state file to prevent concurrent modifications and state corruption.
Although I ended up creating a custom module, if you are looking for something quick, this module from
cloudposse is a great starting point:
Import the module from your existing TF configuration and run
terraform apply (still using your old state in Terraform Cloud).
Manually pull down the state from Terraform Cloud
Next, we need to download the current state from Terraform Cloud:
Once you have downloaded it, be sure to remove the existing local
.terraform directory, as it will contain another
terraform.tfstate still pointing to Terraform Cloud:
We can then finally swap backend.
We are going to change the backend from
You’ll have to set the value of
as you created earlier with the
terraform init with the new configuration to initialise the new backend.
In the process,
terraform will ask you if you want to migrate the existing state.
yes, and it will automatically port all your resources in the new state stored in S3.
As a last step, remember to remove the old
.terraform.old folder and the
state you pulled from Terraform Cloud.
I hope you found this post valuable and interesting, and I’m keen to get feedback on it! If you find the information shared helpful, if something is missing, or if you have ideas on improving it, please let me know on 🐣 Twitter or at 📢 feedback.marcolancini.it.
Thank you! 🙇♂️