Migrating GCP resources to Terraform, from practice

The content here is under the Attribution 4.0 International (CC BY 4.0) license

Managing cloud infrastructure manually through the Google Cloud Console can work for small projects, but as your infrastructure grows, it becomes increasingly difficult to track changes, maintain consistency, and collaborate with team members. This is where Infrastructure as Code (IaC) comes in, and Terraform is one of the most powerful tools for managing cloud resources declaratively.

In this comprehensive guide, I’ll walk you through the process of migrating your existing Google Cloud Platform (GCP) resources to Terraform, based on real-world experiences and challenges. Whether you’re managing a single project or multiple GCP projects with different configurations, this guide will help you establish a solid foundation for infrastructure management.

Why Migrate to Terraform?

Before diving into the technical details, let’s understand why migrating to Terraform is worth the effort:

1. Version Control: Your infrastructure becomes code that can be versioned in Git, providing a complete history of changes.
2. Reproducibility: You can recreate your entire infrastructure from code, making disaster recovery and environment replication straightforward.
3. Collaboration: Teams can review infrastructure changes through pull requests, just like application code.
4. Documentation: Your Terraform configuration serves as living documentation of your infrastructure.
5. Consistency: Terraform ensures resources are configured consistently across environments.
6. Change Management: Terraform’s plan/apply workflow lets you preview changes before applying them.

Prerequisites

Before starting the migration, ensure you have:

• Terraform installed (version 1.0 or later recommended)
• Google Cloud SDK (gcloud) installed and configured
• Service account credentials with appropriate permissions
• Access to your GCP projects with Owner or Editor role
• Basic understanding of Terraform syntax and concepts

Project Structure: Setting Up for Success

A well-organized Terraform project structure is crucial for maintainability. Here’s the structure I recommend:

my-infra/
├── gcp/
│   ├── main.tf           # Provider configuration and module calls
│   ├── variables.tf      # Input variables
│   ├── terraform.tfvars  # Variable values (add to .gitignore)
│   └── project/          # Reusable project module
│       ├── main.tf
│       ├── variables.tf
│       └── output.tf
├── aws/                  # Other cloud providers
├── azure/
└── README.md

This structure allows you to:

• Separate different cloud providers
• Create reusable modules for common patterns
• Manage multiple projects with similar configurations

Step 1: Configure Terraform Providers

The first step is setting up your Terraform configuration with the appropriate GCP providers. For GCP, you’ll typically need both the google and google-beta providers.

Create your gcp/main.tf:

terraform {
  required_version = ">= 1.0"
  
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 5.0"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = "~> 5.0"
    }
  }
}

locals {
  project_id = "your-project-id"
  credentials_file = var.gcp_credentials_file
}

provider "google" {
  credentials = local.credentials_file
  project     = local.project_id
  region      = "us-central1"
}

provider "google-beta" {
  credentials = local.credentials_file
  project     = local.project_id
  region      = "us-central1"
}

Create gcp/variables.tf:

variable "gcp_credentials_file" {
  description = "Path to GCP service account credentials JSON file"
  type        = string
  sensitive   = true
}

Important Note: The google-beta provider is essential for accessing newer GCP features that haven’t been promoted to general availability yet, such as Firebase project configurations.

Step 2: Managing Multiple GCP Projects with Provider Aliases

One of the most powerful features of Terraform is the ability to manage multiple GCP projects from a single configuration using provider aliases. This is particularly useful if you’re managing separate projects for different applications or environments.

Here’s how to configure multiple providers:

locals {
  testable_project_id = "testable-621bf"
  testable_credentials = var.testable_gcp_credentials_file
  
  production_project_id = "production-app"
  production_credentials = var.production_gcp_credentials_file
}

# Provider for testable project
provider "google" {
  alias       = "testable"
  credentials = local.testable_credentials
  project     = local.testable_project_id
  region      = "us-central1"
}

provider "google-beta" {
  alias       = "testable"
  credentials = local.testable_credentials
  project     = local.testable_project_id
  region      = "us-central1"
}

# Provider for production project
provider "google" {
  alias       = "production"
  credentials = local.production_credentials
  project     = local.production_project_id
  region      = "us-west1"
}

provider "google-beta" {
  alias       = "production"
  credentials = local.production_credentials
  project     = local.production_project_id
  region      = "us-west1"
}

Step 3: Creating Reusable Modules

Modules are the key to keeping your Terraform code DRY (Don’t Repeat Yourself). Let’s create a module for managing GCP projects.

Create gcp/project/main.tf:

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
    }
  }
}

resource "google_project" "default" {
  project_id      = var.project_id
  name            = var.project_name
  deletion_policy = "PREVENT"

  labels = {
    "managed-by" = "terraform"
    "firebase"   = "enabled"
  }
}

resource "google_project_service" "services" {
  for_each = toset(var.enabled_services)
  
  project = google_project.default.project_id
  service = each.value

  disable_on_destroy = false
}

Create gcp/project/variables.tf:

variable "project_id" {
  description = "The project ID to create"
  type        = string
}

variable "project_name" {
  description = "The display name of the project"
  type        = string
}

variable "credentials_file" {
  description = "Path to service account credentials"
  type        = string
  sensitive   = true
}

variable "enabled_services" {
  description = "List of GCP services to enable"
  type        = list(string)
  default     = []
}

Create gcp/project/output.tf:

output "project_id" {
  description = "The project ID"
  value       = google_project.default.project_id
}

output "project_number" {
  description = "The project number"
  value       = google_project.default.number
}

Critical Detail: Notice the terraform block with required_providers in the module. This is essential when passing providers via aliases from the parent module. Without this declaration, Terraform will issue warnings about undefined providers.

Step 4: Using Modules with Provider Aliases

Now you can use your module multiple times with different provider configurations:

module "testable" {
  source = "./project"
  
  project_id       = local.testable_project_id
  credentials_file = local.testable_credentials
  project_name     = "Testable Project"
  
  enabled_services = [
    "firebase.googleapis.com",
    "cloudfunctions.googleapis.com",
    "cloudrun.googleapis.com"
  ]
  
  providers = {
    google      = google.testable
    google-beta = google-beta.testable
  }
}

module "production" {
  source = "./project"
  
  project_id       = local.production_project_id
  credentials_file = local.production_credentials
  project_name     = "Production App"
  
  enabled_services = [
    "compute.googleapis.com",
    "storage.googleapis.com",
    "sql-component.googleapis.com"
  ]
  
  providers = {
    google      = google.production
    google-beta = google-beta.production
  }
}

Step 5: Importing Existing Resources

The most critical part of migration is importing your existing GCP resources into Terraform state. This process tells Terraform that certain resources already exist and shouldn’t be created.

Finding Resource IDs

First, you need to identify the resources you want to import. Use the gcloud CLI:

# List all projects
gcloud projects list

# List compute instances
gcloud compute instances list --project=your-project-id

# List storage buckets
gcloud storage buckets list --project=your-project-id

# List cloud functions
gcloud functions list --project=your-project-id

Importing Resources

Once you’ve written the Terraform configuration for a resource, import it:

# Import a project
terraform import 'module.testable.google_project.default' testable-621bf

# Import a compute instance
terraform import 'google_compute_instance.web_server' projects/your-project/zones/us-central1-a/instances/web-1

# Import a storage bucket
terraform import 'google_storage_bucket.data' your-bucket-name

# Import a Cloud Function
terraform import 'google_cloudfunctions_function.api' projects/your-project/locations/us-central1/functions/my-function

The Import Workflow

1. Document the resource: List all resources in your GCP project
2. Write Terraform config: Create the resource block in your .tf files
3. Import the resource: Use terraform import to add it to state
4. Run terraform plan: Verify that no changes are detected
5. Adjust configuration: If changes are detected, update your config to match the actual resource

Step 6: Handling Free Tier Resources

When migrating resources that use GCP’s free tier, ensure your Terraform configuration reflects this:

# Example: Cloud Storage with free tier settings
resource "google_storage_bucket" "free_tier_bucket" {
  name     = "my-free-tier-bucket"
  location = "US"
  
  # Standard storage class (free tier eligible)
  storage_class = "STANDARD"
  
  # Public access prevention
  public_access_prevention = "enforced"
  
  # Versioning disabled to minimize storage
  versioning {
    enabled = false
  }
}

# Example: Compute Engine f1-micro instance (free tier)
resource "google_compute_instance" "free_tier_vm" {
  name         = "free-tier-vm"
  machine_type = "f1-micro"
  zone         = "us-west1-a"  # Free tier eligible zone
  
  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
      size  = 30  # 30GB standard persistent disk (free tier)
    }
  }
  
  network_interface {
    network = "default"
    access_config {
      # Ephemeral IP
    }
  }
}

Common Challenges and Solutions

Challenge 1: Provider Type Mismatch

Error: “The local name ‘testable-google-beta’ in the root module represents provider ‘hashicorp/testable-google-beta’, but ‘google-beta’ in module represents ‘hashicorp/google-beta’.”

Solution: Ensure the provider type is correct, not the alias. The provider block should be:

provider "google-beta" {
  alias = "testable"
  # ...
}

Not:

provider "testable-google-beta" {
  # Wrong!
}

Challenge 2: Undefined Provider in Module

Error: “Reference to undefined provider”

Solution: Add a required_providers block to your child module:

terraform {
  required_providers {
    google = {
      source = "hashicorp/google"
    }
    google-beta = {
      source = "hashicorp/google-beta"
    }
  }
}

Challenge 3: Variable Reference Typo

Error: “Reference to undeclared resource: vars.number_of_resources”

Solution: Use var. (singular) not vars.:

# Correct
count = var.number_of_resources

# Wrong
count = vars.number_of_resources

Best Practices for GCP Terraform Management

1. Use Remote State

Store your Terraform state in Google Cloud Storage for team collaboration:

terraform {
  backend "gcs" {
    bucket = "my-terraform-state"
    prefix = "gcp/production"
  }
}

2. Separate Environments

Use workspaces or separate state files for different environments:

gcp/
├── dev/
│   ├── main.tf
│   └── terraform.tfstate
├── staging/
│   ├── main.tf
│   └── terraform.tfstate
└── production/
    ├── main.tf
    └── terraform.tfstate

3. Use Variables Files

Keep sensitive data and environment-specific values in .tfvars files:

# terraform.tfvars (add to .gitignore)
testable_gcp_credentials_file = "/path/to/credentials.json"
production_gcp_credentials_file = "/path/to/prod-credentials.json"

4. Enable Resource Deletion Protection

For critical resources, use deletion protection:

resource "google_project" "default" {
  project_id      = var.project_id
  name            = var.project_name
  deletion_policy = "PREVENT"  # Prevents accidental deletion
}

resource "google_sql_database_instance" "main" {
  name = "main-instance"
  
  settings {
    tier = "db-f1-micro"
    
    deletion_protection_enabled = true
  }
}

5. Use Count and For_Each Wisely

When you need multiple similar resources, use for_each for better state management:

resource "google_project_service" "services" {
  for_each = toset([
    "compute.googleapis.com",
    "storage.googleapis.com",
    "cloudrun.googleapis.com"
  ])
  
  project = var.project_id
  service = each.value
}

6. Document Your Modules

Add descriptions to all variables and outputs:

variable "project_id" {
  description = "The GCP project ID where resources will be created"
  type        = string
  
  validation {
    condition     = can(regex("^[a-z][a-z0-9-]{4,28}[a-z0-9]$", var.project_id))
    error_message = "Project ID must be between 6 and 30 characters."
  }
}

Testing Your Migration

Before applying changes to production:

1. Run terraform plan: Review all planned changes carefully
2. Test in a dev environment: Apply changes to a non-critical environment first
3. Verify resources: After applying, check the GCP Console to ensure resources match expectations
4. Monitor for drift: Use terraform plan regularly to detect manual changes

# Review changes without applying
terraform plan

# Apply with auto-approval (use cautiously)
terraform apply -auto-approve

# Show current state
terraform show

# List all resources in state
terraform state list

Automation and CI/CD Integration

Once your migration is complete, integrate Terraform into your CI/CD pipeline:

# GitHub Actions example
name: Terraform GCP
on:
  push:
    branches: [main]
    paths:
      - 'gcp/**'

jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        
      - name: Terraform Init
        run: terraform init
        working-directory: ./gcp
        
      - name: Terraform Plan
        run: terraform plan
        working-directory: ./gcp
        env:
          GOOGLE_CREDENTIALS: ${{ secrets.GCP_CREDENTIALS }}
          
      - name: Terraform Apply
        if: github.ref == 'refs/heads/main'
        run: terraform apply -auto-approve
        working-directory: ./gcp
        env:
          GOOGLE_CREDENTIALS: ${{ secrets.GCP_CREDENTIALS }}

Conclusion

Migrating to Terraform is an investment that pays dividends in improved infrastructure management, collaboration, and reliability. While the initial migration requires effort, the long-term benefits of having your infrastructure as code are immense.

Key takeaways:

• Start with a clear project structure
• Use modules for reusability
• Leverage provider aliases for multi-project management
• Import existing resources carefully
• Test thoroughly before production deployment
• Integrate with CI/CD for automation

Remember, migration doesn’t have to happen all at once. You can adopt an incremental approach, migrating one project or service at a time. This reduces risk and allows your team to build expertise gradually.

The journey from manual GCP management to Infrastructure as Code with Terraform transforms how you think about and manage your cloud resources. You’ll find that infrastructure changes become more predictable, reviewable, and auditable—ultimately leading to more reliable and maintainable systems.

Additional Resources

• Terraform GCP Provider Documentation
• Google Cloud Import Guide
• Terraform Module Registry

• GCP Free Tier Details

• https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/firebase_project - firebase project resource documentation

• https://github.com/hashicorp/terraform-getting-started-gcp-cloud-shell/blob/master/tutorial/cloudshell_tutorial.md

Happy terraforming! 🚀