Resolving Terraform Azure environment challenges for complex deployments through Azure CLI and PowerShell

To repeatedly deploy cloud-native applications on Azure through Infrastructure as Code (IaC) scripts, Terraform is an excellent choice. There are situations where Terraform results into execution problems due to nature of Azure Services operations and delayed resources management on Azure. In such scenarios, Terraform is integrated with Azure Command-line Interface (CLI) scripts wrapped inside PowerShell scripts.

Some real-world scenarios which you may come across while working in an Terraform Azure environment have been covered in this blog based on our experience gained while working on an e-Commerce application. In this blog, we intend to explain how combination 3 technologies (Terraform, Azure CLI and PowerShell) helped us address situations better. Feel free to read about a situation interests you or check them all out.

1. If Diagnostic settings are enabled and / or key vault soft delete is enabled, then resource already exist error occurred 
2. Environment gets deleted if we provision multiple environments from the same folder 
3. Unable to destroy created environment
4. Terraform script fails if provisioning environment is configured for IPv6
5. For the case of Private Link, DNS entries for Key Vault and MySQL are not removed 
6. VNET Peering settings are not removed with peered destination VNET 

1. If Diagnostic settings are enabled and / or key vault soft delete is enabled, then resource already exist error occurred

Follow these steps: 

1. Create environment using terraform scripts with the respective settings.
2. Delete environment using Azure Portal / Terraform command.
3. Re-create the environment using Terraform with the same settings
4. Observation: Terraform will throw error that resource already exist and need to be imported.

Diagnosed root cause

1. Diagnostic settings are not deleted along with the resource group deletion. Same kind of issue is referred on Stack Overflow: https://stackoverflow.com/questions/70876645/diagnostic-settings-master-already-exists-to-be-managed-via-terraform-this.
2. Refer the GitHub link for Key Vault specific issue: https://stackoverflow.com/questions/70876645/diagnostic-settings-master-already-exists-to-be-managed-via-terraform-this 
3. Microsoft has kept soft delete enabled for all new Key Vaults. This can’t be managed by Terraform. This is documented on Microsoft docs at: https://docs.microsoft.com/en-us/azure/key-vault/general/soft-delete-overview 

2. Environment gets deleted if we provision multiple environments from the same folder

Follow these steps: 

Try to create multiple environments from the same folder using different vars file. When you try a second time, the earlier environment gets deleted.

Diagnosed root cause:

Terraform maintains only one state file so when we execute the scripts for second time, it destroys the environment present in the existing state file.

The Solution: 

Maintain separate state files by passing file names in both terraform plan and terraform Azure apply commands.

3. Unable to destroy created environment

Follow these steps:  

Use the command terraform apply -destroy –auto-approvewhenever you’ve diagnostic settings enabled.

Diagnosed root cause: 

The plan file gets stale due to drift in the file as Diagnostic settings take some time to fork through.

The Solution: 

Delete diagnostic settings first and use Azure CLI for destroying Resource Group. Terraform destroy doesn’t work mostly in scenarios like this.

4. Terraform script fails if provisioning environment is configured for IPv6

Follow these steps: 

1. Use Internet Service Provider that has IPV6 enabled by default, e.g., Jio in India.
2. From the computer connected to this ISP, run command to fork environment.
3. Script will throw an error while provisioning MySQL server for opening IP Addresses in Firewall to execute SQL Commands from local computer.

The Solution: 

Use (https://ipv4.icanhazip.com) to get IPv4 address and provide the obtained IPV4 address to put entry in Firewall settings of Azure MySQL Database Firewall.

5. For the case of Private Link, DNS entries for Key Vault and MySQL are not removed

Follow these steps: 

1. Run script to create an environment with Private Link enabled.
2. Destroy the environment.
3. Run script to re-create environment with same configuration.
4. Observation: Terraform script will fail with reason that A Record in Private DNS Zone of MySQL and Key Vault already exists.

Diagnosed root cause:

DNS entries in Private Link of MySQL and Key Vault are not deleted by Terraform.

The Solution: 

Delete these entries using Azure CLI scripts invocation.

6. VNET Peering settings are not removed with peered destination VNET

Follow these steps: 

1. Run script to create an environment with Private Link enabled.
2. Destroy the environment.
3. Run script to re-create environment with same configuration.
4. Observation: Terraform script will fail with reason stating that VNET Peering already exists in the parent VNET / VNET with which current environment is being peered.

Diagnosed root cause:

VNET Address peering moves to Disconnected state when Environment is destroyed.

The Solution: 

Delete these entries using Azure CLI scripts invocation.