☁️ Azure Hdinsight Spark Cluster Terraform Module

Provisions a single Azure HDInsight Spark cluster (azurerm_hdinsight_spark_cluster) with a hardened, secure-by-default posture. Targets hashicorp/azurerm ~> 4.0.

🧩 Overview

🔥 Deploys an Apache Spark cluster on Azure HDInsight as one keystone resource named this.
🧱 Models the full node topology — head, worker, and zookeeper roles — with deeply-typed inputs.
🔐 Ships hardened: TLS 1.2 floor and in-transit encryption on out of the box.
🗝️ Prefers SSH public keys over node passwords and a managed-identity ADLS Gen2 backing store over storage keys.
🌐 Supports VNet injection, Private Link, compute isolation, and customer-managed-key disk encryption.
🏢 Optional Enterprise Security Package (Azure AD Domain Services + LDAPS + Ranger) for domain-joined, audited clusters.
📈 Optional Log Analytics monitoring via the monitor and extension blocks.
🧮 Autoscale workers by load or by schedule; scale worker count in place without replacing the cluster.

💡 Why it matters: An HDInsight cluster is largely immutable — most fields force a full replacement. Getting the secure posture, storage model, and network placement right the first time avoids a destructive rebuild of a stateful analytics platform.

❤️ Support this project

If this module saves you time, please consider supporting its continued development:

⭐ Star the repository on GitHub.
🤝 Connect on LinkedIn: linkedin.com/in/microsoftexpert
☕ Buy me a coffee: buymeacoffee.com/microsoftexpert

🗺️ Where this fits in the family

flowchart LR
  rg["terraform-azurerm-resource-group"]
  vnet["terraform-azurerm-virtual-network subnet"]
  uai["terraform-azurerm-user-assigned-identity"]
  sa["terraform-azurerm-storage-account ADLS Gen2"]
  law["terraform-azurerm-log-analytics-workspace"]
  this["terraform-azurerm-hdinsight-spark-cluster"]
  cluster["azurerm_hdinsight_spark_cluster"]

  rg -->|"resource_group_name / location"| this
  vnet -->|"subnet_id (VNet injection)"| this
  uai -->|"managed_identity_resource_id"| this
  sa -->|"filesystem_id + storage_resource_id"| this
  law -->|"workspace id + primary_key (monitor)"| this
  this -->|"creates"| cluster

  classDef me fill:#0078D4,stroke:#004578,color:#fff;
  classDef keystone fill:#004578,stroke:#001f3f,color:#fff;
  classDef sib fill:#eef2f7,stroke:#b8c4d0,color:#1b1b1b;
  class this me;
  class cluster keystone;
  class rg,vnet,uai,sa,law sib;

🧬 What this module builds

flowchart TB
  gw["gateway (username / password)"]
  cv["component_version.spark"]
  this["terraform-azurerm-hdinsight-spark-cluster"]
  cluster["azurerm_hdinsight_spark_cluster.this"]
  subgraph vnetsub["VNet subnet (optional injection)"]
    head["roles.head_node"]
    worker["roles.worker_node (+ autoscale)"]
    zk["roles.zookeeper_node"]
  end
  adls["ADLS Gen2 default storage"]
  uai["user-assigned identity (MSI)"]
  esp["security_profile (ESP / AADDS + LDAPS)"]
  law["Log Analytics (monitor / extension)"]

  gw -->|"gateway credentials"| this
  cv -->|"spark version"| this
  this -->|"creates"| cluster
  cluster -->|"head node"| head
  cluster -->|"worker nodes"| worker
  cluster -->|"zookeeper node"| zk
  cluster -->|"storage_account_gen2 (is_default)"| adls
  uai -->|"managed_identity_resource_id"| adls
  cluster -->|"domain join"| esp
  cluster -->|"logs + metrics"| law

  classDef me fill:#0078D4,stroke:#004578,color:#fff;
  classDef keystone fill:#004578,stroke:#001f3f,color:#fff;
  classDef sib fill:#eef2f7,stroke:#b8c4d0,color:#1b1b1b;
  class this me;
  class cluster keystone;
  class head,worker,zk,adls,uai,esp,law,gw,cv sib;

Resource inventory

Resource	Count	Role
`azurerm_hdinsight_spark_cluster.this`	1	The keystone cluster, with its `component_version`, `gateway`, and `roles` blocks and all optional storage / network / security / monitoring blocks.

✅ Provider / Versions

Requirement	Value
Terraform	`>= 1.12.0`
`hashicorp/azurerm`	`~> 4.0`
Provider block	None in this module — the caller configures `provider "azurerm" { features {} }`, auth, and subscription.

Schema notes that bite (verified against the live provider schema):

Nearly the whole cluster is force-new: name, resource_group_name, location, cluster_version, tier, tls_min_version, encryption_in_transit_enabled, component_version, the roles VM sizes / usernames / SSH keys / subnet & VNet injection, storage_account / storage_account_gen2, network, private_link_configuration, compute_isolation, disk_encryption, and security_profile. Changing any of these replaces the cluster.
Mutable in place: tags, roles.worker_node.target_instance_count, roles.worker_node.autoscale, and the gateway credentials.
Exactly one default storage (is_default = true) is required — a storage_account_gen2 (preferred) or one storage_account.
The Enterprise Security Package (security_profile) requires the Premium tier and a VNet-injected topology.
encryption_in_transit_enabled defaults to true here, diverging from the provider's own default of false.
Node credentials must be a password or one or more ssh_keys — never both, never neither. The provider reports the conflict at apply; this module rejects it at plan.

🔑 Required Azure RBAC Roles / Permissions

Contributor at the target resource group (or a custom role covering Microsoft.HDInsight/clusters/*), scoped as narrowly as the deployment allows.
For ADLS Gen2 + managed-identity storage: the cluster's user-assigned identity needs Storage Blob Data Owner on the storage account, and the deploying identity needs Managed Identity Operator on that identity.
For customer-managed-key disk encryption: the identity behind disk_encryption.key_vault_managed_identity_id needs Key Vault Crypto Service Encryption User on the key.

Azure Prerequisites

An existing resource group in a supported US Azure region.
The Microsoft.HDInsight resource provider registered on the target subscription.
A default backing store already provisioned: an ADLS Gen2 filesystem (preferred) or a blob container.
For VNet injection: an existing VNet and subnet with sufficient address space and the HDInsight NSG / service-endpoint rules in place.
For the Enterprise Security Package: an Azure AD Domain Services managed domain with secure LDAP (LDAPS), a domain-join account, and a user-assigned identity.
The caller configures the provider "azurerm" { features {} } block, auth, and subscription.

📁 Module Structure

terraform-azurerm-hdinsight-spark-cluster/
├── providers.tf   # required_version >= 1.12.0; azurerm ~> 4.0; no provider block
├── variables.tf   # deeply-typed object() schemas + tags/timeouts tail
├── main.tf        # keystone azurerm_hdinsight_spark_cluster.this; dynamic blocks + try()
├── outputs.tf     # id, name, https_endpoint, ssh_endpoint
├── README.md      # this document
├── SCOPE.md       # cross-module contract
├── LICENSE        # MIT
└── .gitignore     # canonical library ignore set

⚙️ Quick Start

provider "azurerm" {
  features {}
}

module "spark" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-hdinsight-spark-cluster.git?ref=v1.0.0"

  name                = "hdi-spark-analytics"
  resource_group_name = "rg-analytics-eastus2"
  location            = "eastus2"
  cluster_version     = "5.1"
  tier                = "Standard"

  component_version = { spark = "3.3" }

  gateway = {
    username = "admin"
    password = var.gateway_password # provision out of band; never commit
  }

  storage_account_gen2 = {
    is_default                   = true
    filesystem_id                = var.adls_filesystem_id
    storage_resource_id          = var.storage_account_id
    managed_identity_resource_id = var.cluster_identity_id
  }

  roles = {
    head_node = {
      vm_size  = "Standard_D12_v2"
      username = "sshuser"
      ssh_keys = [var.ssh_public_key]
    }
    worker_node = {
      vm_size               = "Standard_D13_v2"
      username              = "sshuser"
      ssh_keys              = [var.ssh_public_key]
      target_instance_count = 3
    }
    zookeeper_node = {
      vm_size  = "Standard_D12_v2"
      username = "sshuser"
      ssh_keys = [var.ssh_public_key]
    }
  }

  tags = { environment = "prod", workload = "spark" }
}

ℹ️ The caller owns the provider, its authentication, and the mandatory features {} block. This module never declares them.

🔌 Cross-Module Contract

Consumes

Input	Type	Source module
`resource_group_name`	`string`	`terraform-azurerm-resource-group` (`name`)
`location`	`string`	caller / `terraform-azurerm-resource-group` (`location`)
`roles.*.subnet_id`	`string`	`terraform-azurerm-virtual-network` (`subnet_ids`)
`storage_account_gen2.filesystem_id` / `storage_resource_id`	`string`	`terraform-azurerm-storage-account`
`storage_account_gen2.managed_identity_resource_id`	`string`	`terraform-azurerm-user-assigned-identity` (`id`)
`disk_encryption.key_vault_key_id`	`string`	`terraform-azurerm-key-vault`
`monitor` / `extension.log_analytics_workspace_id`	`string`	`terraform-azurerm-log-analytics-workspace`

Emits

Output	Description	Consumed by
`id`	Cluster Resource ID (first)	downstream modules, diagnostics
`name`	Cluster name	tagging / diagnostics conventions
`https_endpoint`	HTTPS Ambari/gateway endpoint	client tooling, ingress wiring
`ssh_endpoint`	SSH connectivity endpoint	operator access, bastion wiring

📚 Example Library

1 · Minimal Standard cluster with SSH keys (secure baseline)

module "spark" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-hdinsight-spark-cluster.git?ref=v1.0.0"

  name                = "hdi-spark-min"
  resource_group_name = "rg-analytics-eastus2"
  location            = "eastus2"
  cluster_version     = "5.1"
  component_version   = { spark = "3.3" }

  gateway = { username = "admin", password = var.gateway_password }

  storage_account_gen2 = {
    is_default                   = true
    filesystem_id                = var.adls_filesystem_id
    storage_resource_id          = var.storage_account_id
    managed_identity_resource_id = var.cluster_identity_id
  }

  roles = {
    head_node      = { vm_size = "Standard_D12_v2", username = "sshuser", ssh_keys = [var.ssh_public_key] }
    worker_node    = { vm_size = "Standard_D13_v2", username = "sshuser", ssh_keys = [var.ssh_public_key], target_instance_count = 3 }
    zookeeper_node = { vm_size = "Standard_D12_v2", username = "sshuser", ssh_keys = [var.ssh_public_key] }
  }
}

🔒 With no posture overrides, the cluster comes up at TLS 1.2 with in-transit encryption enabled and key-based node auth.

2 · ADLS Gen2 + managed identity default storage

storage_account_gen2 = {
  is_default                   = true
  filesystem_id                = azurerm_storage_data_lake_gen2_filesystem.spark.id
  storage_resource_id          = azurerm_storage_account.spark.id
  managed_identity_resource_id = azurerm_user_assigned_identity.spark.id
}

💡 The managed-identity path keeps no storage account key in module state. This is the preferred backing store for regulated data.

3 · Load-based autoscale workers

roles = {
  head_node      = { vm_size = "Standard_D12_v2", username = "sshuser", ssh_keys = [var.ssh_public_key] }
  zookeeper_node = { vm_size = "Standard_D12_v2", username = "sshuser", ssh_keys = [var.ssh_public_key] }
  worker_node = {
    vm_size               = "Standard_D13_v2"
    username              = "sshuser"
    ssh_keys              = [var.ssh_public_key]
    target_instance_count = 3
    autoscale = {
      capacity = { min_instance_count = 3, max_instance_count = 10 }
    }
  }
}

ℹ️ target_instance_count and autoscale are mutable in place — scaling does not replace the cluster.

4 · Schedule-based autoscale workers

worker_node = {
  vm_size               = "Standard_D13_v2"
  username              = "sshuser"
  ssh_keys              = [var.ssh_public_key]
  target_instance_count = 3
  autoscale = {
    recurrence = {
      timezone = "Eastern Standard Time"
      schedule = [
        { days = ["Monday", "Tuesday", "Wednesday", "Thursday", "Friday"], time = "08:00", target_instance_count = 8 },
        { days = ["Saturday", "Sunday"], time = "08:00", target_instance_count = 3 },
      ]
    }
  }
}

💡 Use a recurrence schedule to pre-scale for business-hours load and shrink off-hours.

5 · VNet-injected cluster

roles = {
  head_node = {
    vm_size            = "Standard_D12_v2"
    username           = "sshuser"
    ssh_keys           = [var.ssh_public_key]
    subnet_id          = var.subnet_id
    virtual_network_id = var.virtual_network_id
  }
  worker_node = {
    vm_size               = "Standard_D13_v2"
    username              = "sshuser"
    ssh_keys              = [var.ssh_public_key]
    target_instance_count = 4
    subnet_id             = var.subnet_id
    virtual_network_id    = var.virtual_network_id
  }
  zookeeper_node = {
    vm_size            = "Standard_D12_v2"
    username           = "sshuser"
    ssh_keys           = [var.ssh_public_key]
    subnet_id          = var.subnet_id
    virtual_network_id = var.virtual_network_id
  }
}

⚠️ Injecting nodes into a subnet is force-new. The subnet must carry the HDInsight NSG rules and service endpoints before apply.

6 · Private Link enabled

network = {
  connection_direction = "Outbound"
  private_link_enabled = true
}

private_link_configuration = {
  group_id = "gateway"
  name     = "spark-plc"
  ip_configuration = {
    name                         = "spark-plc-ip"
    primary                      = true
    private_ip_allocation_method = "Dynamic"
    subnet_id                    = var.subnet_id
  }
}

🔒 Private Link removes the public gateway. Pair it with VNet-injected nodes and private DNS.

7 · Enterprise Security Package (ESP) with AADDS

tier = "Premium" # ESP requires Premium

security_profile = {
  aadds_resource_id       = var.aadds_resource_id
  domain_name             = "contoso.onmicrosoft.com"
  domain_username         = "hdiadmin@contoso.onmicrosoft.com"
  domain_user_password    = var.domain_user_password # sensitive; provision out of band
  ldaps_urls              = ["ldaps://contoso.onmicrosoft.com:636"]
  msi_resource_id         = var.cluster_identity_id
  cluster_users_group_dns = ["CN=Analysts,OU=Groups,DC=contoso,DC=onmicrosoft,DC=com"]
}

🔒 ESP domain-joins the cluster for Kerberos/LDAP auth and Ranger authorization. domain_user_password is redacted from plan and apply output (not from state, which holds the value in plaintext).

8 · In-transit encryption + customer-managed-key disk encryption

encryption_in_transit_enabled = true # default; shown explicitly

disk_encryption = [{
  encryption_algorithm          = "RSA-OAEP-256"
  encryption_at_host_enabled    = true
  key_vault_key_id              = var.cmk_key_id
  key_vault_managed_identity_id = var.cluster_identity_id
}]

🔒 CMK disk encryption binds node disks to your Key Vault key. The identity needs Key Vault Crypto Service Encryption User on that key.

9 · Log Analytics monitoring (monitor + extension)

monitor = {
  log_analytics_workspace_id = var.law_workspace_id # workspace GUID
  primary_key                = var.law_primary_key  # sensitive; provision out of band
}

extension = {
  log_analytics_workspace_id = var.law_resource_id # workspace resource ID
  primary_key                = var.law_primary_key
}

⚠️ monitor takes the workspace GUID; extension takes the workspace resource ID. The primary key is redacted from plan and apply output (not from state, which holds the value in plaintext).

10 · Hardened TLS floor (explicit)

tls_min_version               = "1.2" # default; the only recommended value
encryption_in_transit_enabled = true

🔒 tls_min_version is force-new. Lowering it is an explicit opt-out and is rejected on most current cluster versions.

11 · Compute isolation (dedicated hosts)

compute_isolation = {
  compute_isolation_enabled = true
  host_sku                  = "Standard_D14_v2"
}

💡 Compute isolation places the cluster on dedicated hardware for tenancy-sensitive workloads.

12 · External SQL metastores (Ambari / Hive / Oozie)

metastores = {
  ambari = { server = var.sql_server_fqdn, database_name = "ambaridb", username = "sqladmin", password = var.ambari_password }
  hive   = { server = var.sql_server_fqdn, database_name = "hivedb", username = "sqladmin", password = var.hive_password }
  oozie  = { server = var.sql_server_fqdn, database_name = "ooziedb", username = "sqladmin", password = var.oozie_password }
}

🔒 External metastores make cluster metadata durable beyond the cluster lifecycle. Each password is redacted from plan and apply output (not from state, which holds the value in plaintext).

13 · Provisioning script actions on nodes

roles = {
  head_node = {
    vm_size  = "Standard_D12_v2"
    username = "sshuser"
    ssh_keys = [var.ssh_public_key]
    script_actions = [{
      name = "install-libs"
      uri  = "https://example.blob.core.windows.net/scripts/install.sh"
    }]
  }
  worker_node    = { vm_size = "Standard_D13_v2", username = "sshuser", ssh_keys = [var.ssh_public_key], target_instance_count = 3 }
  zookeeper_node = { vm_size = "Standard_D12_v2", username = "sshuser", ssh_keys = [var.ssh_public_key] }
}

ℹ️ Script actions run at provisioning time. Keep scripts in a store the cluster identity can read.

14 · Legacy WASB storage (blob container + key)

storage_account = [{
  is_default           = true
  storage_container_id = azurerm_storage_container.spark.id
  storage_resource_id  = azurerm_storage_account.spark.id
  storage_account_key  = var.storage_account_key # sensitive; provision out of band
}]

⚠️ Prefer storage_account_gen2 with a managed identity. Use the key-based WASB path only when a blob-container backing store is required; the key is redacted from plan and apply output (not from state, which holds the value in plaintext).

15 · 🏗️ End-to-end composition

provider "azurerm" {
  features {}
}

module "rg" {
  source   = "git::https://github.com/microsoftexpert/terraform-azurerm-resource-group.git?ref=v1.0.0"
  name     = "rg-analytics-eastus2"
  location = "eastus2"
}

module "network" {
  source              = "git::https://github.com/microsoftexpert/terraform-azurerm-virtual-network.git?ref=v1.0.0"
  name                = "vnet-analytics"
  resource_group_name = module.rg.name
  location            = module.rg.location
  address_space       = ["10.40.0.0/16"]
  subnets = {
    spark = { name = "snet-spark", address_prefixes = ["10.40.1.0/24"] }
  }
}

module "identity" {
  source              = "git::https://github.com/microsoftexpert/terraform-azurerm-user-assigned-identity.git?ref=v1.0.0"
  name                = "id-spark"
  resource_group_name = module.rg.name
  location            = module.rg.location
}

module "storage" {
  source              = "git::https://github.com/microsoftexpert/terraform-azurerm-storage-account.git?ref=v1.0.0"
  name                = "stsparkanalytics01"
  resource_group_name = module.rg.name
  location            = module.rg.location
  is_hns_enabled      = true # required for a Gen2 filesystem
}

# The Gen2 filesystem is its own resource, not a collection on the storage account.
module "data_filesystem" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-storage-data-lake-gen2-filesystem.git?ref=v1.0.0"

  name               = "data"
  storage_account_id = module.storage.id
}

module "spark" {
  source = "git::https://github.com/microsoftexpert/terraform-azurerm-hdinsight-spark-cluster.git?ref=v1.0.0"

  name                = "hdi-spark-analytics"
  resource_group_name = module.rg.name
  location            = module.rg.location
  cluster_version     = "5.1"
  component_version   = { spark = "3.3" }

  gateway = { username = "admin", password = var.gateway_password }

  storage_account_gen2 = {
    is_default                   = true
    filesystem_id                = module.data_filesystem.id
    storage_resource_id          = module.storage.id
    managed_identity_resource_id = module.identity.id
  }

  roles = {
    head_node = {
      vm_size            = "Standard_D12_v2"
      username           = "sshuser"
      ssh_keys           = [var.ssh_public_key]
      subnet_id          = module.network.subnet_ids["spark"]
      virtual_network_id = module.network.id
    }
    worker_node = {
      vm_size               = "Standard_D13_v2"
      username              = "sshuser"
      ssh_keys              = [var.ssh_public_key]
      target_instance_count = 4
      subnet_id             = module.network.subnet_ids["spark"]
      virtual_network_id    = module.network.id
    }
    zookeeper_node = {
      vm_size            = "Standard_D12_v2"
      username           = "sshuser"
      ssh_keys           = [var.ssh_public_key]
      subnet_id          = module.network.subnet_ids["spark"]
      virtual_network_id = module.network.id
    }
  }

  tags = { environment = "prod", workload = "spark" }
}

💡 This wiring shows the intended dependency order: resource group → VNet subnet + managed identity + ADLS Gen2 storage → cluster. Output names on sibling modules are illustrative; match them to the versions you pin.

📥 Inputs

Required: name, resource_group_name, location, cluster_version, component_version, gateway, roles.

Posture (secure defaults): tier ("Standard"), tls_min_version ("1.2"), encryption_in_transit_enabled (true), zones.

Storage: storage_account_gen2 (preferred), storage_account (legacy WASB).

Networking / isolation: network, private_link_configuration, compute_isolation.

Encryption / security / monitoring: disk_encryption, security_profile, monitor, extension, metastores.

Universal tail: tags, timeouts.

Full object() schemas

variable "component_version" {
  type = object({ spark = string })
}

variable "gateway" {
  type = object({
    username = string
    password = string # sensitive
  })
  sensitive = true
}

variable "roles" {
  type = object({
    head_node = object({
      vm_size            = string
      username           = string
      ssh_keys           = optional(set(string))
      password           = optional(string)
      subnet_id          = optional(string)
      virtual_network_id = optional(string)
      script_actions     = optional(list(object({ name = string, uri = string, parameters = optional(string) })), [])
    })
    worker_node = object({
      vm_size               = string
      username              = string
      target_instance_count = number
      ssh_keys              = optional(set(string))
      password              = optional(string)
      subnet_id             = optional(string)
      virtual_network_id    = optional(string)
      autoscale = optional(object({
        capacity   = optional(object({ min_instance_count = number, max_instance_count = number }))
        recurrence = optional(object({
          timezone = string
          schedule = list(object({ days = list(string), time = string, target_instance_count = number }))
        }))
      }))
      script_actions = optional(list(object({ name = string, uri = string, parameters = optional(string) })), [])
    })
    zookeeper_node = object({
      vm_size            = string
      username           = string
      ssh_keys           = optional(set(string))
      password           = optional(string)
      subnet_id          = optional(string)
      virtual_network_id = optional(string)
      script_actions     = optional(list(object({ name = string, uri = string, parameters = optional(string) })), [])
    })
  })
}

variable "tier"                          { type = string, default = "Standard" }  # Standard | Premium
variable "tls_min_version"               { type = string, default = "1.2" }       # 1.0 | 1.1 | 1.2
variable "encryption_in_transit_enabled" { type = bool,   default = true }
variable "zones"                         { type = set(string), default = null }

variable "storage_account_gen2" {
  type = object({
    filesystem_id                = string
    is_default                   = bool
    managed_identity_resource_id = string
    storage_resource_id          = string
  })
  default = null
}

variable "storage_account" {
  type = list(object({
    is_default           = bool
    storage_container_id = string
    storage_account_key  = string # sensitive
    storage_resource_id  = optional(string)
  }))
  default = []
}

variable "network" {
  type = object({
    connection_direction = optional(string, "Outbound") # Inbound | Outbound
    private_link_enabled = optional(bool, false)
  })
  default = null
}

variable "private_link_configuration" {
  type = object({
    group_id = string
    name     = string
    ip_configuration = object({
      name                         = string
      primary                      = optional(bool)
      private_ip_address           = optional(string)
      private_ip_allocation_method = optional(string)
      subnet_id                    = optional(string)
    })
  })
  default = null
}

variable "compute_isolation" {
  type = object({
    compute_isolation_enabled = optional(bool, false)
    host_sku                  = optional(string)
  })
  default = null
}

variable "disk_encryption" {
  type = list(object({
    encryption_algorithm          = optional(string) # RSA-OAEP | RSA-OAEP-256 | RSA1_5
    encryption_at_host_enabled    = optional(bool)
    key_vault_key_id              = optional(string)
    key_vault_managed_identity_id = optional(string)
  }))
  default = []
}

variable "security_profile" {
  type = object({
    aadds_resource_id       = string
    domain_name             = string
    domain_username         = string
    domain_user_password    = string # sensitive
    ldaps_urls              = set(string)
    msi_resource_id         = string
    cluster_users_group_dns = optional(set(string))
  })
  default = null
}

variable "monitor" {
  type    = object({ log_analytics_workspace_id = string, primary_key = string }) # primary_key sensitive
  default = null
}

variable "extension" {
  type    = object({ log_analytics_workspace_id = string, primary_key = string }) # primary_key sensitive
  default = null
}

variable "metastores" {
  type = object({
    ambari = optional(object({ database_name = string, server = string, username = string, password = string }))
    hive   = optional(object({ database_name = string, server = string, username = string, password = string }))
    oozie  = optional(object({ database_name = string, server = string, username = string, password = string }))
  })
  default = null
}

variable "tags"     { type = map(string), default = {} }
variable "timeouts" { type = object({ create = optional(string), read = optional(string), update = optional(string), delete = optional(string) }), default = null }

🧾 Outputs

Output	Description	Kind
`id`	The Azure Resource ID of the HDInsight Spark cluster	Passthrough
`name`	The name of the HDInsight Spark cluster	Passthrough
`location`	Azure region the resource is deployed in, in the canonical form Azure uses	Passthrough
`https_endpoint`	The HTTPS Ambari/gateway endpoint of the cluster	Passthrough
`ssh_endpoint`	The SSH connectivity endpoint of the cluster	Passthrough
`name_tier_and_cluster_version_are_all_force_new`	Always true, and invisible to a `ForceNew:` search of the resource file	Constant
`cluster_version_diffs_are_suppressed_below_the_major_minor`	Always true	Constant
`name_is_the_public_dns_label`	Always true	Constant
`cluster_type_is_fixed_by_the_resource`	Always true, and it is why this family is six separate resources rather than one with a type argument	Constant
`monitor_and_extension_both_take_the_workspace_guid`	Always true, and this module previously said otherwise	Constant
`workspace_primary_key_drift_is_invisible`	Always true, for both `monitor` and `extension`	Constant
`gateway_credentials_are_mandatory`	Always true, and it is why this module has no passwordless option to default to	Constant
`disk_encryption_requires_a_versioned_key`	True when disk encryption is configured	Passthrough
`enterprise_security_package_is_entirely_force_new`	True when the cluster runs with an Enterprise Security Package domain profile	Derived
`cluster_kind_is_fixed_by_the_resource`	Always true, and it is why this family is six separate resources rather than one with a type argument	Constant

| default_storage_is_gen2 | Whether the default (primary) filesystem is the ADLS Gen2 account rather than a classic storage account — it decides which secondary storage Azure will accept | Derived | | attaches_unsupported_secondary_storage | True when a classic account holds the default and a Gen2 filesystem is also attached — the one storage combination Azure documents as unsupported, and the provider checks nothing across the two arguments | Derived |

| gen2_account_limit_is_provider_imposed | true — Azure allows several Gen2 accounts sharing one managed identity for this cluster type; the provider's single-entry cap makes that unreachable | Constant |

No password or workspace key is emitted. Those fields are declared sensitive by the live provider schema and are redacted from plan and apply output (not from state, which holds the value in plaintext).

🧠 Architecture Notes

Immutability first. Because almost every field forces replacement, treat a cluster spec as effectively fixed. Only tags, worker target_instance_count, autoscale, and gateway credentials can change without a destroy/create. Plan the storage model, node SKUs, and network placement deliberately.
One default storage. HDInsight rejects a cluster with no default backing store or more than one. Set is_default = true on exactly one of storage_account_gen2 or a single storage_account.
Secrets are inputs, never outputs. The gateway credential is a required, sensitive input. Node passwords, the ESP domain_user_password, monitor / extension primary keys, metastore passwords, and the WASB storage_account_key are declared sensitive by the provider, so they are redacted from plan and apply output (not from state, which holds the value in plaintext); provision each out of band and prefer the passwordless paths (SSH keys, managed identity).
Node credentials fail at plan. The password/ssh_keys exclusive-or is enforced by a validation {} block rather than left to the provider's apply-time check, because a 20-minute cluster provision is an expensive place to learn about a configuration mistake.
features {} dependence. The module carries no provider {} block. If it appears not to initialize in isolation, the cause is a missing caller-side provider "azurerm" { features {} }.
Total rendering. Every optional block is rendered through a dynamic block guarded on a non-null input, with try(...) on each optional nested field, so an omitted key renders as absent rather than erroring.
Which storage may be secondary depends on which is primary, and nothing in the provider says so. Azure supports a Gen2 primary with classic GPv1/GPv2 or Block Blob secondaries; it does not support a classic primary with a Gen2 secondary. The restriction is one-directional — the same two accounts are a supported cluster or a rejected one depending purely on which of them carries is_default. The provider models storage_account and storage_account_gen2 independently and checks nothing across them, so the unsupported pairing plans cleanly and fails when the cluster is created — attaches_unsupported_secondary_storage reports it. The rule is Microsoft's to change, so this module reports it rather than refusing it.
The default is counted across BOTH storage arguments. The provider merges them into one array before sending them to Azure and never examines is_default, so a cluster with none or with two is accepted at plan. Counting only storage_account would refuse a default placed on the Gen2 account — which is the arrangement this module's own inputs recommend.
One Gen2 account is a provider limit here, not an Azure limit. Azure supports attaching several Data Lake Storage Gen2 accounts to this cluster type, as long as they all use the same user-assigned managed identity for cluster access. The provider caps storage_account_gen2 at a single entry — behind a source comment recording that the author was unsure whether more were allowed — so that supported topology cannot be expressed through this module. Nothing signals it: the argument is simply singular. gen2_account_limit_is_provider_imposed reports it, and no module can widen a provider cap.

🧱 Design Principles

Concern	Secure default (empty call)	Opt-out (caller must type it)
TLS floor	`tls_min_version = "1.2"`	set `"1.0"` / `"1.1"` (usually rejected)
In-transit encryption	`encryption_in_transit_enabled = true`	set `false`
Node authentication	`ssh_keys` preferred; `password` omitted	supply a node `password`
Credential ambiguity	exactly one of password / `ssh_keys`, enforced at plan	— (no opt-out)
Default storage	ADLS Gen2 + managed identity (`storage_account_gen2`)	key-based WASB (`storage_account`)
Disk encryption	platform-managed at rest	supply `disk_encryption` CMK
Network placement	platform default (outbound)	VNet injection + `network.private_link_enabled = true`
Enterprise auth	non-ESP	supply `security_profile` (requires Premium `tier`)

🚀 Runbook

terraform init -backend=false
terraform validate
terraform fmt -check

Pin the module with ?ref=v1.0.0 — never a branch.
This library is plan-only during authoring; a human runs terraform plan / apply from CI against real credentials.

🧪 Testing

terraform validate proves the configuration is internally consistent and type-correct against the pinned provider schema — it catches every typing mistake the object() schemas are designed to surface, plus the tier / tls_min_version / connection_direction / encryption_algorithm enums and the node-credential exclusive-or.
terraform fmt -check enforces canonical formatting.
Neither command calls Azure. Only terraform plan (run by a human, from CI) exercises the ARM API — the module ships without any cloud apply.
The default-storage rule is checked offline: exactly one is_default across storage_account and storage_account_gen2 together is enforced by a validation {} block. Because that condition reads a second variable it is evaluated at plan rather than by terraform validate, but either way a wrong count does not survive to the 20-minute cluster provision.

💬 Example Output

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

id             = "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/rg-analytics-eastus2/providers/Microsoft.HDInsight/clusters/hdi-spark-analytics"
name           = "hdi-spark-analytics"
https_endpoint = "hdi-spark-analytics.azurehdinsight.net"
ssh_endpoint   = "hdi-spark-analytics-ssh.azurehdinsight.net"

🔍 Troubleshooting

Symptom	Cause	Fix
`Provider configuration not present` / `features` error	No caller-side `provider "azurerm" { features {} }`.	Add the provider block with `features {}` in the root module.
Plan shows the cluster being replaced on a small change	The edited field is force-new (e.g. `tier`, `cluster_version`, `roles` SKU, storage, network).	Confirm the change is intended; force-new fields require a rebuild.
Plan error: exactly one of password or ssh_keys	A node set both credentials, or neither.	Set one. Prefer `ssh_keys`.
Apply fails: no default storage / multiple defaults	Zero or more than one `is_default = true` store.	Set `is_default = true` on exactly one `storage_account_gen2` or `storage_account`.
ESP apply rejected	`security_profile` set with `tier = "Standard"`.	Set `tier = "Premium"` and VNet-inject the nodes.
`monitor` accepts but no logs appear	`monitor` needs the workspace GUID; `extension` needs the workspace resource ID.	Pass the correct identifier to each block.
Node cannot reach storage	Cluster identity lacks data-plane access to the ADLS Gen2 filesystem.	Grant `Storage Blob Data Owner` to `managed_identity_resource_id` on the storage account.

🔗 Related Docs

azurerm provider — azurerm_hdinsight_spark_cluster
Azure HDInsight documentation
Sibling cluster types: terraform-azurerm-hdinsight-hadoop-cluster (adds an edge node), terraform-azurerm-hdinsight-hbase-cluster and terraform-azurerm-hdinsight-interactive-query-cluster (schedule-based autoscale only), terraform-azurerm-hdinsight-kafka-cluster (broker disks, no autoscale).
Sibling modules: terraform-azurerm-resource-group, terraform-azurerm-virtual-network, terraform-azurerm-storage-account, terraform-azurerm-user-assigned-identity, terraform-azurerm-log-analytics-workspace, terraform-azurerm-key-vault.
This module's SCOPE.md.

💙 "Infrastructure as Code should be standardized, consistent, and secure."

Navigation Menu

Repository files navigation

☁️ Azure Hdinsight Spark Cluster Terraform Module

🧩 Overview

❤️ Support this project

🗺️ Where this fits in the family

🧬 What this module builds

✅ Provider / Versions

🔑 Required Azure RBAC Roles / Permissions

Azure Prerequisites

📁 Module Structure

⚙️ Quick Start

🔌 Cross-Module Contract

📚 Example Library

📥 Inputs

🧾 Outputs

🧠 Architecture Notes

🧱 Design Principles

🚀 Runbook

🧪 Testing

💬 Example Output

🔍 Troubleshooting

🔗 Related Docs

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
SCOPE.md		SCOPE.md
main.tf		main.tf
outputs.tf		outputs.tf
providers.tf		providers.tf
variables.tf		variables.tf

Latest commit

History

Folders and files

Repository files navigation

☁️ Azure Hdinsight Spark Cluster Terraform Module

🧩 Overview

❤️ Support this project

🗺️ Where this fits in the family

🧬 What this module builds

✅ Provider / Versions

🔑 Required Azure RBAC Roles / Permissions

Azure Prerequisites

📁 Module Structure

⚙️ Quick Start

🔌 Cross-Module Contract

📚 Example Library

📥 Inputs

🧾 Outputs

🧠 Architecture Notes

🧱 Design Principles

🚀 Runbook

🧪 Testing

💬 Example Output

🔍 Troubleshooting

🔗 Related Docs

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages