Terraform-provider-aws: aws_acm_certificate subject_alternative_names & domain_validation_options get returned in a different order each time

We've been running plans & applies regularly in a state containing many aws_acm_certificates and aws_route53_records for their validation records. It appears the ordering started varying recently, I assume its related to an AWS API change.

I think one potential solution would be to have the provider sort the records internally prior to storing them in state. This could result in many aws_route53_record recreations for anyone using this workflow, however it would only be a one-time issue after upgrading the aws provider.

I have yet to test it, but I would assume using Terraform's sort() on aws_acm_certificate.foo.domain_validation_options when iterating over aws_route53_records would achieve something similar.

I thought about sort() too, but domain_validation_options is a list of maps rather than strings, so that’s not gonna work (could probably do some fancy surgery on it by turning it into strings, but that’s gonna be super messy!)

I agree that Terraform AWS provider should be sorting the AWS record by record_name prior to any kind of comparison.

FWIW, I did try the terraform sort() approach as a workaround and it worked for me because I only had one SAN and was prepared to enumerate the SANs in a local variable.

Here is a summary of what I did:

provider "null" {
}

locals {
    unsorted = [
        "${aws_acm_certificate.backoffice.domain_validation_options.0.resource_record_name}!0",
        "${aws_acm_certificate.backoffice.domain_validation_options.1.resource_record_name}!1"
    ]
    sorted = "${sort(local.unsorted)}"
    index = [
        "${element(split("!", local.sorted[0]),1)}",
        "${element(split("!", local.sorted[1]),1)}"
    ]
}

# for visibility

output "index" {
    value = "${local.index}"
}

output "unsorted" {
    value = "${local.unsorted}"
}

output "sorted" {
    value = "${local.sorted}"
}

resource "null_resource" "locals" {
    triggers = {
        index = "${local.index}"
        unsorted = "${local.unsorted}"
        sorted = "${local.sorted}"
    }
}
...

# the aws_route53_record

resource "aws_route53_record" "backoffice" {
  count   = "${length(local.index)}"
  name    = "${lookup(aws_acm_certificate.backoffice.domain_validation_options[local.index[count.index]], "resource_record_name")}"
  type    = "${lookup(aws_acm_certificate.backoffice.domain_validation_options[local.index[count.index]], "resource_record_type")}"
  zone_id = "${var.zone_id}"
  records = [
    "${lookup(aws_acm_certificate.backoffice.domain_validation_options[local.index[count.index]], "resource_record_value")}"
  ]
  ttl     = 60
}

Here's an easy way to confirm this change in behavior, run this on a cert with multiple SANs:

(for i in {0..10}; do aws acm describe-certificate --certificate-arn $CERTIFICATE_ARN --query "Certificate.DomainValidationOptions[*].DomainName" --output text; done ) | sort | uniq -c

   7 domain1.com    domain2.com
   4 domain2.com    domain1.com

Is there any workaround for this? I'm wary of allowing validation records to be created/destroyed because I have critical (ALB) resources that transitively depend on them via an aws_acm_certificate_validation.

@eriksw There’s no harm in momentarily destroying the validation records per se; the only thing it will likely do is prevent renewal of the certificate when it is close to expiry - so obviously you don’t want to remove the records forever, but removing/replacing them in a plan/apply is not going to affect anything.

Hi,
We had same issue on us-east-1 (not in us-east2), corrected by followed implementation

I tested proposed bypass by jonseymour
Validated on terraform version 0.11.10 and 0.11.13 for AWS region us-east-1 and us-east-2
Provider AWS version is 2.10.0

Important: on a new terraform project, count() can't be used in aws_route53_record.cert_validation_record
I put a local variable named 'entries_count' (no way to solve this)

locals {
entries_number = 5 // IMPORTANT => count() in aws_route53_record.cert_validation_record can't be computed by terraform
unsorted = [
"${aws_acm_certificate.frontend_cert.domain_validation_options.0.resource_record_name}!0",
"${aws_acm_certificate.frontend_cert.domain_validation_options.1.resource_record_name}!1",
"${aws_acm_certificate.frontend_cert.domain_validation_options.2.resource_record_name}!2",
"${aws_acm_certificate.frontend_cert.domain_validation_options.3.resource_record_name}!3",
"${aws_acm_certificate.frontend_cert.domain_validation_options.4.resource_record_name}!4"
]
sorted = "${sort(local.unsorted)}"
index = [
"${element(split("!", local.sorted[0]),1)}",
"${element(split("!", local.sorted[1]),1)}",
"${element(split("!", local.sorted[2]),1)}",
"${element(split("!", local.sorted[3]),1)}",
"${element(split("!", local.sorted[4]),1)}",
]
}

resource "aws_acm_certificate" "frontend_cert" {
domain_name = "${var.mydomain}"
subject_alternative_names = [
"alt-name-1.${var.mydomain}",
"alt-name-2.${var.mydomain}",
"alt-name-3.${var.mydomain}",
"alt-name-4.${var.mydomain}"
]
validation_method = "DNS"
lifecycle {
create_before_destroy = true
# DO NOT REMOVE THIS DUE TO BUG DETECTED ON AWS CERT ALTERNATIVE ORDERING
ignore_changes = ["subject_alternative_names"]
}
}

resource "aws_route53_record" "cert_validation_record" {
count = "${(local.entries_number)}"
name = "${lookup(aws_acm_certificate.frontend_cert.domain_validation_options[local.index[count.index]], "resource_record_name")}"
type = "${lookup(aws_acm_certificate.frontend_cert.domain_validation_options[local.index[count.index]], "resource_record_type")}"
zone_id = "${aws_route53_zone.public_zone.id}"
records = [
"${lookup(aws_acm_certificate.frontend_cert.domain_validation_options[local.index[count.index]], "resource_record_value")}"
]
ttl = 60
}

resource "aws_acm_certificate_validation" "cert" {
certificate_arn = "${aws_acm_certificate.frontend_cert.arn}"

validation_record_fqdns = [
"${aws_route53_record.cert_validation_record.*.fqdn}"
]
}

We faced the same problem in us-east-1 & eu-west-1 while in us-west-1 we do not face such an issue.

This is less than ideal...

Has anyone raised this to AWS/asked on the forum?

Also having this generate a lot of noise in TF runs, and whilst it might not do any harm it does make it challenging to convince people to review changes when there are some that need to be overlooked like this.

Has anyone raised this to AWS/asked on the forum?

We just opened a support case after running into this issue in eu-central-1 today. They have reached out to their internal service team for investigation.

@mlafeldt Cool, thanks for the info - would you be able to share their response please?

@mlafeldt Cool, thanks for the info - would you be able to share their response please?

Of course.

@mlafeldt Great, thanks!

I received this response:

My name is ** from AWS Premium Support and I will be assisting you with your case as it pertains to inconsistent result when querying the aws api.

Thank you for providing such a great level of detail it really does assist in being able to test and reproduce the issue.

I was able to run a similar test on my environments and noticed similar patterns of variance, in my case there where three names and the variance was correspondingly more random.

The reasons for seeing this is because the aws api uses asynchronous processing and multiple threads can be handled simultaneously, because of this the order of the responses is not guaranteed any consistency is incidental rather than intentional.

I see this or a regular bases when we use our tooling on the back end. We always get the same set of results but the order varies.

The best publicly available documentation I could find for this that explains the concept is:

https://docs.aws.amazon.com/sdk-for-java/v2/developer-guide/basics-async.html

The aws cli itself is an implementation of the sdk.

I can understand that this is something that might be causing an issue now, but as a general guide going forward I would recommended that you can and should expect variance in the order of results as there is no guarantee.

I trust this information will assist you. If however you need any further information or guidance please do not hesitate to reach out to us.

It sounds like the terraform provider will need to be able to handle inconsistent ordering.

We got this error recently, but we're using Terraform templates to add (or rename) alternative names (the alternative names is an input variable of type array), so the workaround

ignore_changes = ["subject_alternative_names"]

is not really useful for us :(

I just want to add that I don't believe that sorting the returned list is a good enough solution. It will only work if all SANs use the same hosted zone.

We have this scenario:
Cert with SANs.
List of Hosted Zones.

Let's say we have the following SANS:

• example.hosted.zone.one
• example.hosted.zone.two

We import the hosted zones and create our certs like this:

variable "uri_prefix" {
  default = "example"
}

variable "domains" {
  type = "list"
  default = ["hosted.zone.one","hosted.zone.two"]
}

data "aws_route53_zone" "example" {
  count = "${length(var.domains)}"
  name = "${var.domains[count.index]}"
}

resource "aws_acm_certificate" "example" {
  domain_name       = "${var.uri_prefix}.${var.domains[0]}"
  subject_alternative_names = "${formatlist("${var.uri_prefix}.%s", slice(var.domains,1,length(var.domains)))}"
  validation_method = "DNS"
}

resource "aws_route53_record" "example" {
  count   = "${length(var.domains)}"
  name    = "${lookup(aws_acm_certificate.example.domain_validation_options[count.index], "resource_record_name")}"
  type    = "${lookup(aws_acm_certificate.example.domain_validation_options[count.index], "resource_record_type")}"
  zone_id = "${element(data.aws_route53_zone.example.*.id, count.index)}"
  records = ["${lookup(aws_acm_certificate.example.domain_validation_options[count.index], "resource_record_value")}"]
  ttl     = 300
}

What will happen (current state):

If we are lucky and we get stuff in the right order the cert validated with the following route53 records:

• _7c0eff29a73892d0e65a23bc2b14f137.example.hosted.zone.one
• _4317e3f335540dce25b5580c159b55da.example.hosted.zone.two

If we are unlucky and get them in the wrong order, they will be assigned to the wrong hosted zone, resulting in this:

• _4317e3f335540dce25b5580c159b55da.example.hosted.zone.two.hosted.zone.one
• _7c0eff29a73892d0e65a23bc2b14f137.example.hosted.zone.one.hosted.zone.two

AWS (or Terraform) seems to assume we missed adding the domain name for our selected zone and helpfully add it for us.

Adding sort to the same scenario would guarantee the following:

• _4317e3f335540dce25b5580c159b55da.example.hosted.zone.two.hosted.zone.one
• _7c0eff29a73892d0e65a23bc2b14f137.example.hosted.zone.one.hosted.zone.two

If we are lucky, of course, the random id will coalign with the order of our domains.

With the current state, we can atleast rerun terraform apply until it gets right (which is cumbersome and stupid, but still possible). If the sort solution is added, you can rerun for infinity without it ever getting right. The risk of getting stuff wrong increases with every additional SAN you add to the above description.

I guess if aws_acm_certificate.example.domain_validation_options would be returned not as list of maps but rather as a nested map with domain_name and SANs as top level keys, that would be easier to handle. Not sure how much work it would take to implement on provider level though, or if it's even doable.

Here is my current workaround:

resource "aws_acm_certificate" "this" {
  validation_method = "DNS"
  domain_name       = "${local.dns_record}"

  subject_alternative_names = [
    "${local.dns_record_https}",
  ]

  lifecycle {
    ignore_changes = ["subject_alternative_names"]
  }
}

locals {
  dns_record       = "test.example.com"
  dns_record_https = "https.${local.dns_record}"

  validations = {
    "${replace(aws_acm_certificate.this.domain_validation_options.0.resource_record_name, "/(_[[:alnum:]]*\\.|\\.$)/", "")}" = {
      "name"  = "${aws_acm_certificate.this.domain_validation_options.0.resource_record_name}"
      "type"  = "${aws_acm_certificate.this.domain_validation_options.0.resource_record_type}"
      "value" = "${aws_acm_certificate.this.domain_validation_options.0.resource_record_value}"
    }

    "${replace(aws_acm_certificate.this.domain_validation_options.1.resource_record_name, "/(_[[:alnum:]]*\\.|\\.$)/", "")}" = {
      "name"  = "${aws_acm_certificate.this.domain_validation_options.1.resource_record_name}"
      "type"  = "${aws_acm_certificate.this.domain_validation_options.1.resource_record_type}"
      "value" = "${aws_acm_certificate.this.domain_validation_options.1.resource_record_value}"
    }
  }
}

resource "aws_route53_record" "this-validation" {
  zone_id = "${local.route_53_zone_id}"
  name    = "${lookup(local.validations[local.dns_record], "name")}"
  type    = "${lookup(local.validations[local.dns_record], "type")}"
  ttl     = "300"
  records = ["${lookup(local.validations[local.dns_record], "value")}"]
}

resource "aws_route53_record" "this-validation-https" {
  zone_id = "${local.route_53_zone_id}"
  name    = "${lookup(local.validations[local.dns_record_https], "name")}"
  type    = "${lookup(local.validations[local.dns_record_https], "type")}"
  ttl     = "300"
  records = ["${lookup(local.validations[local.dns_record_https], "value")}"]
}

resource "aws_acm_certificate_validation" "this-validation" {
  certificate_arn = "${aws_acm_certificate.this.arn}"

  validation_record_fqdns = [
    "${aws_route53_record.this-validation.fqdn}",
    "${aws_route53_record.this-validation-https.fqdn}",
  ]
}

AWS (or Terraform) seems to assume we missed adding the domain name for our selected zone and helpfully add it for us.

@Frogvall This is partially due to the way DNS works. If you include a . at the end of your domain name, it'll be treated as 'the end' of the record (though it's not actually going to help in this case anyway).

What you _could_ do, though, is look up the aws_route53_zone data source with the domains you get back in the domain_validation_options (rather than the local.domains that you initially used). That way, you'll be looking up the right zone, and the record will be applied to the right zone.

You'll still have the ordering issue wanting to replace the records at each apply, but at least it'll work.

@tdmalone I don't know how I missed that dns validation options also comes with a domain field.
Anyway, it would mean I have to import the routes twice, as we use the original ones in other places, but that would be fine. Still need to do the sorting stuff though. Or just accept them being recreated about every second time.

Possible workaround until #8657 is merged: https://github.com/terraform-providers/terraform-provider-aws/issues/8747#issuecomment-495029337

Anyone knows of a workaround that:

1. doesn't recreate the aws_acm_certificate when there's actually no change (only order of SAN), this is a problem because the "new" aws_acm_certificate would actually have the same ID as the old one, and then it will fail while trying to delete the old one because it's still in use.
2. allows me to add new SANs or change them dynamically

ignore_changes = ["subject_alternative_names"] doesn't help with 2.

I've no problem with the recreation of route53 records, other than being a bit slower.

The only workaround I've found is to run this before each terraform apply, so the certificate is always created in terraform "in place" when there's no SAN changes, and with a new resource when there are new/different SANs.

terraform state rm aws_acm_certificate.cert

@tdmalone Thanks for the tag there. We have the zone resource directly available in our context in that example, it also works with the data.aws_route53_zone

@aserrallerios I haven't tested it, but I think the idea would be to 'manually' sort the domain names coming from the validation_options, constructing hashes to gather the appropriate values (like some of the other examples here). Looks a bit of a mess though.

But if I'm not mistaken, sorting the validation_options would only prevent the recreation of the route53 records, so in addition to ignore_changes = ["subject_alternative_names"] it would prevent the certificate from being recreated in place (and fail because of that), but it wouldn't allow any further change to the certificate, making the certificate resource practically useless.

I just got word from AWS support that they're going to implement a fix:

Thank you for your patience while our service team was working on a fix for this issue. I am happy to inform you that our service team is deploying a fix for the issue which should put the order of the 'DomainValidationOptions' entries back to the deterministic order. Please note that it may take up to a week before the fix get deployed in all regions.

So we might not need to do anything in the provider. We'll see.

Oh wow! That’s the opposite to what they told @rifelpet.

Sorting in the provider _probably_ is still a good idea but an AWS fix would make it much less urgent.

Does it fixes for someone? I am on eu-west-1 region, still has an error. Can someone from the ohter region confirm that issue fixed?

I encountered same problem on us-east-1 & ap-northeast-1 regions. This problem has not been solved yet.

Same here from eu-west-1, still nothing :|

mlafeldt commented 3 days ago
...
Please note that it may take up to a week before the fix get deployed in all regions.

Not fixed in eu-central-1 yet.

But here's an easy way to check the (still random) order:

aws acm describe-certificate --certificate-arn arn:aws:acm:eu-central-1:xxx --region eu-central-1 | jq -r '.Certificate.DomainValidationOptions[].DomainName'

It looks like it has been fixed in us-east-1 and ap-northeast-1.

eu-west-1 has been fixed also

Interestingly, I'm seeing the domain_validation_options coming back in the same order each time now in ap-southeast-2, but the subject_alternative_names (the other part of the issue described here) are _kindof_ still in a random order.

That is, the random order of the subject_alternative_names now seems to be determined at the time the certificate is created. It stays the same after that, but it doesn't match the order I originally supplied in my TF resource.

I could of course re-order the list to match what AWS returns, which is probably a better workaround that using ignore_changes, but unfortunately in most of the cases I use the aws_acm_certificate resource the list of SANs is somewhat long, dynamically generated, and massaged through a few local values.

So - in other words, re-trying the example I gave at the top of this issue, I can now:

• run terraform apply to create a new certificate
• run terraform plan to see what the returned order of the SANs is
• change the order of my SANs (in my test case, to: three.example.com, five, nine, four, one, seven, two, eight, six)
• run terraform plan several times to observe no more diffs

@tdmalone I was wondering: is the order different on create only in ap-southeast-2 or could you observe this in other regions as well?

I'm seeing subject_alternative_names being reordered in us-east-1 for my CloudFront certificates, so I needed to use the workaround here.

We had this issue in us-east-1 yesterday. provider "aws" (1.60.0)

@tdmalone I have the same problem as you. "Random" order being defined at creation time (I'm using us-east-1 region).

@bflad Just checking, does the additional of the upstream label mean that this is being treated as a bug that we're waiting on AWS to resolve - i.e. that nothing will be done in the provider? (Or it more broad and simply means that the issue was caused by upstream changes?)

23 days ago I wrote:

I'm seeing the domain_validation_options coming back in the same order each time now in ap-southeast-2, but ... the random order of the subject_alternative_names now seems to be determined at the time the certificate is created.

Unfortunately it appears the same regression in AWS' API is back, and I'm now seeing domain_validation_options come back in a random order (in ap-southeast-2) on each request, which means a large number of erroneous diffs in many of our Terraform states.

If this is going to stick around, it'd be really great to get https://github.com/terraform-providers/terraform-provider-aws/pull/8708 merged.

We are still seeing the issue in us-east-1 too.

Which issue though? I believe there is 2 separate issues:

domain_validation_options in random order

This was an AWS issue where they were returned in random order from AWS API and Terraform was not sorting them. This is resolved by AWS and they are now returned in the same order from AWS API (or so they said)

subject_alternative_names

I think this is TF problem, they should sort them or something. The problem looks like this:

-/+ aws_acm_certificate.multi_eu_legacy (new resource required)
id: "xxxx" => (forces new resource)
arn: "xxxx" =>
domain_name: ".xxx" => ".xxx"
domain_validation_options.#: "4" =>
subject_alternative_names.#: "3" => "3"
subject_alternative_names.0: ".domain1.com" => "domain3.com" (forces new resource)
subject_alternative_names.1: ".domain2.com" => ".domain1.com" (forces new resource)
subject_alternative_names.2: "domain3.com" => ".domain2.com" (forces new resource)
```

You can workaround this by just changing the order in the TF file to the order in first column, i.e. before =>

It seems like that the AWS API is responding with an sorted domain_validation_options order, e.g:

[
   "*.zebra.net",
   "my.zebra.com",
  "*.zebra.com"
]

will generate domain_validation_options in the following order:

[
   "my.zebra.com",
  "*.zebra.com",
  "*.zebra.net"
]

Even if you use "*.zebra.net" as the domain name, it will return it in a different order. So an initial workaround would be to configure the aws_acm_certificate like this:

resource "aws_acm_certificate" "cert" {
  domain_name       = "my.zebra.com"
  validation_method = "DNS"

  subject_alternative_names = [
     "*.zebra.com",
     "*.zebra.net"
  ]

  lifecycle {
    create_before_destroy = true
  }
}

In case you have a list with an wildcard domain (*) together with a non wildcard domain, like *.zebra.com and zebra.com, as they require the same validation information, the API will return the non wildcard at the end of the list. e.g:

# vi acm.tf
resource "aws_acm_certificate" "cert" {
  domain_name       = "my.zebra.com"
  validation_method = "DNS"

  subject_alternative_names = [
    "*.my.zebra.com",
    "*.zebra.com",
    "*.zebra.net",
  ]

  lifecycle {
    create_before_destroy = true
  }
}


# CERT_ARN=ARN
# aws acm describe-certificate --certificate-arn $CERT_ARN  --query "Certificate.DomainValidationOptions[]..ValidationDomain"

[
    "*.my.zebra.com",
    "*.zebra.com",
    "*.zebra.net",
    "my.zebra.com",
]

This is still an issue in us-east-1
The list is still coming back garbled even though Ive put a list in order, on terraform plan it forces a new resource, if I use ignore_changes = ["subject_alternative_names"] it forces a destroy?? pretty weird behaviour.

this is the resource block:

resource "aws_acm_certificate" "this" {
  count                     = "${var.create_acm ? 1 : 0}"
  domain_name               = "${var.environment}.com"
  subject_alternative_names = [
                                "*.${var.environment}.click",
                                "*.${var.environment}.com",
                                "*.${var.environment}.info",
                                "*.${var.environment}.net",
                                "*.${var.environment}.org"
  validation_method         = "DNS"

  lifecycle {
    create_before_destroy = true
    ignore_changes = ["subject_alternative_names"]
  }

When might we see #8708 merged, as this is clearly problematic for a large number of folks.

Just to clear up some possible misunderstandings:

• AWS provided a fix for the ordering of domain_validation_options
• AWS has not, nor has stated anywhere I can see, that the same treatment is planned for subject_alternative_names (Some comments of the nature "let's see if they fix this" on this point, which I think confuse the two issues)
• The ordering of these things only matters in so far as terraform needs to compare for changes. So locally sorting the planned vs. realised resource prior to comparison should be safe, effective, and aside from incurring a one-time update for some folks, have no impact to users.

Thoughts?

I want to mention that this is not just about validation_record but also about aws_acm_certificate subject-alternative-names. This causes re-creation of ACM certificates which could potentially break a lot of things.

This issue also appears to affect auto adding records for SSL verification:

resource "aws_acm_certificate" "default" {
  domain_name       = "domain.com"
  validation_method = "DNS"

  subject_alternative_names = [
    "*.cdn.domain.com",
    "*.assets.cdn.domain.com",
    "*.api.domain.com",
    "*.domain.com",
  ]

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_route53_record" "ssl-verification-default" {
  zone_id = "${aws_route53_zone.default.zone_id}"
  ttl     = 300

  # Find the total for the loop (-1 since there is a duplicate due to "domain.com" and "*.domain.com")
  count = length(data.terraform_remote_state.global-certs-domain-com.outputs.cert_domain_validation_options)-1

  # This is the loop
  type = "CNAME"
  name = "${element(data.terraform_remote_state.global-certs-domain-com.outputs.cert_domain_validation_options.*.resource_record_name, count.index)}"
  records = [
    "${element(data.terraform_remote_state.global-certs-domain-comoutputs.cert_domain_validation_options.*.resource_record_value, count.index)}"
  ]

}

The -1 works for some project and some it won't. I believe this may be related. Just makes this impossible to use at this point. It will error out due to already having the entry in the list. AWS should fix this on their end IMHO.

I'm seeing the issue on new builds where the cert request is created but the route53 validation record isn't so the cert stays pending. When that happens the elb creation fails because it depends on the acm. Running the build a second time generates the route53 records, the certs validates, and the elb creates.

Update: a Hashicorp rep has confirmed that this issue is on the radar and needs some internal discussion.

would it be possible to change the subject_alternative_names to be a set instead of a list. AFAIK the order isn't significant in any way.

Also, is there any workable workaround for aws_acm_certificate? As @mzhaase said, recreating the certificate can cause service interruption.

New potential fix PR: https://github.com/terraform-providers/terraform-provider-aws/pull/10791

Started to have a look on this. IMHO we only need to use a set for subject_alternative_names.. That should fix all issues, right?

I think so. The only issue I see is maybe it breaks backwards compatibility. But semantically, the order isn't deterministic anyway...

I also thought that and I would need to use toset() now, but I am still assigning a list to subject_alternative_names in my test.. That works perfectly, but I do not understand why TF is not complaining about it. :-/

Did I missed something here?

TF will implicitly convert a list to a set if the schema requires a set. Backwards compatibility issues are more likely to stem from the fact that sets aren't indexable, or that the order of the domain_validation_options is no longer necessarily the same as the subject_alternative_names (although from the documentation, I don't think this is garanteed anyway).

Started to have a look on this. IMHO we only need to use a set for subject_alternative_names.. That should fix all issues, right?

I think sorting domain_validation_options is required (#10791 or similar) to fix not knowing which domain_validation_options entry to use for a given aws_route53_record/zone_id. (Or, better yet, keying domain_validation_options by domain_name instead of a flat list. )

... (Or, better yet, keying domain_validation_options by domain_name instead of a flat list. )

domain_name contains dots. It cannot be used as a key...

I think sorting domain_validation_options is required (#10791 or similar) to fix not knowing which domain_validation_options entry to use for a given aws_route53_record/zone_id.

This is an inadequacy of pre-TF0.12 and I do not see why a TF0.12 provider should behave in a way, that you may use multiple resources interoperably in conjunction with a pre-TF0.12 language subset.

With all due respect, it is a question of interpolation, which depends on the language version you are using. And you can do that with TF0.12.

In short:
From my point of view we face a bug which leads to the recreation of aws_acm_certificate for every apply - nothing else. We should fix this issue in a way old legacy implementations are not affected by changes in the behavior of this resource.

Everything else described here are deprecated feature requests (deprecated as its a question of interpolation capabilities) to get complex things done in a legacy version of the language, which had evolved in a manner, that you can accomplish this today.

domain_name contains dots. It cannot be used as a key...

what do you mean? map keys can contain arbitrary characters afaik.

I can confirm that the map keys can contain dots (in another issue apparentlymart stated that map keys are _always_ strings).

I am using an approach modelled after the terraform-aws-acm module, which creates a map of domain names with their options as the values. It works as desired except for var.domain_names needing to be reordered only after the first apply, because AWS returns the subject_alternative_names in a random order. If you don't run into this issue, it's luck of the draw that the AWS API returned the subject alternative names in the same order that was in your terraform code.

Here's what I'm doing:

variable "domain_names" {
  description = "Domains on certificate. First list item will be main domain on certificate."
  type        = list(string)
}

locals {
  # Trim off wildcards to get a list of distinct domains for validation
  distinct_domain_names = distinct([for domain in var.domain_names : replace(domain, "*.", "")])
  # Map of domain_validation_options for each distinct domain for cert validation
  validation_domains = { for option in aws_acm_certificate.cert.domain_validation_options : option.domain_name => option if contains(local.distinct_domain_names, option.domain_name) }
}

data "aws_route53_zone" "selected" {
  for_each = toset(local.distinct_domain_names)
  name     = each.value
}

# First domain name in var.domain_names is used as domain_name on the certificate
# All other domain names in var.domain_names go into subject alternative names
resource "aws_acm_certificate" "cert" {
  domain_name               = var.domain_names[0]
  # SANs may need to be reordered after creation, due to this bug:
  # https://github.com/terraform-providers/terraform-provider-aws/issues/8531
  subject_alternative_names = slice(var.domain_names, 1, length(var.domain_names))
  validation_method         = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_route53_record" "cert_validation" {
  for_each = toset(local.distinct_domain_names)
  name     = local.validation_domains[each.value].resource_record_name
  type     = local.validation_domains[each.value].resource_record_type
  zone_id  = data.aws_route53_zone.selected[each.value].zone_id
  records  = [local.validation_domains[each.value].resource_record_value]
  ttl      = "60"
}

resource "aws_acm_certificate_validation" "cert_validation" {
  certificate_arn         = aws_acm_certificate.cert.arn
  validation_record_fqdns = [for cert_validation in aws_route53_record.cert_validation : cert_validation.fqdn]
}

Looks like that breaks aws_acm_certificate management, this resource is going to be recreated on each apply:

resource "aws_acm_certificate" "cert" {
  domain_name       = "test1.example.com"
  validation_method = "DNS"
  subject_alternative_names = ["test2.example.com", "test3.example.com"]
}

Why not just sort subject_alternative_names before comparing?

Sorting before compare will break all current implementations on update.

@mzhaase I mean sort for both lists

Sorting before compare will break all current implementations on update.

Why not sort both the provider response and the user supplied data? That shouldn't break existing implementations?

The sorting would have to be done _only_ for purposes of determining whether the SANs had changed.

One thing that can't change without breaking users' existing modules is the ordering of domain_validation_options—that needs to be stable for the life of the aws_acm_certificate to avoid breaking pre-for_each workarounds like mine: acm_certificate.tf

Can we get it tracked for a major (breaking) provider version that subject_alternative_names should be a _set_ and that domain_validation_options should be a _map_?

👉For the meantime (non-breaking), it would be a convenience win if a for_each-friendly domain_validation_options_map was added.

I am also facing this issue, If i add new SAN it is trying to create the new ACM domain and trying to delete the existing one which has been already used in ALB https listener ... is this can be solved?

I am also facing this issue, If i add new SAN it is trying to create the new ACM domain and trying to delete the existing one which has been already used in ALB https listener ... is this can be solved?

@Bharathkumarraju You need to use

lifecycle {
   create_before_destroy = true
}

on your aws_acm_certificate resource.

Also, to work around this issue I am simply creating one certificate per domain and then attaching them all to my load balancer with aws_lb_listener_certificate. To be honest it works a lot better anyway.

@asztal Thanks for your recommendation. I am already using it but my problem using i am also using SANs as list and these SANs can be customised as well. If i add another SAN to my domain it is trying to delete old domain itself and trying to create new ACM domain :(

@Bharathkumarraju it isn't possible to add a new SAN to an existing certificate (this is just how SSL/TLS certificates work). You need to create a new certificate, change the ALB to point to the new certificate, then remove the old one.

@asztal that's not going to work with cloudfront.

or classic ELBs

some interesting solutions here. Sadly I am working with a handful of * certs and their corresponding non-star form (and making matters worse, each is its own route53-zone):

• example: foo00.example.com, *.foo00.example.com, bar00.example.com, *.bar00.example.com

The validation-options were originally in the a reverse-sort manner with just *-certs (or just the plain-form), however when mixing it looked very mangled.

I've settled with the ignore_changes, as once this cert gets used, we'll likely create a new cert to blue-green integrate it with more domains.

Dynamically the route53 certificate validation became a bit more difficult, due to the number of duplicated elements within the list-of-maps.

My solution is kind of ugly; I decided to turn the list-of-maps into a list of list-of-strings (where the strings are separated with whatever-characters). Then i ran sort+distinct+compact on the resulting list, eliminating duplicates enabling me to split out items per-element in the route53 block.

for TF 0.12.21

locals {
  multi_star_cert_helper_text = <<EOF
%{~ for amap in aws_acm_certificate.multi_star_cert.domain_validation_options ~}
${amap.resource_record_name},wtf,${amap.resource_record_type},wtf,${amap.resource_record_value};wtf;
%{~ endfor ~}
EOF

  multi_star_cert_helper_text_list = sort(distinct(compact(split(";wtf;", local.multi_star_cert_helper_text))))
}


data "aws_route53_zone" "sub_zones" {
  for_each = toset(["foo00", "bar00", "baz00"])
  name = "${each.value}.example.com"
}


resource "aws_route53_record" "multi_star_cert_validation" {
  count = length( local.multi_star_cert_helper_text_list )

  name    = element( compact(split(",wtf,", local.multi_star_cert_helper_text_list[count.index])), 0 )
  type    = element( compact(split(",wtf,", local.multi_star_cert_helper_text_list[count.index])), 1 )

  zone_id = data.aws_route53_zone.sub_zones[ replace(  element( compact(split(",wtf,", local.multi_star_cert_helper_text_list[count.index])), 0 ),  "/(^_[^.]+.|.example.com.?$)/",  "") ].id
  records = [ element( compact(split(",wtf,", local.multi_star_cert_helper_text_list[count.index])), 2 ) ]
  ttl     = var.ttl
}

Edit: dynamic zone_id based on the first component of sub-dns (IE: foo00 or bar00 or baz00)

@olenm This won't work during a plan when the resource has not been created yet

@finiteinfinity: yes, both count and for_each has the cert-validation components need to be after the cert is created.

• I've re-coded the whole thing utilizing for_each instead of the local helper string+lists
• same dependency errors occurred when using a for_each referencing the aws_acm_certificate (used a variety of try's and empty-maps)
  
  
  
  • workaround https://github.com/hashicorp/terraform/issues/22281#issuecomment-517080564 gave me an "empty" map of { "": "" } at one point due to the try logic.
  
  

My current solution is to have a 2 initial lists, one with N domains, and the 2nd is a list of domains to validate - initially empty. Run TF twice for now.

I'll edit this thread tomorrow with my code and maybe there's still yet a tweak to make it work seamlessly.

Edit 1: I've found it much easier to create aws_route53_record based on its own list of data vs output from aws_acm_certificate. So if you have a dynamic list of DNS's (with or without * certs), use that list to generate another list without the * certs. Then when creating the route53's, use the new-list to query the output from the acm_cert (either via count or for_each) is calculated from a static+known list.

Edit 2: this solution is slightly off-topic to the orig issue now, but long story short, ignore_changes = [ subject_alternative_names ] resolves that

It made more sense to include the cert-resource in this code as I try to utilize for_each vs count in defining resources.

Goal (to re-iterate): create single cert for multiple domains (and * domains), where each domain is its own zone, and utilize DNS auto-cert validation

• note: route53 record could not be directly used with a count/for-each referencing the output of aws_acm_certificate and this has also been addressed. Happy Cert'ing everybody

locals {

  ## fqdn list where each dns is its own zone, and needs a cert for DNS and *.DNS
  fqdn_list = [
    "foo1.bar.com",
    "foo2.bar.com",
    #"foo1.baz.com",
    #"foo2.baz.com",
  ]
  enable_cert_validation    = true

}






#### multi_star_cert start cert

resource "aws_acm_certificate" "multi_star_cert" {
  count = length(local.fqdn_list) > 0 ? 1 : 0

  domain_name       = try(local.fqdn_list[0],"")
  validation_method = "DNS"

  ## list sorting bug/issue: https://github.com/terraform-providers/terraform-provider-aws/issues/8531
  subject_alternative_names = reverse(sort(distinct( concat(
    local.fqdn_list,
    formatlist( "*.%s", local.fqdn_list )
  ) )))


  tags = {
    k1 = "v1"
    k2 = "v2"
  }

  lifecycle {
    create_before_destroy = true
    ignore_changes = [ subject_alternative_names ]
  }
}

output "aws_acm_certificate__multi_star_cert__arn" {
  value = try(join("",aws_acm_certificate.multi_star_cert.*.arn),"")
}

#### end multi_star_cert start cert










#### cert auto verification via DNS 

data "aws_route53_zone" "validation_dns" {
  for_each = try(toset(local.fqdn_list),{})
  name = "${each.value}"
}



resource "aws_route53_record" "multi_star_cert-validation" {
  for_each = toset(local.fqdn_list)

  name    = join("", [for x in try(aws_acm_certificate.multi_star_cert[0].domain_validation_options,list([])): x.resource_record_name if x.domain_name == each.value ])
  type    = join("", [for x in try(aws_acm_certificate.multi_star_cert[0].domain_validation_options,list([])): x.resource_record_type if x.domain_name == each.value ])

  zone_id = data.aws_route53_zone.validation_dns[ each.value ].id

  records = [for x in try(aws_acm_certificate.multi_star_cert[0].domain_validation_options,list([])): x.resource_record_value if x.domain_name == each.value ]

  ttl     = 60

  depends_on = [
    aws_acm_certificate.multi_star_cert[0]
  ]
}


resource "aws_acm_certificate_validation" "multi_star_cert-validation" {
 count = length(local.fqdn_list) > 0 && local.enable_cert_validation == true ? 1 : 0
 certificate_arn         = aws_acm_certificate.multi_star_cert[0].arn
 validation_record_fqdns = [for x in aws_route53_record.multi_star_cert-validation: x.fqdn ]
}

#### END cert auto verification via DNS 

Just to add yet another somewhat ugly and shouldn't-be-necessary solution to the pile this is what we ended up with in 0.12.

The caveat is you need to manually taint the aws_acm_certificate.ssl_cert resource to create a new one when you update the list of SANs. Otherwise it seems to work really well.

locals {
  // Inspiration: https://github.com/terraform-aws-modules/terraform-aws-acm/blob/43a87c2c9cd692516d9d4acdff2fcc9c5cdea21a/main.tf#L2-L6
  // Get distinct list of domains and SANs
  distinct_domain_names = distinct([for s in concat([var.domain_name], var.subject_alternative_names) : replace(s, "*.", "")])

  // Create a map of domain_validation_options for the distinct domain names
  domain_validation_map = {
    for d in local.distinct_domain_names :
    d => [
      for k, v in aws_acm_certificate.ssl_cert.domain_validation_options :
      tomap(v) if d == replace(v.domain_name, "*.", "")
    ]
  }
}

variable "domain_name" {
  description = "The domain name of the cert"
  type        = string
}

variable "subject_alternative_names" {
  description = "A list of domains that should be SANs in the issued certificate"
  type        = list(string)
  default     = []
}

// Using create_before_destroy lets you replace a cert on an existing LB
resource "aws_acm_certificate" "ssl_cert" {
  domain_name               = var.domain_name
  subject_alternative_names = var.subject_alternative_names
  validation_method         = "DNS"

  lifecycle {
    create_before_destroy = true
    ignore_changes        = [subject_alternative_names]
  }
}

// The length(each.value) > 0 is necessary when adding a SAN and creating a new certificate
// allow_overwrite had to be true because the latest AWS provider doesn't wait until the records are removed before trying to create the new ones
resource "aws_route53_record" "cert_validation" {
  for_each = local.domain_validation_map

  name            = length(each.value) > 0 ? each.value[0].resource_record_name : "undefined"
  type            = length(each.value) > 0 ? each.value[0].resource_record_type : "CNAME"
  zone_id         = var.route53_zone
  allow_overwrite = true

  records = [
    length(each.value) > 0 ? each.value[0].resource_record_value : "undefined"
  ]

  # This doesn't appear to always work automatically
  depends_on = [
    aws_acm_certificate.ssl_cert,
  ]
}

EDIT: Fixed a bug with issuing certs for *.example.com

should not need to taint; if you have cert#1 and you want to "add a dns to cert" for any reason, simply create TF resource cert#2 and scale the items vs re-use existing. the DNS validation can be toggled off explicitly and you can avoid targeting explicit items, too.

this problem can be mitigated if we pass the zones explicitly

• modules/acm_certificate/main.tf

resource "aws_acm_certificate" "certificate" {
  domain_name               = var.domain_names[0]
  subject_alternative_names = slice(var.domain_names, 1, length(var.domain_names))
  validation_method         = "DNS"

  lifecycle {
    create_before_destroy = true
    ignore_changes        = [subject_alternative_names]
  }

  tags = {
    ManagedBy = "Terraform"
  }
}

locals {
  validation_options_by_domain_name = {
    for validation_option in aws_acm_certificate.certificate.domain_validation_options :
    validation_option.domain_name => validation_option
  }
}

resource "aws_route53_record" "validation_records" {
  depends_on = [aws_acm_certificate.certificate]
  for_each   = var.zone_ids_by_domain_name

  name    = local.validation_options_by_domain_name[each.key].resource_record_name
  type    = local.validation_options_by_domain_name[each.key].resource_record_type
  zone_id = each.value
  records = [local.validation_options_by_domain_name[each.key].resource_record_value]
  ttl     = 60
}

resource "aws_acm_certificate_validation" "certificate_validation" {
  certificate_arn         = aws_acm_certificate.certificate.arn
  validation_record_fqdns = values(aws_route53_record.validation_records).*.fqdn
}

• modules/acm_certificate/variables.tf

variable "domain_names" {}

variable "zone_ids_by_domain_name" {}

• modules/acm_certificate/output.tf

output "arn" {
  value = aws_acm_certificate.certificate.arn
}

output "certificate" {
  value = aws_acm_certificate.certificate
}

• acm.tf

module "acm_example {
  source = "./modules/acm_certificate"

  domain_names = ["my.example.com", "*.my.example.com", "my.example.co.uk", "*.my.example.co.uk"]
  zone_ids_by_domain_name = {
    "my.example.com" = aws_route53_zone.example_com.id
    "my.example.co.uk" = aws_route53_zone.example_co_uk.id
  }
}

The gist here is that we need to link zones to domain_validation_options. I do this by mapping domain_validation_options to domain names and mapping zone_ids to domain_names explicitly.

This way, no matter how subject_alternative_names changes, or domain_validation_options for this matter, the resources will stay the same. And it will also not have problems with generating resources dynamically at apply time, which is not supported by terraform, because you are explicitly passing the zones that will be used in the certificate validation.

Hi Everyone! :wave:

We recognize the significant friction this issue has caused for the community and want to provide an update about its status.

In order to solve all the issues described, we’ve decided to rewrite the original implementation.

Because of the significant effort needed for a total rewrite, we’ll need to spend some time researching before diving into the implementation.

Once we’ve finished our research, we’ll post an update to the issue detailing the next steps.

We appreciate all the contributions and feedback thus far!

Hi @breathingdust

Thanks for the update here. I'm not sure why this needs a redesign - the specific issue (continual changing SANs) involved would be a List -> Map change in the schema? Changing the parameter type and adding a migration script in the schema would allow this to be fixed in place

Would this this not the correct approach?

Paul

Hi @stack72,

It's less about solving this particular issue, and more that this resource is showing its age and is becoming difficult to work with. The team wants to spend some cycles to make sure implementing this fix (and others) is going to be maintainable and extendable. We will post an update once the research phase is complete.

@breathingdust is it not possible to do the quick patch proposed by @stack72 on an issue that's been open for a year for a fairly critical resource in the provider, and THEN refactor the implementation?

Certs bound to listeners can't be deleted, and targetted applies aren't possible with Terraform Cloud or Enterprise.. so I have to take an outage when this happens, and I assume others do as well

Hi all! 👋

Just wanted to update that we have completed the research phase for the redesign and begun the implementation work.

This research is covered here: #13053.

We appreciate your patience, and hope to have a resolution for this and the other ACM issues soon.

I'm sharing my full solution using Terraform 0.11.x and AWS 1.x based on @jonseymour

The two things that are different in my solution:

• I use a map (<resource_record_name> = <index>) instead of a list of strings (<resource_record_name>!<index>)
• It supports domains in different zones

locals {
  certificate_domains = [
    "*.${local.foo_domain_name}",
    "*.admin.${local.foo_domain_name}",
    "*.${local.bar_domain_name}"
  ]
  certificate_domain_zones = {
    "${local.certificate_domains[0]}" = "${data.aws_route53_zone.foo.id}"
    "${local.certificate_domains[1]}" = "${data.aws_route53_zone.foo.id}"
    "${local.certificate_domains[2]}" = "${data.aws_route53_zone.bar.id}"
  }
}

resource "aws_acm_certificate" "company" {
  domain_name = "${local.certificate_domains[0]}"
  subject_alternative_names = ["${slice(local.certificate_domains, 1, length(local.certificate_domain_zones))}"]
  validation_method = "DNS"

  lifecycle {
    create_before_destroy = true
  }
}

locals {
  # NOTE: Improve on Terraform 0.12
  # Unfortunately there's no way to iterate over domain_validation_options in Terraform 0.11 and AWS 1.x and so we need to
  # create a list from it manually from each element index
  # Reference: https://github.com/terraform-providers/terraform-provider-aws/issues/10997
  unsorted_validation_domains_with_indices = {
    "${aws_acm_certificate.company.domain_validation_options.0.resource_record_name}" = 0
    "${aws_acm_certificate.company.domain_validation_options.1.resource_record_name}" = 1
    "${aws_acm_certificate.company.domain_validation_options.2.resource_record_name}" = 2
  }
  sorted_validation_domains = "${sort(keys(local.unsorted_validation_domains_with_indices))}"
  validation_domain_indices = [
    "${local.unsorted_validation_domains_with_indices[local.sorted_validation_domains[0]]}",
    "${local.unsorted_validation_domains_with_indices[local.sorted_validation_domains[1]]}",
    "${local.unsorted_validation_domains_with_indices[local.sorted_validation_domains[2]]}",
  ]
}

resource "aws_route53_record" "certificate_validation" {
  count = "${length(local.certificate_domains)}"
  name = "${lookup(aws_acm_certificate.company.domain_validation_options[local.validation_domain_indices[count.index]], "resource_record_name")}"
  type = "${lookup(aws_acm_certificate.company.domain_validation_options[local.validation_domain_indices[count.index]], "resource_record_type")}"
  records = ["${lookup(aws_acm_certificate.company.domain_validation_options[local.validation_domain_indices[count.index]], "resource_record_value")}"]
  zone_id = "${local.certificate_domain_zones[lookup(aws_acm_certificate.company.domain_validation_options[local.validation_domain_indices[count.index]], "domain_name")]}"
  ttl = 60
}

resource "aws_acm_certificate_validation" "company" {
  certificate_arn = "${aws_acm_certificate.company.arn}"
  validation_record_fqdns = ["${aws_route53_record.certificate_validation.*.fqdn}"]
}

I only have 2 SANs but they seem to be coming back in alphabetical order each time.
I added sort() so its alphabetical into the resource to match the expected result.
I did have to run apply once to have state updated with the sorted values.

but they seem to be coming back in alphabetical order each time.

that was not my experience. For me the ordering wasn't even consistent across multiple calls.

but they seem to be coming back in alphabetical order each time.

Yes I just tested again, and can confirm that not only are they not in alphabetical order, but the order is not consistent across multiple runs.

We only have 2 SANs ( most certs are one name and we create one for each use) so it was coming back in the same order each time.
When running the plan as part of a larger group they don't keep the order.
I stand corrected.

Hi folks 👋 Thank you for your patience and understanding on this manner. As part of #13053 and the extensive feedback provided in this issue (thank you again!), it was determined that a few changes were in order with the aws_acm_certificate resource, which will likely have implications on your existing configurations:

• Changing subject_alternative_names from an ordered list of strings to an unordered set of strings
• Changing domain_validation_options from an ordered list of objects to an unordered set of objects
• Pre-calculating the keys for those domain_validation_options during Terraform's plan phase, so they can be used in downstream count or (preferably) for_each resource handling

We certainly do not take breaking configuration changes lightly, so we wanted to be sure that if anything was going to cause any sort of effort to change your configurations, that those changes would be worth the effort. We think we hit a good balance here and hope that you find the updated handling to be much more pleasant and work as expected.

Given the above, most environments will now be able to do the following directly:

resource "aws_route53_zone" "example_com" {
  name = "example.com"
}

resource "aws_acm_certificate" "example_com" {
  domain_name               = "example.${aws_route53_zone.example_com.name}"
  subject_alternative_names = [
    "example1.${aws_route53_zone.example_com.name}",
    "example2.${aws_route53_zone.example_com.name}",
    "example3.${aws_route53_zone.example_com.name}",
  ]
  validation_method         = "DNS"
}

resource "aws_route53_record" "example_validation" {
  for_each = {
    for dvo in aws_acm_certificate.example_com.domain_validation_options: dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }
  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = aws_route53_zone.example_com.zone_id
}

resource "aws_acm_certificate_validation" "example_com" {
  certificate_arn         = aws_acm_certificate.example_com.arn
  validation_record_fqdns = [for record in aws_route53_record.example_validation: record.fqdn]
}

These changes will land as part of the Terraform AWS Provider version 3.0.0, next week. I have copied information that will be present in the Version 3 Upgrade Guide below for these changes, that will be fully updated when the release occurs.

If you find unexpected behavior after upgrading to version 3.0.0 when its released, please create a new GitHub issue following the bug template and we will take a fresh look given the new resource handling. Cheers.

subject_alternative_names Changed from List to Set

Previously the subject_alternative_names argument was stored in the Terraform state as an ordered list while the API returned information in an unordered manner. The attribute is now configured as a set instead of a list. Certain Terraform configuration language features distinguish between these two attribute types such as not being able to index a set (e.g. aws_acm_certificate.example.subject_alternative_names[0] is no longer a valid reference). Depending on the implementation details of a particular configuration using subject_alternative_names as a reference, possible solutions include changing references to using for/for_each or using the tolist() function as a temporary workaround to keep the previous behavior until an appropriate configuration (properly using the unordered set) can be determined. Usage questions can be submitted to the community forums.

domain_validation_options Changed from List to Set

Previously, the domain_validation_options attribute was a list type and completely unknown until after an initial terraform apply. This generally required complicated configuration workarounds to properly create DNS validation records since referencing this attribute directly could produce errors similar to the below:

Error: Invalid for_each argument
  on main.tf line 16, in resource "aws_route53_record" "existing":
  16:   for_each = aws_acm_certificate.existing.domain_validation_options
The "for_each" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the for_each depends on.

The domain_validation_options attribute is now a set type and the resource will attempt to populate the information necessary during the planning phase to handle the above situation in most environments without workarounds. This change also prevents Terraform from showing unexpected differences if the API returns the results in varying order.

Configuration references to this attribute will likely require updates since sets cannot be indexed (e.g. domain_validation_options[0] or the older domain_validation_options.0. syntax will return errors). If the domain_validation_options list previously contained only a single element like the two examples just shown, it may be possible to wrap these references using the tolist() function (e.g. tolist(aws_acm_certificate.example.domain_validation_options)[0]) as a quick configuration update, however given the complexity and workarounds required with the previous domain_validation_options attribute implementation, different environments will require different configuration updates and migration steps. Below is a more advanced example. Further questions on potential update steps can be submitted to the community forums.

For example, given this previous configuration using a count based resource approach that may have been used in certain environments:

data "aws_route53_zone" "public_root_domain" {
  name = var.public_root_domain
}
resource "aws_acm_certificate" "existing" {
  domain_name               = "existing.${var.public_root_domain}"
  subject_alternative_names = [
    "existing1.${var.public_root_domain}",
    "existing2.${var.public_root_domain}",
    "existing3.${var.public_root_domain}",
  ]
  validation_method         = "DNS"
}
resource "aws_route53_record" "existing" {
  count = length(aws_acm_certificate.existing.subject_alternative_names) + 1
  allow_overwrite = true
  name            = aws_acm_certificate.existing.domain_validation_options[count.index].resource_record_name
  records         = [aws_acm_certificate.existing.domain_validation_options[count.index].resource_record_value]
  ttl             = 60
  type            = aws_acm_certificate.existing.domain_validation_options[count.index].resource_record_type
  zone_id         = data.aws_route53_zone.public_root_domain.zone_id
}
resource "aws_acm_certificate_validation" "existing" {
  certificate_arn         = aws_acm_certificate.existing.arn
  validation_record_fqdns = aws_route53_record.existing[*].fqdn
}

It will receive errors like the below after upgrading:

Error: Invalid index
  on main.tf line 14, in resource "aws_route53_record" "existing":
  14:   name    = aws_acm_certificate.existing.domain_validation_options[count.index].resource_record_name
    |----------------
    | aws_acm_certificate.existing.domain_validation_options is set of object with 4 elements
    | count.index is 1
This value does not have any indices.

Since the domain_validation_options attribute changed from a list to a set and sets cannot be indexed in Terraform, the recommendation is to update the configuration to use the more stable resource for_each support instead of count. Note the slight change in the validation_record_fqdns syntax as well.

resource "aws_route53_record" "existing" {
  for_each = {
    for dvo in aws_acm_certificate.existing.domain_validation_options: dvo.domain_name => {
      name   = dvo.resource_record_name
      record = dvo.resource_record_value
      type   = dvo.resource_record_type
    }
  }
  allow_overwrite = true
  name            = each.value.name
  records         = [each.value.record]
  ttl             = 60
  type            = each.value.type
  zone_id         = data.aws_route53_zone.public_root_domain.zone_id
}
resource "aws_acm_certificate_validation" "existing" {
  certificate_arn         = aws_acm_certificate.existing.arn
  validation_record_fqdns = [for record in aws_route53_record.existing: record.fqdn]
}

After the configuration has been updated, a plan should no longer error and may look like the following:

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
  - destroy
-/+ destroy and then create replacement
Terraform will perform the following actions:
  # aws_acm_certificate_validation.existing must be replaced
-/+ resource "aws_acm_certificate_validation" "existing" {
        certificate_arn         = "arn:aws:acm:us-east-2:123456789012:certificate/ccbc58e8-061d-4443-9035-d3af0512e863"
      ~ id                      = "2020-07-16 00:01:19 +0000 UTC" -> (known after apply)
      ~ validation_record_fqdns = [
          - "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com",
          - "_812ddf11b781af1eec1643ec58f102d2.existing.example.com",
          - "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com",
          - "_d7112da809a40e848207c04399babcec.existing1.example.com",
        ] -> (known after apply) # forces replacement
    }
  # aws_route53_record.existing will be destroyed
  - resource "aws_route53_record" "existing" {
      - fqdn    = "_812ddf11b781af1eec1643ec58f102d2.existing.example.com" -> null
      - id      = "Z123456789012__812ddf11b781af1eec1643ec58f102d2.existing.example.com._CNAME" -> null
      - name    = "_812ddf11b781af1eec1643ec58f102d2.existing.example.com" -> null
      - records = [
          - "_bdeba72164eec216c55a32374bcceafd.jfrzftwwjs.acm-validations.aws.",
        ] -> null
      - ttl     = 60 -> null
      - type    = "CNAME" -> null
      - zone_id = "Z123456789012" -> null
    }
  # aws_route53_record.existing[1] will be destroyed
  - resource "aws_route53_record" "existing" {
      - fqdn    = "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com" -> null
      - id      = "Z123456789012__40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com._CNAME" -> null
      - name    = "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com" -> null
      - records = [
          - "_638532db1fa6a1b71aaf063c8ea29d52.jfrzftwwjs.acm-validations.aws.",
        ] -> null
      - ttl     = 60 -> null
      - type    = "CNAME" -> null
      - zone_id = "Z123456789012" -> null
    }
  # aws_route53_record.existing[2] will be destroyed
  - resource "aws_route53_record" "existing" {
      - fqdn    = "_d7112da809a40e848207c04399babcec.existing1.example.com" -> null
      - id      = "Z123456789012__d7112da809a40e848207c04399babcec.existing1.example.com._CNAME" -> null
      - name    = "_d7112da809a40e848207c04399babcec.existing1.example.com" -> null
      - records = [
          - "_6e1da5574ab46a6c782ed73438274181.jfrzftwwjs.acm-validations.aws.",
        ] -> null
      - ttl     = 60 -> null
      - type    = "CNAME" -> null
      - zone_id = "Z123456789012" -> null
    }
  # aws_route53_record.existing[3] will be destroyed
  - resource "aws_route53_record" "existing" {
      - fqdn    = "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com" -> null
      - id      = "Z123456789012__8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com._CNAME" -> null
      - name    = "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com" -> null
      - records = [
          - "_a419f8410d2e0720528a96c3506f3841.jfrzftwwjs.acm-validations.aws.",
        ] -> null
      - ttl     = 60 -> null
      - type    = "CNAME" -> null
      - zone_id = "Z123456789012" -> null
    }
  # aws_route53_record.existing["existing.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_812ddf11b781af1eec1643ec58f102d2.existing.example.com"
      + records         = [
          + "_bdeba72164eec216c55a32374bcceafd.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing1.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_d7112da809a40e848207c04399babcec.existing1.example.com"
      + records         = [
          + "_6e1da5574ab46a6c782ed73438274181.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing2.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com"
      + records         = [
          + "_638532db1fa6a1b71aaf063c8ea29d52.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing3.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com"
      + records         = [
          + "_a419f8410d2e0720528a96c3506f3841.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
Plan: 5 to add, 0 to change, 5 to destroy.

Due to the type of configuration change, Terraform does not know that the previous aws_route53_record resources (indexed by number in the existing state) and the new resources (indexed by domain names in the updated configuration) are equivalent. Typically in this situation, the terraform state mv command can be used to reduce the plan to show no changes. This is done by associating the count index (e.g. [1]) with the equivalent domain name index (e.g. ["existing2.example.com"]), making one of the four commands to fix the above example: terraform state mv 'aws_route53_record.existing[1]' 'aws_route53_record.existing["existing2.example.com"]'. It is recommended to use this terraform state mv update process where possible to reduce chances of unexpected behaviors or changes in an environment.

If using terraform state mv to reduce the plan to show no changes, no additional steps are required.

In larger or more complex environments though, this process can be tedius to match the old resource address to the new resource address and run all the necessary terraform state mv commands. Instead, since the aws_route53_record resource implements the allow_overwrite = true argument, it is possible to just remove the old aws_route53_record resources from the Terraform state using the terraform state rm command. In this case, Terraform will leave the existing records in Route 53 and plan to just overwrite the existing validation records with the same exact (previous) values.

-> This guide is showing the simpler terraform state rm option below as a potential shortcut in this specific situation, however in most other cases terraform state mv is required to change from count based resources to for_each based resources and properly match the existing Terraform state to the updated Terraform configuration.

$ terraform state rm aws_route53_record.existing
Removed aws_route53_record.existing[0]
Removed aws_route53_record.existing[1]
Removed aws_route53_record.existing[2]
Removed aws_route53_record.existing[3]
Successfully removed 4 resource instance(s).

Now the Terraform plan will show only the additions of new Route 53 records (which are exactly the same as before the upgrade) and the proposed recreation of the aws_acm_certificate_validation resource. The aws_acm_certificate_validation resource recreation will have no effect as the certificate is already validated and issued.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement
Terraform will perform the following actions:
  # aws_acm_certificate_validation.existing must be replaced
-/+ resource "aws_acm_certificate_validation" "existing" {
        certificate_arn         = "arn:aws:acm:us-east-2:123456789012:certificate/ccbc58e8-061d-4443-9035-d3af0512e863"
      ~ id                      = "2020-07-16 00:01:19 +0000 UTC" -> (known after apply)
      ~ validation_record_fqdns = [
          - "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com",
          - "_812ddf11b781af1eec1643ec58f102d2.existing.example.com",
          - "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com",
          - "_d7112da809a40e848207c04399babcec.existing1.example.com",
        ] -> (known after apply) # forces replacement
    }
  # aws_route53_record.existing["existing.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_812ddf11b781af1eec1643ec58f102d2.existing.example.com"
      + records         = [
          + "_bdeba72164eec216c55a32374bcceafd.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing1.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_d7112da809a40e848207c04399babcec.existing1.example.com"
      + records         = [
          + "_6e1da5574ab46a6c782ed73438274181.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing2.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_40b71647a8d88eb82d53fe988e8a3cc1.existing2.example.com"
      + records         = [
          + "_638532db1fa6a1b71aaf063c8ea29d52.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
  # aws_route53_record.existing["existing3.example.com"] will be created
  + resource "aws_route53_record" "existing" {
      + allow_overwrite = true
      + fqdn            = (known after apply)
      + id              = (known after apply)
      + name            = "_8dc56b6e35f699b8754afcdd79e9748d.existing3.example.com"
      + records         = [
          + "_a419f8410d2e0720528a96c3506f3841.jfrzftwwjs.acm-validations.aws.",
        ]
      + ttl             = 60
      + type            = "CNAME"
      + zone_id         = "Z123456789012"
    }
Plan: 5 to add, 0 to change, 1 to destroy.

Once applied, no differences should be shown and no additional steps should be necessary.

This has been released in version 3.0.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!