@DanijelMalik , Thanks for your feedback. We are looking into this query and will update you as soon as possible.

@DanijelMalik , Could please provide us with the powershell output for this error. Which look similar to this,
Note: Please make sure to mask confidential areas like subscription ID or secrets.
{
"status": "Failed",
"error": {
"code": "ApplicationGatewayKeyVaultSecretException",
"message": "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/XXXXXXXXXXXXXXXX/resourceGroups/XXXXXXXXXXXX/providers/Microsoft.Network/applicationGateways/applicationGatewayV2'. See details below:",
"details": [
{
"code": "ApplicationGatewayKeyVaultSecretAccessDenied",
"message": "Access denied for KeyVault Secret 'XXXXXXXXXXXXXXXXX' for Application Gateway '/subscriptions/XXXXXXXXXXX/resourceGroups/XXXXXXXXXXXXX/providers/Microsoft.Network/applicationGateways/applicationGatewayV2'. Make sure that Identity assigned to Application Gateway has access to the KeyVault associated with secret."
}
]
}
}

@DanijelMalik , Just checking whether you got some time to view the previous response.

@DanijelMalik, Since we have not heard back from you we will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly reopen the issue and continue the discussion.

Same issue here.

For me this was resolved by enabling soft deletion on the key vault.

There is an existing issue about this requirement not being documented: #34382

Hi, I'm experiencing the same issue. What seems to be the problem is that although the keyvault allows access from the appGw subnet (via the Firewalls and virtual networks settings blade in the keyvault configuration); and the appGw subnet is being input to New-AzApplicationGateway via the -GatewayIpConfigurations argument the appGw do not seem to be able to access the keyvault (the identity has access rights in the keyvault access policies as well).

When I allow access to the keyvault from "all networks" it seems to work fine.

Perhaps in the appGw creation process the keyvault is being accessed before the appGw is configured with the subnet and the keyvault then sees this access from somewhere else than the subnet?

I'm running the powershell script locally but my IP has access to the key vault as well so that should not be the problem.

I see the same issue

@SubhashVasarapu-MSFT

I have the same issue. I'm using Azure CLI 2.0.76. My application gateway and key vault are in different resource groups in the same subscription. The key vault has soft delete enabled, can be accesses from all networks and has an access policy for the application gateway's assigned user assigned identity with the get secrets permission.

az network application-gateway ssl-cert create -g XXX --gateway-name XXX --name XXX --key-vault-secret-id https://XXX.vault.azure.net/secrets/XXX --debug

results in (abbreviated)

msrest.http_logger : {
  "status": "Failed",
  "error": {
    "code": "ApplicationGatewayKeyVaultSecretException",
    "message": "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. See details below:",
    "details": [
      {
        "code": "ApplicationGatewayKeyVaultSecretAccessDenied",
        "message": "Access denied for KeyVault Secret 'https://XXX.vault.azure.net/secrets/XXX' for Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. Make sure that Identity assigned to Application Gateway has access to the KeyVault associated with secret."
      }
    ]
  }
}
msrest.exceptions : Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. See details below:
cli.azure.cli.core.util : Deployment failed. Correlation ID: 623f5539-b652-49fc-9a29-326bcadaa055. Access denied for KeyVault Secret 'https://XXX.vault.azure.net/secrets/XXX' for Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. Make sure that Identity assigned to Application Gateway has access to the KeyVault associated with secret.
Deployment failed. Correlation ID: 623f5539-b652-49fc-9a29-326bcadaa055. Access denied for KeyVault Secret 'https://XXX.vault.azure.net/secrets/XXX' for Application Gateway '/subscriptions/XXX/resourceGroups/XXX/providers/Microsoft.Network/applicationGateways/XXX'. Make sure that Identity assigned to Application Gateway has access to the KeyVault associated with secret.

If I edit the application gateway HTTP listener in Azure Portal and try to get the SSL certificate from the key vault there, I get this error:

Failed to save configuration changes to application gateway 'XXX'. Error: Invalid value for the identities 'XXX/providers/Microsoft.ManagedIdentity/userAssignedIdentities/XXX'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property.

Hi, I'm experiencing the same issue. What seems to be the problem is that although the keyvault allows access from the appGw subnet (via the Firewalls and virtual networks settings blade in the keyvault configuration); and the appGw subnet is being input to New-AzApplicationGateway via the -GatewayIpConfigurations argument the appGw do not seem to be able to access the keyvault (the identity has access rights in the keyvault access policies as well).

When I allow access to the keyvault from "all networks" it seems to work fine.

Perhaps in the appGw creation process the keyvault is being accessed before the appGw is configured with the subnet and the keyvault then sees this access from somewhere else than the subnet?

I'm running the powershell script locally but my IP has access to the key vault as well so that should not be the problem.

This worked for me too. Only change I made from failure to success was to allow all networks

If I edit the application gateway HTTP listener in Azure Portal and try to get the SSL certificate from the key vault there, I get this error:

Failed to save configuration changes to application gateway 'XXX'. Error: Invalid value for the identities 'XXX/providers/Microsoft.ManagedIdentity/userAssignedIdentities/XXX'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property.

@wolfganggallo I'm getting this exact same error. My azure portal seems to be stuck in an error state now as well and does not let me make any changes. Did you manage to fix this?

Att: @SubhashVasarapu-MSFT -- please reopen.

I"m getting the same issue: "_The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property_", and my -- keyvault in a different resource group and vnet. I think in my case the vnets are not peered, so there is no route between the APIM instance and the keyvault at runtime, but the Azure Portal UI will still list the keyvault as available to use, and allow it to be linked to the listener. I am going to create a temporary vnet peering to see if it resolves the issue so that I may remove the listener. My Appgw is currently broken as a result:

UPDATE Yes, that's the problem. The Azure Portal will show you key vaults that are not actually available at runtime, and saving the listener's settings will break your Appgw. The quick fix is to go to the keyvault and temporarily open it to "all networks / internet" then re-save your listener. What you do then is up to you - copy the cert to a local keyvault, or add a vnet peering, or add the missing subnet. Ultimately the portal should validate the network accessibility between appgw and the kyvault. Bug.

2nd UPDATE Just to be clear: This completely BREAKS the Portal UI. AppGw CANNOT be fixed over the portal. Subsequent saves fail.

I just hit this error as well in Portal UI. Soft delete is enabled. Allowing all networks work-around worked.

Please reopen, being able to use the whitelist is good security.

@SubhashVasarapu-MSFT Please reopen. We are coping with a similar same issue. Portal UI is completely broken.

Every operations fails with the following message:
Set-AzApplicationGateway : Either Data or KeyVaultSecretId must be specified for Certificate '/subscriptions/********-****-****-****-************/resourceGroups/**-***-ApplicationGateway-RG/providers/Microsoft.Network/ applicationGateways/**-***-Shared-WAF/sslCertificates/wildcard2022' in Application Gateway.

@PgInsight @oising Did you only need to set the "Allow All Networks" to get the AppGw working again? We have a problem with the same symptoms, but we have the KeyVault in the same subnet, possibly a different root cause.

I cannot do anything using UI, PowerShell, CLI or Resource Explorer.

@PgInsight @oising Did you only need to set the "Allow All Networks" to get the AppGw working again? We have a problem with the same symptoms, but we have the KeyVault in the same subnet, possibly a different root cause.

I cannot do anything using UI, PowerShell, CLI or Resource Explorer.

Setting all networks Allow on the KeyVault and we also needed to set the Soft Delete flag on the Key Vault using PowerShell cmd.

($resource = Get-AzureRmResource -ResourceId (Get-AzureRmKeyVault -VaultName "YOUR-VAULT-NAME").ResourceId).Properties | Add-Member -MemberType "NoteProperty" -Name "enableSoftDelete" -Value "true"

Set-AzureRmResource -resourceid $resource.ResourceId -Properties $resource.Properties

I had the same issue. However I was able to solve issue by removing the Key Vault certificate using PowerShell and not resource explorer using explorer showed the below error:
{
"error": {
"code": "MissingIdentityIds",
"message": "The identity ids must not be null or empty for 'UserAssigned' identity type."
}

}

used the below sample script to remove certificate and listener and the app gateway went back into working state

delete listener and cert from app gw

$AppGw = Get-AzApplicationGateway -Name "app-gw-ssl-key" -ResourceGroupName "lab"
Remove-AzApplicationGatewayHttpListener -ApplicationGateway $AppGw -Name "https"
Remove-AzApplicationGatewaySslCertificate -ApplicationGateway $AppGW -Name "victor-cer"

save changes

set-azapplicationgateway -ApplicationGateway $AppGW

when the app gateway was out of failed state, I checked the access policy of the key vault and saw that the identity of the app gw was in another category that's not "APPLICATION". I guess this was the cause of the issue.
Added the identity again to the access policy from the key vault setting and it was able to show as "APPLICATION". This solved the issue for me and I was able to add listener with cert from key vault without issues.

@SubhashVasarapu-MSFT This needs to be reopened. The App Gateway should never end up in a disabled/broken state like this -- I've hit the same issue again. How can this be escalated? This is not a documentation issue. This is a product quality issue.

To summarize:

When configuring a listener to use an SSL certificate in a remote keyvault, the portal UI will let you configure the listener with a keyvault that is unavailable at runtime to the appgw due to network restrictions (not open to internet, selected subnets only). After you do this, the App Gateway portal interface is BROKEN for all listeners.

I will be taking this forward to the respective team.

What is the status of this?

We've found one of the causes of this error with the PG. If you had the Retention Period on the KeyVault set to something different than the default 90 days, AppGW was throwing this error message even if you had soft-delete enabled and all-network set as the accessibility. The update is rolling out currently to fix this bug.

When will it be rolling out? I've run into this issue as well and it is really a big issue not being able to configure or make any changes from Portal once you've run into it.

Just hit this today. Found this thread after setting soft delete retention to 30 days and finding you can't change it. Would love this fix to roll out.

I have exactly the same issue even with the default retention period of 90 days.
Wether running Azure CLI, Powershell, TerraForm, or Portal, I always get the same error: "Problem occured while accessing and validating KeyVault Secrets associated with Application Gateway ...".

Triple checked the identity and access policies, all resources in the same resource group in location 'West Europe'

@CMS-seglo so you are telling, that you have:

• [x] soft delete enabled
• [x] retention period set to the default 90 days
• [x] networking set to all-network
• [x] and appgw identity correctly configured

and you are still getting this error?

@mark-szabo I have all of that check boxes checked. And ARM template is throwing me an error:

"Invalid value for the identities '/subscriptions/<sub-id>/resourceGroups/wb-all-rg-commons/providers/Microsoft.ManagedIdentity/userAssignedIdentities/wb-application-gateway-identity'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property."

Property is set like this:

"identity": {
    "type": "UserAssigned",
    "userAssignedIdentities": {
           "[parameters('gatewayIdentityId')]": {
                 "principalId": "[parameters('identityObjectId')]",
                 "clientId": "[parameters('identityClientId')]"
            }
     }
}

Parameters are:

New-AzResourceGroupDeployment -Name "Gateway" `
        -ResourceGroupName "wb-devtest-core" `
        -TemplateFile './arm_templates/appgateway.azrm.json' `
        -TemplateParameterObject @{ `
            certificateSecretUrl    = "https://my-key-vault.vault.azure.net/secrets/some-name/some-id"; `
            certificateName         = "certNamel"; `
            gatewayIdentityId       = "/subscriptions/sub-id/resourceGroups/wb-all-rg-commons/providers/Microsoft.ManagedIdentity/userAssignedIdentities/wb-application-gateway-identity"; `
            identityClientId        = "id-here"; `
            identityObjectId        = "objIdHere";

EDIT:

Even when I changed ARM to it's own ssl cert with raw data. Identity property still throws the same error.

Any solution to that?

@mark-szabo thank you for your reply, I was able to go over the whole setup again with a fresh set of eyes and found one point, where my configuration differed: The KeyVault Firewall was set to "Private endpoint and selected networks", but with "Allow trusted Microsoft services to bypass this firewall?" set to enabled.
When I allowed "All networks" I could successfully create a https listener in the portal with an SSL certificate from KeyVault. So it looks like there is still a problem with the network restrictions.

@CMS-seglo yup, it's a known issue, tracked here: https://github.com/MicrosoftDocs/azure-docs/issues/48866

@kszymanski What did you set for the 'certificateSecretUrl'? 'some-id' in your url must be the 'version' of the certificate object in KeyVault. Unfortunately, this neither returned by 'az keyvault certificate list --vault-name ...' nor by 'Get-AzKeyVaultSecret' or 'Get-AzKeyVaultCertificate' in Powershell, but is displayed in the Portal when you look at the certificate in KeyVault.

I had the same misleading error you mentioned above, because a 'broken' certificate was created. Once I figured out the correct id, it worked.

@kszymanski What did you set for the 'certificateSecretUrl'? 'some-id' in your url must be the 'version' of the certificate object in KeyVault. Unfortunately, this neither returned by 'az keyvault certificate list --vault-name ...' nor by 'Get-AzKeyVaultSecret' or 'Get-AzKeyVaultCertificate' in Powershell, but is displayed in the Portal when you look at the certificate in KeyVault.

I had the same misleading error you mentioned above, because a 'broken' certificate was created. Once I figured out the correct id, it worked.

It is obfuscated here. I'm setting an id from portal, I've tried both Certificate Identifier and Secret Identifier no luck in both cases. Also as mentioned even setting certificate with raw data and password (totally removed Key Vault references) is not changing anything still same error. So I think it might be a problem with Identity assignment rather than key vault cert URL.

I eventually had to manually upload the cert and password instead of using KeyVault because I couldn't get the KeyVault to work. However, if you're doing it from the portal once you get this error, you need to create a fresh Application Gateway as someone else mentioned. You won't be able to adjust your errored gateway for some reason.

So as you can see I'm creating it on and on from ARM template. And making change manually after it is created btw. then in portal I can assign this identity and cert without any problems.

@kylehayes that's not quite correct. Yes, you cannot get it out of the failed state from the portal currently, but you can do that from PowerShell or the CLI.

Eg. delete the failed listener, rule and cert and update the GW.

$AppGw = Get-AzApplicationGateway -Name "<name>" -ResourceGroupName "<rg_name>"
Remove-AzApplicationGatewayHttpListener -ApplicationGateway $AppGw -Name "<listener_name>"
Remove-AzApplicationGatewayRequestRoutingRule -ApplicationGateway $AppGW -Name "<rule_name>"
Remove-AzApplicationGatewaySslCertificate -ApplicationGateway $AppGW -Name "<cert_name>"
Set-AzApplicationGateway -ApplicationGateway $AppGW

I had the same issue with a new KV with a network firewall configured. As a test after I turned off the Firewall it threw the error in the ARM deployment. I too agree that this needs to work as my client wants to lock down the KV.

1. There is still an issue with non-90 days soft delete (retention) period. In this case, the error is "_The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property._" (BTW, there is a typo in this error message.) Workaround is to set KV object property 'softDeleteRetentionInDays' to 90. You can use https://resources.azure.com/ for this.

2. App GW management plane uses some random IPs within Azure DC to access KV. See https://github.com/MicrosoftDocs/azure-docs/issues/48866. As a workaround, you can find the IP from KV audit logs (OMS or storage account) and add it to allowed list instead of allowing all the networks. Unfortunately, there is no guaranty that the IP will stay the same. Also, you have to keep this IP in the list all the time because any change in App GW configuration rebuilds the whole configuration object.

So, to fix "broken" App GW, from portal, you have to:

1. Make sure KV soft delete period is 90 days
2. KV firewall is disabled
3. perform any management action in the portal that will trigger App GW config object update

Unfortunately, you will still have to use PS to cleanup config object from links to certs in the KV as cert objects are not exposed in the portal's UI.

This whole KV/AppGW integration story had to be better tested before being released.

We have encountered the same symptom ("The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property."), our soft delete period is 90 days and firewall open. We have KV in a "persistent" resource group with the application stack in a transient group. Torn the DEV stack down a week ago and tried to stand it back up (via ARM) and process fails on Application Gateway, meaning we can't do any development until this works again.

I am also facing the same issue while deploying from ARM template, I kept soft delete as 90 days and firewall open for all networks, but still facing the same issue .It's working from portal but not from ARM template.
. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property.

This is the 3rd time same appgateway has gone to la-la-land. Cant stop and or update:
{
"error": {
"code": "MissingIdentityIds",
"message": "The identity ids must not be null or empty for 'UserAssigned' identity type."
}
}

To be honest. I gave up on it as it’s unstable.

On Thu, Apr 23, 2020 at 3:46 PM pererap01 notifications@github.com wrote:

This is the 3rd time same appgateway has gone to la-la-land. Cant stop and
or update:
{
"error": {
"code": "MissingIdentityIds",
"message": "The identity ids must not be null or empty for 'UserAssigned'
identity type."
}
}

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/MicrosoftDocs/azure-docs/issues/33157#issuecomment-618624144,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABD32Z2KPUY7KBFNPTCVMA3ROCLJ3ANCNFSM4HXFRP4A
.

we had to stand up another one and use this because the original one is flaky! I have a ticket with MS regarding this issue... let's see if it is gonna get resolved this time?

@SubhashVasarapu-MSFT is there a way to list potential blockers why such inconsistent is happening ? May be the root cause/limitation.

It would be even better if we get what's the current status with the fix.

Like others in this thread, I too got an error:

Failed to save configuration changes to application gateway <REDACTED>. Error: Either Data or KeyVaultSecretId must be specified for Certificate <REDACTED>...

In the key vault's access policy I enabled all key, secret, and certificate permissions for the managed identity used by Azure Gateway. I'm sure that this is overkill but after trying a mix of other permissions I got frustrated and just enabled everything. Afterwards the error went away.

P.S. e2e TLS while using a cloud provisioned cert is a super common scenario. As a newcomer to Azure I'm very disappointed that this is not a polished experience yet.

I have been trying to fetch secrets and certificates from KeyVault onto my cluster via Application Gateway which fails my application since it can't authenticate to KV and that results in a Bad Gateway error.

Connection id "0HM0BI3H5TUJP", Request id "0HM0BI3H5TUJP:00000001": An unhandled exception was thrown by the application.
Connection id "0HM0BI3H5TUNQ", Request id "0HM0BI3H5TUNQ:00000002": An unhandled exception was thrown by the application.

I get these errors on my logs of the pod which is trying to fetch the secret.

My KeyVault has soft delete enabled with default 90 days set as retention period and Networking set to all networks. My App Gateway Identity is also configured correctly onto the Vault. Does anybody know where I'm going wrong?

I have been working with an Engineer from MS
This the summary of what i have found out

1. Delete any certs associate with Appgateway listeners. Basically removing all appgateway installed certs
   to check: PS
   az network application-gateway ssl-cert list -g (resource group) --gateway-name (gateway name)
   if the following shows up
   "keyVaultSecretId": null,
   the cert is installed on appgateway and most be disassociated
2. Once the cert is missing from the appgateway associate the cert from key vault and it should work

I have been working with an Engineer from MS
This the summary of what i have found out

1. Delete any certs associate with Appgateway listeners. Basically removing all appgateway installed certs
   to check: PS
   az network application-gateway ssl-cert list -g (resource group) --gateway-name (gateway name)
   if the following shows up
   "keyVaultSecretId": null,
   the cert is installed on appgateway and most be disassociated
2. Once the cert is missing from the appgateway associate the cert from key vault and it should work

this is not the case in my scenario as the entire Resource Group, application gateway included, has been destroyed and recreated, but this problem continues.

Then my assumption is there is a problem with the Key vault and how it communicates with the apgateway...
Have you taken a look at the resource manager and run a get request- you might get better error handling and or run in debug mode

I'm not touching the Key Vault itself at the moment so MS support can see it in it's current state.

Where is the error message generated from? Is it version 2 with WAF enabled?

Yes, V2 WAF. Error comes out of ARM. Executed from an ADO hosted Windows Agent using AZ CLI

az group deployment create --debug --mode Complete

Note: the ARM template includes the entire stack, VNET, VMs, APG, etc. With Key Vault and other persist in a separate resource group.

Ah ok….

From: Continuous Delivery Automation Framework notifications@github.com
Sent: Monday, June 8, 2020 1:01 PM
To: MicrosoftDocs/azure-docs azure-docs@noreply.github.com
Cc: Perera, Priyantha, Arvato SCS Priyantha.Perera@arvato.com; Comment comment@noreply.github.com
Subject: Re: [MicrosoftDocs/azure-docs] Application Gateway: Integration with Key Vault does not work (#33157)

Yes, V2 WAF. Error comes out of ARM. Executed from an ADO hosted Windows Agent using AZ CLI

az group deployment create --debug --mode Complete

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoftDocs%2Fazure-docs%2Fissues%2F33157%23issuecomment-640855594&data=02%7C01%7C%7C742cf0b3589c42d4021508d80be6b732%7C1ca8bd943c974fc68955bad266b43f0b%7C0%7C0%7C637272432826294946&sdata=qdtJwYk5QV8evtt8Bp%2B1%2Bn7lkz5ebEG%2BCMAfdQVmSS4%3D&reserved=0, or unsubscribehttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAF2ZC6CPCTZUSOKGAT7F523RVU7RBANCNFSM4HXFRP4A&data=02%7C01%7C%7C742cf0b3589c42d4021508d80be6b732%7C1ca8bd943c974fc68955bad266b43f0b%7C0%7C0%7C637272432826304943&sdata=TgPtoOi5neDLlb%2B0hk%2BEULQyZSUFV8fy4deb9RgUsKk%3D&reserved=0.

@pererap01
you are using "--mode Complete" - this potentially removes 'hidden" resources not defined in the template.
Also, user/service principal that runs deployment must have ability to read MSI that will be used WAF to access KV if this MSI is not a part of the deployment.

@gaikovoi , I think that reference was for me. We use the same ARM and execution on 3 environments and they all worked fine. The DEV environment was torn down each Friday and recreate each Monday so the team had a fresh environment to work on through the week (we sometimes did an ad-hoc teardown/standup if one of the DEV made a mess). One week it just didn't come up, although no other aspect of the deployment had changed.

@cdaf there was ARM issue today
Resolved:RCA - Azure Resource Manager - Failures creating or deleting resources

Anyhow, sometimes, ARM deployments are failing without meaningful reason. It is usually fixed by re-runnig AzDO pipeline.

@gaikovoi , yes, ARM can be very temperamental (especially if you try and deploy APIM via ARM), however that is not the case here, it's been retried nearly 30 times (by different people trying to diagnose it)

Hi All. I created an account just to respond to this because it was so frustrating. For anyone who like me, cannot or does not want to set the KeyVault to All Networks...I found a workaround. If you go to Resource Explorer and delete the following it should make your App Gateway happy again.

1. Delete the identity object
2. Delete the certificate that caused the failure under sslCertificates
3. Delete all occurrences of your failed listener
4. PUT your changes

This immediately caused an update of my App Gateway and got it in a healthy provisioning state. Admittedly I was too burnt out/frustrated by the process to try to make the KeyVault integration so I just uploaded the certificate directly to the App Gateway and it worked fine. The KeyVault integration really needs to be fixed by Microsoft...

Anywho, hope that helps someone.

Cheers.

Just to echo all of the above. This issue is definitely still present in Azure and honestly blows my mind.

After adding a managed identity to manage permissions for a wildcard certificate PURCHASED FROM AZURE, then following all of the documented steps to add this certificate to the gateway, an error still occurs indicating that the identity does not have access to the vault. Check both vault and certificate to ensure that it does in fact have access, which it does. Decide to promote the permissions from simple read-only to owner to see if this fixes the issue, and hit an error regarding locks on the wildcard certificate.

So not only can I not add the certificate via the Key Vault method, but now trying to clean up items or change the identity's permissions after trying multiple configurations, I can't even remove the identity from Azure without removing the "Delete" lock on the wildcard certificate the client purchased. Honestly, the simplest method I have found here is to export the certificate, add a password to the .pfx file and finally re-upload it during the creation of the Gateway. This was all done via the GUI, in hopes to produce a correct ARM template to use during provisioning via Terraform. Still broken.

This is seems like a pretty foundational thing that Azure is just happy to ignore.

What an absolute nightmare - I cannot believe that over a year of feedback and issues and this still isn't fixed. Our key vault was open to all networks, soft deletes were enabled and set to 90 days, and the first attempt at linking the two borked the gateway.

This really is a badly broken product. 😠

Same here. How can this be an open issue for 1+year. It's horrific to use App Gateway WAF v2.
When will this be fixed?

Still a thing! In my case it appears to be the keyvault network integration. The portal definitely does not check for that and fails with a random property keys error (not a forbidden error as you would expect). Can anyone confirm if enabling VNET Service endpoints for Microsoft.Keyvault on the gateway subnet would allow this integration to work with the KeyVault FW enabled?

Attempted to "Renew or edit" a pre-existing cert uploaded to GW + Use PAAS KV with FW enabled + IPWhitelist enabled

Invalid value for the identities '/subscriptions/{sub-id}/resourcegroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{uami-name}'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property.

(on same listener above)Attempted to "Renew or edit" a pre-existing cert uploaded to GW + Use PAAS KV with FW DISABLED

• SUCCESS

_On a different listener_
Attempted to "Renew or edit" a pre-existing cert uploaded to GW + Use PAAS KV with FW DISABLED

• SUCCESS

Hi All. I created an account just to respond to this because it was so frustrating. For anyone who like me, cannot or does not want to set the KeyVault to All Networks...I found a workaround. If you go to Resource Explorer and delete the following it should make your App Gateway happy again.

1. Delete the identity object
2. Delete the certificate that caused the failure under sslCertificates
3. Delete all occurrences of your failed listener
4. PUT your changes

This immediately caused an update of my App Gateway and got it in a healthy provisioning state. Admittedly I was too burnt out/frustrated by the process to try to make the KeyVault integration so I just uploaded the certificate directly to the App Gateway and it worked fine. The KeyVault integration really needs to be fixed by Microsoft...

Anywho, hope that helps someone.

Cheers.

In the same situation!. I will add the certificate directly and do not use KeyVault, since I also have to restrict Key Vault to subnets
Cheers ,Rui

Bump! This is still an issue using Terraform as well. If the Firewall is turned on on the KV the AppGW provisioning fails.

Tried a lot of things already:

• On the KV turned on
  
  
  
  • Azure Resource Manager for template deployment
  
  
  • Allow trusted Microsoft services to bypass this firewall?
  
  
• Service Endpoint on the AppGW subnet and adding the network on the KV FW
• Adding the publicIP of the AppGW to the KV FW

...None of them works. The only solution is to disable the firewall on the key vault.

Also having this issue. Please investigate!

Still a thing! In my case it appears to be the keyvault network integration. The portal definitely does not check for that and fails with a random property keys error (not a forbidden error as you would expect). Can anyone confirm if enabling VNET Service endpoints for Microsoft.Keyvault on the gateway subnet would allow this integration to work with the KeyVault FW enabled?

Attempted to "Renew or edit" a pre-existing cert uploaded to GW + Use PAAS KV with FW enabled + IPWhitelist enabled

Invalid value for the identities '/subscriptions/{sub-id}/resourcegroups/{rg-name}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/{uami-name}'. The 'UserAssignedIdentities' property keys should only be empty json objects, null or the resource exisiting property.

(on same listener above)Attempted to "Renew or edit" a pre-existing cert uploaded to GW + Use PAAS KV with FW DISABLED

* SUCCESS

_On a different listener_
Attempted to "Renew or edit" a pre-existing cert uploaded to GW + Use PAAS KV with FW DISABLED

* SUCCESS

Doesn't seem to work for me. I also tried many combinations. Only solution was to entirely disable my KV firewall as well.

This seems to be the case for me also; only fully disabling the keyvault firewall makes this work.

Same issue for me as well... how can this not be resolved yet?

I don't understand - we just ran into this as well. What is the proper configuration here??

After opening a ticket with Azure support, I have received their official answer:
"As explained during our conversation, Application Gateway currently does not support integration with Key Vault if Key Vault is not configured to allow “Public Endpoints (all networks)” access.
We are currently working internally with the necessary teams to support all networking configurations on Key Vault with regards to integration with Application Gateway.

The official documentation on this is forthcoming."

@lonevvolf You can open the keyvault to all networks, bind the SSL Cert with a managed identity to your listeners and then firewall the keyvault (or lock it to a vnet.) Appgw will continue to work with the _cached_ ssl cert, but then when you replace the cert in the keyvault before it expires, appgw won't be able to rollover to the renewed cert. So, you have to be careful about that. But yeah, it's broken.

@Microsoft Can we get an update on this issue, please? It is still not resolved and forces your enterprise Azure customers to have Key Vault exposed to All Networks. It has been over a year since this was reported, plenty of time to put it on the roadmap.

As explained during our conversation, Application Gateway currently does not support integration with Key Vault if Key Vault is not configured to allow “Public Endpoints (all networks)” access.
We are currently working internally with the necessary teams to support all networking configurations on Key Vault with regards to integration with Application Gateway.

Also, this restriction needs to be documented somewhere. Can we get this statement into MSFT documentation somewhere?

For example, this document should have that statement: https://docs.microsoft.com/en-us/azure/application-gateway/key-vault-certs

@oising as you mentioned" You can open the keyvault to all networks, bind the SSL Cert with a managed identity to your listeners and then firewall the keyvault (or lock it to a vnet.) Appgw will continue to work with the cached ssl cert."

Does az cli support this feature? can we create http listener in App gateway with ssl cert (pulling it from key vault certs) using az cli? If so can you please provide az cli command. thanks in advance.

@lonevvolf You can open the keyvault to all networks, bind the SSL Cert with a managed identity to your listeners and then firewall the keyvault (or lock it to a vnet.) Appgw will continue to work with the _cached_ ssl cert, but then when you replace the cert in the keyvault before it expires, appgw won't be able to rollover to the renewed cert. So, you have to be careful about that. But yeah, it's broken.

Hmm. What if you turned the Key Vault firewall off before installing the renewed cert, and then flipping it back on again afterwards? Would the AGW see the change, and grab the new cert quickly enough while the firewall is down? I'm currently working on an Azure Function with a timer to handle renewing the cert in Key Vault using certbot/Let's Encrypt. I should be able to script a few az commands to disable and reenable the Key Vault firewall while the cert is updated in Key Vault.

It's silly that any of this is even necessary. :/

Does anyone know if there is any plan to support FW on key vault -> "Private endpoint and selected networks" ?

Adding another comment to echo other's sentiments. This issue is very frustrating, as exposing all networks for Key Vault is often not a solution for many organizations. I am running into this using Terraform and am currently having to work around it. A fix from @microsoft would be awesome.

Same for me, we can't have an open to internet kv or appgw, and cannot use the feature as much as we may want to. :/

Having the same problem today. :(

Another viable option for people pissed off with the struggle with appgw is to use Azure Frontdoor (it's basically a trimmed down appgw) but it only works on public networks (no vnets) - it can autogenerate public facing SSL certs.

We're actively working on allowing AppGW to work with KeyVaults set to only allow private endpoints and select network access. As per @jmcshane's suggestion, we've updated the docs to include an important note under the "How integration works" section regarding this limitation. We'll update this thread as well as the docs once AppGW is able to support all KeyVault configurations.

@skinlayers that could work -- won't know until you try it though!

I was able to fix this following the next steps:

• Remove the existing ssl-cert profile using az network application-gateway ssl-cert remove
• Disable the firewall temporally and allow access from all networks
• Bind the certificate again from azure key vault (I checked twice that all the service identity permissions were ok).
• Disable access from all networks again

Everything working ok

NOTE: The above workaround is not recommended. Microsoft has alerted us that since this method was used, and the firewall was enabled again, future autoscaling operations on the Application Gateway will fail! This really needs a full fix from Microsoft.