Azure-docs: Azure DevOps Build Pipeline can't get secrets from Key Vault when secured with vnet and firewall

@aspnet4you Thank you for the feedback.
We are actively investigating and will get back to you soon.

@KalyanChanumolu-MSFT - Thanks for looking into the problem. Key Vault is a critical infrastructure component and by its nature of business we don't want to expose it to untrusted networks. If I had build machine in my vent, it would have been okay but my goal is to use Azure DevOps for CI/CD. I need to vault the keys and secrets, and be able to retrieve them securely.

@aspnet4you , Thank you for sharing the query. Whitelisting the IPs is not an efficient way, but its definitely a workaround. We are working with the Product team to get Azure Dev Ops as a Trusted service for Azure Key Vault and that would be a complete fix in all terms. Do allow us sometime and I will share my next updates with you.

You can add a step in the build definition to whitelist the agent IP address, then remove it from the whitelist at the end of the build. This is not a solution but a workaround until Azure product team adds Azure DevOps as a trusted service. Thanks to @Daniel Mann for providing the idea.

The solution is simple but I was not going to trust ipify.org as REST API endpoint to get my build agent’s ip address. Instead, I created my own (and trusted) service at Azure Function- GetClientIP. DevOps is not my day job and I was having hard time to figure out how to assign and use user defined variables and pass them on to next step/task/stage in pipeline! Microsoft documentation on variables usages was not helping me enough but I figured it out after lot of unsuccessful runs!

See the complete solution at my blog- Azure DevOps Build Pipeline- use keys and secrets from Key Vault.

@aspnet4you, There are talks on getting Azure DevOps as a trusted service. Azure DevOps has a distinct scenario where the identity that accesses Key Vault belongs to the customer, not to Microsoft. This violates scalability and security requirements for “trusted service”.

Right now the only workaround appears to be adding the IP addresses of DevOps machines to AKV firewall. This is a smaller subset than all IPs of Datacenter, and will fit the limit of 127 IP rules.

I am currently working on getting the IPs, and will update the thread shortly.

@aspnet4you , Please find the details about the IPs below:

For whitelisting AzureDevops services -https://docs.microsoft.com/en-us/azure/devops/organizations/security/allow-list-ip-url?view=azure-devops

For whitelisting Hosted Agents - https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#agent-ip-ranges

Hope this helps.

@aspnet4you, Hope this helps. Please feel free to reach out to us in case there are any more queries around this. We are closing this thread for now.

@souravmishra-msft it seams that the whitelisting azure devops services , the ip's in the link are it is not enough .
For example trying to link the variable group from devops to the secrets from keyvault I needed to also add the ip: 52.236.x.x (I intentionally hid the lat 2 bytes). Do you know from where these ip's are? Can you provide the full range of ip's for devops variable groups linkage ?

@aspnet4you , Thank you for sharing the query. Whitelisting the IPs is not an efficient way, but its definitely a workaround. We are working with the Product team to get Azure Dev Ops as a Trusted service for Azure Key Vault and that would be a complete fix in all terms. Do allow us sometime and I will share my next updates with you.

Any update on this feature? Is there any link to track the feature 'Azure Dev Ops as a Trusted service for Azure Key Vault'

Same request for other VNET Service Endpoint PAAS offerings - https://feedback.azure.com/forums/217298-storage/suggestions/35850511-make-azuredevops-a-trusted-microsoft-services-to

@aspnet4you , how is whitelisting the IP alone enough to get Secrets/Keys without a relevant Key Vault Access Policy?

@oviliz - Very good question. Whitelisting of IP is one of the last defense against unauthorized access to vault. Service Principal must be configured with Key Vault Access Policy to list and get keys and secrets. It's the 2nd requirement as stated in my post - Azure DevOps Build Pipeline- use keys and secrets from Key Vault.

Hello @KalyanChanumolu-MSFT any updates? Today I faced same issue. I can't retrieve secret from my KV in Azure Key Vault step in my Release pipeline (Azure DevOps). Links with IP's above are broken... So how I can retrieve secrets in my pipeline? Allow trusted Microsoft services to bypass this firewall? (Yes) - didn't help too.

I'm facing this issue as well and it's clear as mud as to which IPs need to be whitelisted. Plus, this is opening up our vault to services that we don't control so I'm pretty unhappy with this solution.

And by the way, the articles referenced above say the IP ranges can change weekly. Suggesting we update the firewall weekly and even try to keep track of the changes is pretty much ridiculous.

I'm facing the same issue, and whitelisting IPs weekly is not an option at all.

Would it be possible to have a self-hosted agent for the Build Pipeline inside a VM, setup this VM inside a VNet, and then allow it access within the Key Vault? I'm no pro by any means, so how absurd is this approach?

FWIW I was inspired by this solution here: https://www.visualstudiogeeks.com/DevOps/PrivateAzureAseV2AppServiceVstsDeploymentUsingAzureDtl

Add us to the list. How can it be that ADO isn't a trusted service for Key Vaults?

If you search the weekly files for devops or hosted agents, it does not highlight any ips. What ips are you using in https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519??? and if you search by Datacenter there is appservice, devspaces, etc,but what correlates to devops and host agents?

@ShaneBala-keyvault, As requested, adding you to this thread.

Add us to the list. How can it be that ADO isn't a trusted service for Key Vaults?

With you There, Looking to move Many projects to AZ Devops but this is an Issue

+1, this should be fixed asap. Even if secret pipeline variables can be used for releases I would like to have a single source of truth. The secret pipeline variable value can not be viewed again after save so any way you put it the information needs to be stored somewhere else anyway.

SO question about this issue:

https://stackoverflow.com/q/61411653/3850405

+1 this is really crucial. Any comment from Microsoft on that?

@souravmishra-msft Should this be reopened perhaps?

@Ogglas This should definitely be re-opened. They should be listing ips or tagging the service to bypass the Azure Keyvault Firewall. They do it for other services, why not azure devops?

+1, same problem here

+1

also joining in on the pile on.. huge problem for me too

How on earth is this close???? the links links of the "workaround" don't work either :)

@jsburckhardt - Not sure which workaround you are referring to but you may review my blog post,
https://blogs.aspnet4you.com/2019/09/20/azure-devops-build-pipeline-use-keys-and-secrets-from-key-vault/

Do we need to create a new issue for this, or find a way to reopen it, @souravmishra-msft ?

@captainbrown, I have reopened this thread. I have also updated the Program Manager internally once again, since from the time this thread was opened initially the internal thread was created with the Product Team.

I am also assigning this thread to the Program Manager. It might take sometime for them to respond on this thread as I am not sure about the complexity that this change might require form the product side.

I have also bumped up the internal thread that is going on against this github thread and will keep you updated on the progress.

I would also like to state that I have fixed the URLs that was mentioned in one of my comments earlier in this thread. You might want to take a look at those also once in the mean time and let us know if those work for your solution and if not do update us on that too.

@souravmishra-msft thank you very much for reopening this issue. Allowing the ranges for the devops services does not work to allow access to the keyvault (perhaps the ranges are out of date on the website) and the list of possible subnets is entirely too long to add to the network access control on the keyvault.

My team and I will be be eagerly waiting for word from your Program Manager

Thank you for reopening...@captainbrown I saw the same issue where the documentation or the json file with the ip services is definitely out of date/not documented properly.

Agreed, thank you for reopening. This is an issue we'd really like to see addressed as well.

@souravmishra-msft Thank you, I hope this will be prioritized soon.

+1, this is also an issue for us. Not only the Azure DevOps access to Key-vault, but we also need to the DevOps pipeline to generate some files and upload those to a firewall-protected Storage Account, but we face the same issue, that the Azure DevOps is not a trusted service and the workaround would be to whitelist ~250 IP addresses on a weekly basis, so that would be very ugly (plus also not sure if there is any limit as to how many IP addresses can be added to the storage account firewall). It would be great if this can be addressed!

There is currently a workaround for this issue.

You can use a self-hosted agent and add the following IPv4 ranges to the allow-list of the key vault firewall. Details here: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops

(The Azure DevOps team is aware of this issue and work for a permanent fix is currently in progress)

This feature is currently in preview.

Here is another work around...you can setup your own vnet with a scale set.

https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/scale-set-agents?view=azure-devops

Another trick...i don't take ownership of this trick, someone showed me.
If you have an enterprise devops account, there is a way to allow devops to read the keyvault secrets for your library.
If you have a personal account this does not work.

You could open up the firewall to the Azure /11 network, setup the connection from devops to azure with the keyvault. Setup the variable library and add the secret. Once the secrets are added, remove the firewall rule and try to add a secret. The error message will pass you an IP that you can place in the firewall. We have found that it has not changed in a month and we do understand that it can change. At this point, we are looking for workarounds like everyone else. I'm going to do further testing with builds and releases, but this allows the Native Devops GUI to interact with Keyvaults directly.

Again, this does not work with a personal account.

+1 we have similar problem using AzDO to SQL server via Private link/endpoint.

+1 Similar issue here.

Workarounds are provided here. It is not issue with document itself. This is a workaround https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops #please-close you can also post feature request here https://developercommunity.visualstudio.com/spaces/21/index.html

Feature request here:

https://developercommunity.visualstudio.com/idea/1152209/azure-devops-build-pipeline-cant-get-secrets-from-1.html

+1, hit this issue today.

IF Microsoft is going to enable Service Tags for Azure DevOps, can it somehow be used on the Keyvault Firewall? https://devblogs.microsoft.com/devops/new-ip-address-ranges-with-service-tags-for-azure-devops-services/

@JQUINONES82 I'm afraid not for hosted agents. From your link:

The Service Tag does not apply to Microsoft Hosted Agents.

+1, hitting this for the last 2 years

+1, hit this issue today.

This is closed, so how do I get the fix?

Security isn't a feature request, it's a necessity to any cloud component. If this isn't fixed, can it be re-opened? I saw mentioned of something being in preview... I'll take a preview feature. This is going to hold up our deployment as I don't want to spend even more money spinning up VMs in azure to do this...

@jsburckhardt - Not sure which workaround you are referring to but you may review my blog post,
https://blogs.aspnet4you.com/2019/09/20/azure-devops-build-pipeline-use-keys-and-secrets-from-key-vault/

Hi mate, is a good workaround. Unfortunatelly, due to security restrictions. The team doesn't have control of the KV/vNet whitelisted IPs