Updating Variable Groups from an Azure DevOps pipeline

We often need a permanent data store across Azure DevOps pipelines, for scenarios such as:

• Passing variables from one stage to the next in a multi-stage release pipeline. Any variables defined in a task are only propagated to tasks in the same stage.
• Storing state between pipeline runs, for example a blue/green deployment release pipeline could store the currently deployed “color” in a variable group.
• Passing data between pipelines, for example one build pipeline could create a VM immutable image and store its ID in a variable group used by a pipeline that deploys VMs.

While there is always the option to store the data externally (for example, using az storage entity to interact with an Azure Storage Table), we usually prefer to keep the Azure DevOps environment self-contained. Solutions include using Variable Groups with PowerShell and using Pipeline Artifacts as well as an Azure DevOps extension. Here’s a self-contained solution using Variable Groups and bash scripting.

Note that the REST API command we use deletes any variables defined in the Variable Group that are not passed within the query body, rather than merging definitions. Therefore, you must use a dedicated Variable Group for each instance in which want to assign a variable(s) within a pipeline (i.e. usually for each single variable).

1. Create a Variable Group (in the Azure DevOps project Library tab).
2. Create a dummy variable with any name and value (this is only so that we can save the Variable Group).
3. Save the variable group, and take note of the Variable Group ID (variableGroupId query parameter in the browser URL, when navigating to a variable group).
4. Create a single variable called FOO_VARIABLE_GROUP_ID, replacing FOO with something descriptive and set it to your variable group ID. Don’t define any other variables in the group, as they would be overwritten by the script below.
5. In the Variable Group security settings, grant the role “Project Collection Build Service” Administrator privilege for the variable group.
6. Link the variable group to the pipeline.
7. In the agent job settings, enable “Allow scripts to Access OAuth Token” (unless it is a YAML build, in which case this option is always on).

Now create a YAML task with the definition below, or if editing a release pipeline with the GUI, just paste the curl command into a script task.

- bash: |
     curl -fL -XPUT -H "Authorization:Bearer $(System.AccessToken)" -H "Content-Type:application/json" \
     -d '{
       "id": "$(FOO_VARIABLE_GROUP_ID)",
       "type": "Vsts",
       "name": "Foo",
       "variables": {
         "FOO_VARIABLE_GROUP_ID": {
           "isSecret": false,
           "value": "$(FOO_VARIABLE_GROUP_ID)"
         },
         "FOO": {
           "isSecret": false,
           "value": "$(valueIWantToSet)"
         }
       }
     }' \
     $(System.TeamFoundationCollectionUri)/$(System.TeamProject)/_apis/distributedtask/variablegroups/$(FOO_VARIABLE_GROUP_ID)?api-version=5.0-preview.1
   displayName: 'Set variable in variable group to current foo'

Adapt FOO_VARIABLE_GROUP_ID with the variable you defined previously, and the FOO variable and valueIWantToSet with any variable(s) you want to set.

Note the trick that we store the variable group ID into the variable group itself, rather than hard-coding it into the pipeline definition. This makes the definition more portable.