Azure Hosted Services offers several really awesome features over using physical servers or standard VM infrastructure. Two of these are the staged deployment model and management SDK, which includes a powershell module. Using these two features, we are going to build a deployment script that deploys a new set of services (servers) in Azure, using a Virtual IP swap to replace the existing production instances only after the new deployment is fully running.

The goal of this post is to build a powershell script that will:

• Upload a compiled Package to Azure Storage
• Create a new Staging deployment
• Wait for all of the instances of the new deployment to be running
• Promote the new deployment to Production
• Stop the instances of the old production deployment and keep them handy in the Staging slot

The sample project and script are available on github: tarwn/AzureHostedServiceDeploymentSample on github

This script is not intended to be production ready. I have spent no time at all refactoring into readily re-usable methods and do not use it in a production environment myself. It will show you how to use the individual methods and give you the pieces you need to build one that fits your processes.

Initial Steps

If you would like to build a sample project of your own and follow along, here’s the steps you will need to perform first:

1. Create an Azure project in Visual Studio – Create/attach one or more web or worker roles
2. Remove the Diagnostics entry in the web.config or add storage settings
3. In the Project References, select "Microsoft.Web.Infrastructure" and set "Copy Local" to "True"
4. Create a Hosted Service in the Azure Dashboard
5. Create a Storage Account in the Azure Dashboard (pick the same region as prior step)
6. Install the latest Azure SDK + Azure Powershell Module (available in Web Platform Installer)
7. Download your publish settings from https://windows.azure.com/download/publishprofile.aspx

If you know your way around Azure, steps 4-7 are mostly reading xkcd while the installers run.

Create the Deployment Script

Now that we have a project and all the prerequisites out of the way, let’s start building the script. As a reminder, these are the steps we intend to follow:

• Upload a compiled Package to Azure Storage
• Create a new Staging deployment
• Wait for all of the instances of the new deployment to be running
• Promote the new deployment to Production
• Suspend the instances of the old production deployment and keep them handy in the Staging slot

Let’s go!

Connect to Azure

The first thing we need to do is import the Powershell module and use the publish settings to set our subscription.

Import-Module "C:\Program Files (x86)\Microsoft SDKs\Windows Azure\PowerShell\ServiceManagement\Azure\Azure.psd1"

Import-AzurePublishSettingsFile $publishSettingsPath
 
Set-AzureSubscription $subscriptionName -CurrentStorageAccount $storageAccountName

Select-AzureSubscription $subscriptionName -Current

$publishSettingsPath, $subscriptionName, and $storageAccountName are parameters I have passed into my script

We load the Azure module from the Microsoft SDKs folder (this is where it installs from Web PI). We then use the *.publishsettings file to “log in” to the Azure subscription, set the storage account we will be using by default, and set this subscription as the default one for our current powershell session.

Upload a compiled Package to Azure Storage

Now that we have access to Azure, we can move on to upload the package. This package can be generated from Visual Studio by right clicking on the Cloud Project and choosing "Package". In an automated process, we can use MSBuild to create this package before calling this script to upload and deploy it.

$container = Get-AzureStorageContainer -Name $containerName -ErrorAction SilentlyContinue

if(!$container){
    New-AzureStorageContainer -Name $containerName
}

Set-AzureStorageBlobContent -File $packagePath -Container $containerName `
                            -Blob $fullTargetPackageName -Force

$blobInfo = Get-AzureStorageBlob  -Container $containerName -blob $fullTargetPackageName

$packageUri = $blobInfo.ICloudBlob.Uri

$packagePath and $containerName are parameters passed to the script, $fullTargetPackageName is generated with a timestamp.

First we create the container if it doesn't already exist, then we upload the package (without prompting), and once that is complete we capture the blob information and extract the URL for later use in the deployment.

Create a new Staging Deployment

Once we have the package uploaded to blob storage, we are ready to create the new Staging deployment.

$deployment = Get-AzureDeployment -ServiceName $serviceName -Slot Staging `
                                  -ErrorAction SilentlyContinue 

if($deployment.name -ne $null){
    Remove-AzureDeployment -ServiceName $serviceName -Slot Staging -Force
}

New-AzureDeployment -ServiceName $serviceName -Slot Staging -Package $packageUri `
                    -Configuration $configPath -Name $fullTargetDeploymentName `
                    -TreatWarningsAsError

The $servicename, $fulltargetDeploymentName, and $configPath are assumed to have been provided, while the $packageUri was defined in the previous step

Before we can create the new deployment, we check to see if there is already a deployment present in the Staging slot and delete it. We then create the new deployment, using the package we just uploaded and a local configuration (*.cscfg) file.

Wait for all of the instances...

The new deployment has been created and told to start, but it takes time for the individual instances to be provisioned and to go through their start-up sequence.

$statusReady = "ReadyRole"
$statusStopped = "StoppedVM"

function Get-AllInstancesAreStatus($instances, $targetStatus){
    foreach ($instance in $instances)
    {
        if ($instance.InstanceStatus -ne $targetStatus)
        {
            return $false
        }
    }
    return $true
}

# ... ... ...

$deployment = Get-AzureDeployment -ServiceName $serviceName -Slot Staging

$waitTime = [System.Diagnostics.Stopwatch]::StartNew()
while ((Get-AllInstancesAreStatus $deployment.RoleInstanceList $statusReady) -eq $false)
{
    if($waitTime.Elapsed.TotalSeconds -gt $instancePollLimit){
        Throw "$instancePollLimit seconds elapsed without all the instances reaching 'ReadyRun'"
    }

    Start-Sleep -Seconds $instancePollRate

    $deployment = Get-AzureDeployment -ServiceName $serviceName -Slot Staging
}

$serviceName is supplied as a script parameter.

While there are any instances that are not in 'ReadyRun' status, we sleep for $instancepollRate seconds and continue to check again. If more than $instancePollLimit seconds go by while waiting, we'll throw an error that will cause our script to exit.

This poll limit is necessary. In the real world of Azure, you can have azure instances that do not boot for long periods of time. Additional logic has been added in Azure that is supposed to detect VMs not booting and replace them, but no one writes perfect code and I have experienced deployments hung for hours or more due to non-booting instances. We also can break our own code, resulting in rapidly re-booting instances that we would not want to deploy to production.

Promote the new deployment to Production, Suspend the old one

Once the staging deployment is up and running, we can promote it to the Production slot.

Move-AzureDeployment -ServiceName $serviceName

$deployment = Get-AzureDeployment -ServiceName $serviceName -Slot Staging `
                                  -ErrorAction SilentlyContinue

if($deployment.DeploymentName -ne $null){
    Set-AzureDeployment -Status -ServiceName $serviceName -Slot Staging `
                        -NewStatus Suspended
}

Remove-AzureAccount -Name $subscriptionName -Force

$serviceName is a parameter passed to the script

Performing the VIP swap is a simple command and the Powershell cmdlet turns that asynchronous method into a synchronous call for us, like so many of the others. Once the swap is complete, if we have a deployment in the Staging slot (the old Production one), we go ahead and tell it to suspend, but don't wait for the individual instances to stop before exiting.

And we're deployed...

There is a full script available in the github repository here: /scripts/deployHostedService.ps1. It is not clean and pretty, but it does have more output and error handling than the snippets above. Among other things, it does not clean out all those packages it uploads to blob storage and it most definitely should not be blindly pasted and used for your production environment.

While this may not be a production-ready script, it's not far off (and I've used worse). The few cmdlets above should start to show the pattern that Microsoft used with this Powershell library. There are plenty of additional cmdlets to interact with storage services, VMs, affinity groups, HDInsight, Media Services...you name it, it's probably in there.

Writing this post, I am reminded how magical this all is. That sample project was only configured to ask for a single server, but I could just as easily have asked for 4 16-core servers and then added in additional web or worker roles, each with their own servers. And I could have done all of that without changing anything at all about this script and I would have had tons of servers deployed, load balanced, and ready to go with just a minor blip as I swapped them into production. I can remember projects with multi-hour manual deployment processes (and month or more system provisioning times), and we just replaced them with a one page script.

The best part is that, unlike some Microsoft frameworks/packages, this magic doesn't just make a great demo, it also works in real production environments.