The basic Azure Worker Role consists of a run method, an endless loop, and a sleep statement. Earlier this week, Magnus Martensson walked through implementing a more sophisticated wait object than the generic Thread.Sleep call. Which reminded me of a problem inherent in the basic Microsoft template.

Every exit is a crash.

The basic Worker Role is a while(true) statement that alternates between doing work and sleeping for a period of time. When it's time for Azure to recycle the instance, deploy a new one, scale...what happens to this while(true) statement?

It's killed.

The more critical it was, the higher our chances it was in the work side of the work/sleep loop.

(Ouch.)

The Basic Worker Role

Here is the worker class that Visual Studio generates on when creating a new Worker Role project:

Azure calls the OnStart when it starts, then calls the Run method. This sample will hard crash when Azure scales it out of existence, swaps in new instances, decides it's Windows patch time, lets us press the Stop button, and so on.

Let's see it in action. I've added a DoWork() method that sleeps for 10 seconds to simulate important work being done. I've also added Trace.WriteLine calls to the existing methods and to an override of the OnStop method, so we can see what's happening.

If we run this in the emulator and suspend the worker role in the middle of our important work, it exits right on cue, in the middle of the work.

Diagnostic Trace Output

note: I later added timestamps to the Trace output for readability

If this were a real worker role, we could have been doing just about anything in that step when it was killed. Is our system still in a good state?

A Cancel-able Worker Role

The base class for a WorkerRole is the RoleEntryPoint. As we saw above, it offers an OnStop method that will be called when the instance is suspended. More importantly, though, we are allowed to delay that OnStop method up to 30 seconds to finish up what we are working on.

Note: Early last year (2012) this was extended to 5 minutes, though it's not reflected in the documentation above.

The first change we want to make is to replace the while(true) construct with a method that we can cancel. Using a CancellationTokenSource, we can instead loop while that token is not cancelled.

Replacing the Thread.Sleep with a WaitOne() call will allow us to reduce the time to cancel. Unless it receives a signal (cancellation), the token will wait the specified number of milliseconds before continuing. Moving the WaitOne to the end ensures that if a cancellation is signaled, we won't pick up one last bit of work before exiting.

The other piece of the equation is making the OnStop wait until the we have safely exited the loop. We can achieve this by creating a "Safe to exit" WaitHandle that is only set after successfully exiting the loop. The OnStop will Cancel via the CancellationToken, then wait for the "Safe to exit" token to be set before returning.

Now if we run this like the BasicWorker above, hitting the Suspend button in the middle of the DoWork call, we see the system takes the time to exit out safely:

Diagnostic Trace Output

note: I later added timestamps to the Trace output for readability

We can see the OnStop call come in in the middle of our Starting some Work/Finished some work output, but instead of exiting immediately, the worker calmly finished up it's work and then announced it was ready to exit (by setting the _safeToExitHandle WaitHandle).

Wrapping Up

Letting your application die in the middle of an operation is typically not a good idea.

The example code for this post is available on github at tarwn/AzureWorkerRole_Cancellation. After finishing the code samples above, I went back and add seconds elapsed to the trace output message. I didn't update the code samples above because it would have only served to distract from the real code.

I haven't posted on Azure as much as I probably should have, given how much of my time I spend working with it. Expect to see more posts on this in the upcoming months.