Why not _________ tool?

There are a lot of tools out there to handle database deployments, but this is a small project that I am building incrementally as a minimum viable product. Rather than tie up a bunch of time researching and experimenting with database deployment tools early on, I decided to do something simple that would work for the time being and free me up to work on the actual product.

What I want from the deployment is to:

1. Spend as little time on this as possible
2. Work against SQL Azure and a local 2008 R2 Server
3. Call it from TeamCity for a local server or a remote one
4. Produce readable output for TeamCity logs
5. Create the databases and users from the ground up
6. Include randomly generated data
7. Manage scripts for 2 independent databases in the same build
8. Allow real SQL (I'm not scared of SQL and I don't want to learn a code abstraction just to deploy changes)
9. Not worry about rollbacks. I am deploying small changes and if something breaks I'll be charging forward
10. Not expose credentials, as the code will be visible to the public
11. Be replaceable. I might replace it with a tool one day, so keep the deployment logic separate from the application

Looking back at this list and what I eventually created, I probably could have used something like DBDeploy. The scripts I created ended up taking a very similar approach.

The Deployment Scripts

My deployment consists of 4 PowerShell scripts:

• ApplyDatabaseUpdates.ps1 - Responsible for generically applying changes from a folder to a specified database
• UpdateCoreDatabase.ps1 - Responsible for the application's Core Database, calls ApplyDatabaseUpdates
• UpdateSampleDatabase.ps1 - Response for the application's Sample Database, calls ApplyDatabaseUpdates
• RunLocally.ps1 - Executes the two Update scripts against the local database on my development machine(s)

This project has been spread out over 6 months, intermixed with life, other projects, blog posts, etc. Along the way I also upgraded my local development machine to SQL Server 2012 but my main test database server is on 2008 still and my release environment is Azure Database/SQL Azure/(whatever the name is this week).

ApplyDatabaseUpdates.ps1

The purpose of the ApplyDatabaseUpdates script is to apply all of the scripts in a specified folder to the specified server. To do this it creates a tracking table on the target database, then iterates through the contents of the folder, building a script containing any files that were not previously logged in the tracking table.

Core Database Scripts Folder

The deployment script wraps the contents of each script file in an EXECUTE statement, followed by an INSERT to add it to the tracking table for the database.

ApplyDatabaseUpdates.ps1

Note: this has only been run in the context of my personal project. That means don't copy, paste, and run it immediately against your production environment. Running stuff blindly from the internet is known as both a bad idea and a career limiting maneuver.

Update____Database.ps1

My application has two databases which it will access via different accounts. I want the ability to rebuild these databases from scratch as well as manage their credentials from an external system (in this case, TeamCity). If the worst should happen and these databases are compromised or overwritten in some fashion, I want to be able to recreate them with new credentials, account names, the works.

To make life more difficult, many of these commands have to be executed individually in order to work with Azure Databases.

Both scripts detect if their specified database exists and, if not, create them. The UpdateSampleDatabase is capable of recreating the database in Azure, provided some extra options are passed in to it (the Core Database script is missing this bit, unfortunately):

Excerpt from UpdateSampleDatabase.ps1:

They also generate the users specified by the build server (which will also be dynamically added into the relevant web.config files for the website):

Excerpt from UpdateCoreDatabase.ps1:

Once the database and users are created, the SampleDatabase script produces a replacement for one of it's script files that will contain some randomized data. The original file is a placeholder and produces an error if it hasn't been replaced.

Excerpt from UpdateSampleDatabase.ps1:

This generated data is a necessary part of the application that I initially replaced on each deployment, but now only use on new database deployments. Keeping it random and replaceable prevents the application code from making any assumptions about the data in this table.

The final step for both scripts is to run the ApplyDatabaseUpdates function on their respective folders and databases.

RunLocally.sample.ps1

The RunLocally.sample.ps1 script will bring a local development environment all the way up to the latest version without requiring me to type out a long series of arguments and credentials. It is basically just a list of hardcoded variables and then calls to the two Update______Database.ps1 scripts.

RunLocally.sample.ps1:

The reason it is a sample file is because the real one is going to be different for my desktop and laptop and I knew if they were under source control I would constantly be accidentally committing them and having to change back and forth as I switched systems. To use it, I create a copy of the sample file, rename it to RunLocally.ps1 (which is ignored via the .gitignore for the project) and fill in the real values.

Future Plans

I'm not a fan of the libraries that try to abstract away the SQL in SQL deployments. I know SQL and don't need to spend the time learning a library abstraction that, at best, can offer me no better control and ease of use then raw SQL. Many developers feel less than confident in their SQL skills and an abstracted library seems to reduce the need to learn SQL, but in reality the less you know about SQL the harder it will be to determine if the abstraction is doing what you think it is doing, and doing it in a safe and performant manner (and frequently the answer is "it's not").

This set of scripts evolved as a I built the project. In the future I'll probably switch over to something like DbDeploy, as I mentioned earlier. Having the scripts in file system folders makes it easy to glance over the list to see what has changed, makes them easily accessible in my git repositories, can be copied and pasted (or opened and saved) directly from SSMS, and requires no extra tooling.

The scripts above are part of the deployment process for a personal project I have been working on called SQLisHard.com. Launching in the next few weeks (quietly), the site is designed to help people interactively learn or improve their SQL skills and help build the knowledge and confidence that comes from writing and solving real SQL problems.

There are a few ways to do this, but if you have a schema (XSD) for the file, then here are the steps to make the magic happen.

Sample Files

So let's assume for the moment that you have the following sample files:

AwesomeFile.xml

Awesome.xsd

Visual Studio gives us handy intellisense suggestions and warnings when we're writing the schema because we have specified a namespace it knows, but how do we get that usefulness when we're adding more content to our awesome XML file?

More Cowbell

Turns out, adding this functionality is pretty easy. Visual Studio is smart enough to use know schemas, so all we have to do is provide the information that makes our schema known and relevant for our XML file.

For this example I am assuming that the two files are in the same folder. I've also made the file complex enough that the additions we make should work for far more complex setups also.

Here's the steps:

1. Define a target namespace on the schema
2. Define the empty namespace of the schema as this namespace
3. set attributeFormDefault to unqualified so attributes in our XML file won't require namespace declarations
4. Add the namespace declaration to the XML file

Updating our files (and adding comments to reflect the list above), we have:

AwesomeFile.xml

Awesome.xsd

And there we have it.

Results

Now when we start typing in the XML file we will get intellisense suggestions/completion:

We also get warnings when we forget a required attribute:

And when we use the wrong type:

Hopefully the custom color scheme isn't confusing, was feeling too lazy to switch it and switch it back

Part of the way through the show, a conversation took place that I couldn't get out of my head.

Here's the situation:

A business person is asking IT to implement a blog using a specific NoSQL database solution and we push back and try to examine why they want it, and oh by the way we have this great business opportunity for you (around 29:30).

Which reminded me of a related question I heard (I thought in the show, but apparently it must have been earlier in the day):

Why it is OK for the IT person to talk business to the business person, when it's not OK for the business person to talk IT to the IT person.

What bothered me the whole drive home is the built in assumption that the business and IT are separate entities. The panel went deep into value streams and hypotheses-driven development from here, but I think there was something wrong with the root of this story, where we accepted the separation of IT person and business. This may seem overly detail-oriented, but I think far too many IT people see themselves exactly this way.

And it's wrong.

Here's the flaw in this story (and every other place we talk about IT vs the business):

Joe Random Executive doesn't know all the intricacies of accounting, purchasing, warehousing, marketing, sales, law, human resources, or any of a dozen other areas. They will probably know a couple in detail, but they are no more an expert in the rest then they are in IT (and of course we assume they didn't have an IT Background). These are all skills or roles that get casually lumped under the "business" umbrella. Because somehow all areas but IT are business, while only IT is non-business.

Is this ego or laziness? Some of both. Only in the IT department do we somehow think our role gives us a magic ticket to be totally ignorant about the company we work for.

This is absurd.

It makes no more sense then suggesting that accounting is far too intricate to spend time learning how the business around them works. Or that the ever-changing logistics and supply market supersedes understanding the products we are purchasing or shipping. Or that knowing what markets we sell to has any importance to the lawyers. Or that our business growth strategies should take any time away from HR activities around screening and hiring,.

Maybe the whole company is siloed, but seeing it as IT vs The Business is silly (see Gene Kim's point 25 minutes in).

But wait, these are IT ops people talking, surely this is different, right?

I'll give you a hint, Gene Kim talks early on about how The Phoenix Project was influenced by The Goal (Eliyahu Goldratt), whose main character is a plant operations manager. So no, IT Ops, IT QA, IT Technician, you are no more off the hook then the enormous variety of lawyers a business may need to work with over the course of it's life.

There are two types of workers, and they aren't IT and non-IT, they are engaged and not engaged.

Are you doing your job?

Are you doing the job, or just playing with technology?

An engaged worker knows how the company works, at some level. They can discuss the strategies the business is executing, what their company does, and who their customers are. They may not be able to discuss these things with equal facility as the executives or experts in specific departments, but they can have an intelligent conversation about their business.

The not-engaged worker is just there to do a job and go home. They are the protectors of turf, the ones that deny every request they can get away with, or the ones that simply don't care.

So the reason an IT person (even an IT Ops person) can speak to a non-IT worker about the business is because "the business" is the common language they share. IT, accounting, purchasing, human resources, whoever ... when we need to discuss why something is important, we discuss it's impact on the the business, the strategies or targets it effects, etc. In fact, being in IT typically requires a higher level of day-to-day, cross-functional contact then any of the prior examples. More, not less.

Engaging the business has come and gone as part of many buzzy things, and in DevOps it's rising again. Regardless of it's current popularity, if you work for a company, being an IT person doesn't magically separate you from the goals everyone else is working towards. Engage, learn the business, help them succeed, you'll find that you're company is doing some really interesting things and works with some really interesting people. If it's not, find one that resonates with you and learn what they're doing. Either way, you're part of the business, an you chose whether you're competent or not.

Every exit is a crash.

The basic Worker Role is a while(true) statement that alternates between doing work and sleeping for a period of time. When it's time for Azure to recycle the instance, deploy a new one, scale...what happens to this while(true) statement?

It's killed.

The more critical it was, the higher our chances it was in the work side of the work/sleep loop.

(Ouch.)

The Basic Worker Role

Here is the worker class that Visual Studio generates on when creating a new Worker Role project:

Azure calls the OnStart when it starts, then calls the Run method. This sample will hard crash when Azure scales it out of existence, swaps in new instances, decides it's Windows patch time, lets us press the Stop button, and so on.

Let's see it in action. I've added a DoWork() method that sleeps for 10 seconds to simulate important work being done. I've also added Trace.WriteLine calls to the existing methods and to an override of the OnStop method, so we can see what's happening.

If we run this in the emulator and suspend the worker role in the middle of our important work, it exits right on cue, in the middle of the work.

Diagnostic Trace Output

note: I later added timestamps to the Trace output for readability

If this were a real worker role, we could have been doing just about anything in that step when it was killed. Is our system still in a good state?

A Cancel-able Worker Role

The base class for a WorkerRole is the RoleEntryPoint. As we saw above, it offers an OnStop method that will be called when the instance is suspended. More importantly, though, we are allowed to delay that OnStop method up to 30 seconds to finish up what we are working on.

Note: Early last year (2012) this was extended to 5 minutes, though it's not reflected in the documentation above.

The first change we want to make is to replace the while(true) construct with a method that we can cancel. Using a CancellationTokenSource, we can instead loop while that token is not cancelled.

Replacing the Thread.Sleep with a WaitOne() call will allow us to reduce the time to cancel. Unless it receives a signal (cancellation), the token will wait the specified number of milliseconds before continuing. Moving the WaitOne to the end ensures that if a cancellation is signaled, we won't pick up one last bit of work before exiting.

The other piece of the equation is making the OnStop wait until the we have safely exited the loop. We can achieve this by creating a "Safe to exit" WaitHandle that is only set after successfully exiting the loop. The OnStop will Cancel via the CancellationToken, then wait for the "Safe to exit" token to be set before returning.

Now if we run this like the BasicWorker above, hitting the Suspend button in the middle of the DoWork call, we see the system takes the time to exit out safely:

Diagnostic Trace Output

note: I later added timestamps to the Trace output for readability

We can see the OnStop call come in in the middle of our Starting some Work/Finished some work output, but instead of exiting immediately, the worker calmly finished up it's work and then announced it was ready to exit (by setting the _safeToExitHandle WaitHandle).

Wrapping Up

Letting your application die in the middle of an operation is typically not a good idea.

The example code for this post is available on github at tarwn/AzureWorkerRole_Cancellation. After finishing the code samples above, I went back and add seconds elapsed to the trace output message. I didn't update the code samples above because it would have only served to distract from the real code.

I haven't posted on Azure as much as I probably should have, given how much of my time I spend working with it. Expect to see more posts on this in the upcoming months.

Quality

Often in the software world, Quality seems to mean "Does it work or not". Occasionally there is the idea of a Definition of Done, and the definition of Quality includes additional documents, automated tests, formal communications, etc.

Dictionary.com - character with respect to fineness, or grade of excellence
Merriam Webster - degree of excellence, superiority in kind

If Quality is a scale of excellence, what level do we assign to "it works"? What grade do we get for "I didn't find any broken bits"? Why do we keep assuming "it isn't obviously broken" means we have achieved quality?

I ran into this post that examines a number of different definitions of quality and refines them all into:

1. A characteristic of a product or service that bears on its ability to satisfy and exceed* stated or implied customer needs, and thereby provide customer satisfaction.
2. Freedom from deficiencies.

Which is similar to my less formal definition of:

Quality is how well the product does what we said it would, from the customer's perspective

Quality is a scale, not an on/off switch.

• If the product has defects (ie, broken bits), then we have Quality problems.
• If we cut features at the end of the development cycle and don't tell the customer, we have Quality problems.
• If we have a breakdown in communications to the customer while explaining what the product does, we have a Quality problem.
• If the customer has difficulty interacting with the product, we have Quality problems.
• If the product behaves inconsistently when the customer is using it, we have Quality problems.
• If we deviate from our industry standards without informing the customer, we have Quality problems.
• If we deviate from their industry standards without informing them, we have Quality problems.
• If we have to argue with the customer over whether we built what they asked for, we have Quality problems

On a scale of "it destroyed our business" to "it was generations beyond what we could have imagined", "we didn't find any obvious bugs" sits firmly at zero.

Quality Assurance

So after a rather long-winded definition of Quality, what about Quality Assurance?

I'm not a quality professional, but most of what I hear or read seems to be using the definition of Quality Assurance that means we have testers testing our product before delivery. Possibly with some automated tests. Possibly even with some automated tests that can be run by the developers (not to be confused with forms of testing that are the developer's responsibility).

Sound about right?

Except this is Quality Ensurance. Testing the mostly finished product is designed to Ensure our product meets a specific standard of quality.

Assurance is a proactive term, "it will be good" vs Ensure's "we checked at the end and it was good". The purpose of Quality Assurance is to build confidence in the level of Quality (Assure) by driving improvements, so that improved Quality is a natural result. We Assure, are confidant of, better quality by helping it get cooked in from the start.

We are all responsible for Quality, but Quality Assurance is responsible for improving the processes, tooling, environment, and business to increase the Quality of our output. Inspection at the end serves as a good first step, so we can gather data to drive improvements, but if this is all you are getting from your Quality Assurance efforts, then you don't actually have a Quality Assurance effort.

Devlin's Post

So back to Devlin's post. He listed a number of points from the presentation and his thoughts on them. While I didn't attend and am therefore missing some of the context, here are my thoughts.

1. Risk Mitigation is the main job of QA

Absolutely not. The job of QA is to improve our ability to build Quality in as we build the product. This may have the side effect of mitigating risk, but that is a natural outgrowth of building something that engenders greater satisfaction from it's purchaser.

2. Cost of finding bugs is higher as you get later in the deployment cycle, so we should incentivize finding bugs early because we saved that money.

Incentivising is one way to improve the ability to build Quality as close to the source as possible, so it passes the Quality Assurance definition above. However, I think incentivizing drives the wrong behavior. Even the most honest developer could choose to spend their time finding bugs instead of building the product. And at the end of the day, what we really want is fewer bugs created, which incentivizing could prevent us from improving towards.

3. High Quality / Focus on Quality cost more money

Yes, but only in the short term. Producing Quality products is taking the long view, thinking past the immediate contract or deal that is on the table. Measuring the effects can be difficult, but that doesn't mean they are nonexistent. How much would you pay to turn your customers into part of your marketing and sales teams? How much more productive is the team that is proud of what they build? How much less do you spend manually inspecting for Quality at the end of the process when it's built in early on? How much do you save with a better understanding of requirements as your building the product, rather than after you have delivered? What is the value of being able to deliver a product confidently instead of worrying the customer may not accept it?

Technically, firing the quality team will also save you money. So will firing the development team. That doesn't make them good long term strategies.

4. Unit Testing should be a task for each development task

I feel the same as Devlin on this, if you're in an environment that doesn't include it as part of getting a task done, then, yes, it may be helpful to assign each task with a paired "so the unit tests" task until people get used to doing them together. But this isn't an end state, at some point there should be agreement that doing the tests is simply part of calling the work done.

5.There are Different Kinds of testers

Here I am going to have to diverge from Devlin again, but I'm pretty sure it's a context issue. I think the question of whether there are different types of testers depends on whether we are talking about types of tests or areas being tested. A manual tester at the end of our process is going to have very different skills then an experienced security tester, which will in turn be different than the skills of a usability tester.

7. Team Collaboration on testing

Again, this is an example of one type of improvement a Quality Assurance team could make in a business. This one may be more successful than the incentivising one above, but I think it's going to depend on the environment it's implemented in and, as such, isn't something I can comment on.

8. QA is the Whole Team’s responsibility.

I started out agreeing with Devlin, but I'm going to have to revise my opinion. I think the idea of continuous improvement should be built into the whole team, so that everyone takes responsibility for improving their surroundings, processes, and selves. And I think this directly feeds Quality, because the improvements we are making should be improving the quality of things, either as measured by the customer (product quality) or our coworkers, management, or even cleaning staff. So in a perfect world, building in improvements to improve Quality is part of the individual team members responsibilities.

However, I think there should still be someone who has overall responsibility for QA, either in the role of a manager or coach, to ensure that decisions are being made soundly on real data, ensure we aren't over-optimizing one portion of the business at the expense of another, and so on.

Thoughts?

A lot of people in the software world dislike Quality assurance, but I think this is because there are a lot of poor Quality Assurance efforts out there that never get past the manual test stage (and thus aren't actually assuring quality).

Regardless, if "It ain't broke" is the highest measure of quality in your company, it may be time to recalibrate expectations.

When I switch modes and work in TeamCity, I miss having that information available, with no extra steps from me. Despite several searches, though, I was never able to find a plugin that duplicated that behavior I liked in the Jenkins plugin. Turns out that TeamCity makes it pretty easy to roll your own, with just a little bit of powershell and some built-in features.

In this post I am going to cover capturing the warnings from an MSBuild build step, adding that warning count to the main dashboard, adding a statistics chart for the warning count over time, adding a condensed list to the end of the build log, adding the formatted list as a build artifact, and adding a custom report tab to report the warnings for each build.

Because who doesn't need five different ways to see their warnings?

Capturing the Build Warnings

Since I am using MSBuild, the build warnings have a consistent pattern and MSBuild itself has an option to log out to a logger. We can add this attribute in either the build step or the Build Parameters. My preference is using the parameters of the build step in case I have multiple MSBuild calls in the build.

Parameter to add to MSBuild: /l:FileLogger,Microsoft.Build.Engine;logfile=%BuildLogFilename%

Adding the MS Build Parameter

Each time MSBuild runs, it will log it's output to the specified file. We can use powershell to extract the warnings from the output, like so:

Once we have the warnings extracted, we can move on to decide how we want them delivered.

Each section below will continue to add on to this script until it contains all the pieces we need to meet the display goals at the beginning.

Condensed Warning List in Build Log

The powershell script that is extracting warnings will need to run as a build step in the appropriate build configuration. This means that displaying a formatted list of warnings at the end of the build log is as simple as outputting that list from the powershell script we are building.

This will output a section at the bottom of the build log that contains our warnings, like so:

Warnings in the Bottom of a Build Log

Which I suppose is fine, but doesn't really add that much value over the ones listed further up the log by MSBuild itself.

Condensed Warning List in Archived Text File

Now that I have formatted warnings, it's pretty easy to create a file with those warnings and archive it. First I'll update the script to take an output parameter and add some file output:

Then I'll configure the project to capture that output file as an artifact:

Artifact Configuration

Et voila, the file shows up in my archived items:

List of archived items from a run

And I have a clean, archived list of my warnings:

Display of archived text file

But, really, we can do better.

Warning count in build status

Part of the goal was to be able to see the warning count change with no extra work, the best place I can think of to meet this is the final build status on each build.

Before:

Build status on dashboard

TeamCity provides support for setting the build status from a build script. By adding some output to the powershell script, like so:

Each successful build will also display the number of warnings that were captured.

Build status on dashboard, with warnings

Better, but what about historical values? And I still don't like that text file artifact.

Warning Count as a Custom Chart

TeamCity also provides the ability to add custom charts based on either built-in or custom statistics. Custom statistics are reported similar to the build status output above:

Adding a custom chart requires us to dig into the configurations of TeamCity. I'm going to add a chart that will be displayed for any build that provides the warning count number above, so I'll open the [teamCity data dir]/config/main-config.xml file and add the following section:

This will add a chart to the Statistics tab of the build. After a few builds this is what I have:

Build Warning Statistics

It probably would look better if I hadn't built with the same number of warnings each time, but you get the point. The mouse hover works just like the built-in charts, linking to the run status for the individual point.

Ok, getting better, but I think we can take it one step further.

Adding a Custom Build Warnings Tab

So far we have improved methods of seeing the warning count and watching how it changes over time, but the actual list still leaves something to be desired. Luckily, TeamCity supports custom report tabs in the Build Results. This gives us an easily accessible place to put the warnings and, since it uses HTML, better formatting options than the text file.

First I need to update the powershell script to output the HTML file. TeamCity will be picking up an entire folder for the report tab, so I could add some external CSS or image files for my report, but I'll leave that for another day.

I've added HTML output to the script with a hardcoded output location that ensures the report directory exists before writing the index.html page. I've hardcoded this value to reduce the amount of thinking 'll need to do as I add this to other projects (keeps it consistent from output name to artifact setting to report tab configuration).

The next step is to configure the project to capture the folder as an artifact:

Artifact configuration

Then the last step is to modify the TeamCity configuration to recognize that when I output archives like that, I want to treat them as a report. To do this I add the following chunk of XML to my [TeamCity data directory]/config/main-config.xml file (per the documentation link above):

And there we go, the custom report tab is available in the build results:

Build Warnings tab in Run Results

Which takes us from no visibility into our warnings, to five different methods of viewing the information.

Wrap-up

From having to Ctrl+F through the build log all the way to plugin-level output in a few easy steps. After setting this up one time, the only pieces that needed to be repeated for additional builds are the addition of the /logger parameter for MSBuild and the powershell build step to extract the results, and capturing the artifact for the HTML page. All of the output is either built in to the script or applies to the whole server and is displayed whenever the statistics or archive are present in a build.

Here is the finished script:

To recap, we started with some warning messages randomly scattered across the build log. We ended with the warning count automatically showing in the build status on the dashboard, a nice chart of the number over time, and three different ways to view the detailed list. I hope this proves useful to others as well, now I have to go and fix the sample warnings I added before I forget about them.

While I could use the live REST API to verify the general logic worked, I had limited options when it came to the full range of HTTP responses and communication failures. Add in the impact those live API calls would have on my build process performance, the occasional failures of my build due the imperfections of talking to a live service, and the overhead of maintaining separation between my test and live data in that service...what I really had on my hands was the beginning of years of random, painful maintenance.

If only I could mock WebRequest and have it return carefully crafted responses to test my code with, all without ever touching the real network.

MSDN: You should never directly create an instance of the HttpWebResponse class.

Hmm, ok, maybe not.

Despite having done this search before, this time around I uncovered a couple posts that helped me find a solution to this whole mess. Although not before I left some helpful feedback on the MSDN page about the difference between opinion and documentation.

Yep, making friends.

The source code for this post is located on github, with the sample service implementation and test projects: tarwn/TestableHttpWebResponse

Let's Start with the Tests

I've created a sample service implementation with two API call implementations. Each one builds a WebRequest, executes it, and analyzes the response. A retry policy wraps around the request execution, evaluating exceptions to determine whether to retry or map them to a local exception type to be rethrown.

Here is an example of one of those calls and the synchronous method it uses internally:

TestableHttpWebResponse.Sample/SampleService.cs

Testing a method like this typically requires an integration test against the live service. With the provided TestableHttpWebResponse and TestableWebRequest, however, we can set up an expected request and response and verify the service reacts appropriately.

1: Register the TestableWebRequestCreateFactory
WebRequest.Create(uri) uses a factory to produce the relevant WebRequest instance of a Uri, based on the prefix. So first things first, lets register a new prefix and a factory to serve up requests:

TestableWebRequestCreateFactory.GetFactory() exposes a singleton that can be referenced from any of the tests in this class. When the WebRequest object receives a Uri starting with "test://", it will call the associated factory, giving us the opportunity to respond with a Request object of our choosing.

A common base URI will prove helpful as we write the tests:

2: Building a Test

The easiest test to start with is one that will test the "happy path" where our API call receives a 200 Success response.

First we need to set up the request:

Next we need to set up the response the request will return when it is executed:

And then add it to the Factory so it will be available when WebRequest calls it:

Put all of this together and we have:

TestableHttpWebResponse.Sample.Tests/SampleServiceTests.cs

This exercises the entire successful path of the operation without any additional abstractions in our API code or reliance on external communications and services.

Testing Http Status Codes

Another tricky part of testing a service is figuring out how to test HTTP codes other then the success case.

The sample service maps received Protocol Errors (401, 404, etc) to a local exception so the code consuming this library doesn't have to know how to parse WebExceptions.

TestableHttpWebResponse.Sample/SampleService.cs

Exercising the mapping logic is going to require the WebRequest to receive a WebException. Let's make that happen.

Yes, I know a HEAD request will break this, that's why it's called "sample" code

In the first test, we used the EnqueueResponse method of the TestableWebRequest to set up a 200 Success response. It's just as simple to return a 404 Http code with the expected message and request body:

TestableHttpWebResponse.Sample.Tests/SampleServiceTests.cs

Which allows us to create the ExpectedException test:

Testing Other WebExceptions

What about connection failures? Well there is another version of the EnqueueResponse method that allows us to queue up an exception to be returned from the Request, like so:

Just like the previous test, we can use that response to put together a full test

TestableHttpWebResponse.Sample.Tests/SampleServiceTests.cs

Testing the Retry Policy

Retry policies are trickier, in that they need to be able to execute a Request multiple times and receive new responses. By enqueueing (sp?) multiple responses on the request, we can exercise the retry policy:

TestableHttpWebResponse.Sample.Tests/SampleServiceTests.cs

And there we have it, all the various flavors of an HttpWebRequest.

The Testable Classes

The Testable classes are still under construction. At the time of this post they support the functionality above as well as the ability to set and verify Request Headers and write and verify the Request stream contents (upload). Currently the asynchronous methods (BeginGetResponse/EndGetResponse) are not implemented, but I'll be adding those soon along with SampleService calls that exercise those via TPL and async/await logic. I'll also be looking through WebRequest for other properties or methods I haven't imlpemented yet to see what's useful.

Hopefully others will find this useful as well.

Probably the number one failure of system scaling is when people dive right in and start building. No baselines, limited measurements, no analysis, just a hypothesis and a whole lot of late nights tweaking the system. With extra complexity comes extra costs, from the initial development through more expensive maintenance. Scale poorly and not only do we take on those extra complexity costs, but also the more obvious additional costs of the actual implementation (new servers, more resources, etc).

The Somewhat Contrived Example

Recently I've been working on a system to simulate parallelizing workloads, specifically workloads that depend on external resources with rate or load thresholds. Let's use it to look at a somewhat contrived example.

So the backstory is that I have a batch process that is running more and more slowly as we take on larger and more frequent workloads.

I start by testing the batch process locally so I can see how slow it is before I make changes.

1 Client, 60 Requests, 100rpm

Locally it runs pretty quickly, but I'm betting that parallelizing the process will give me a significant increase in speed.

1 Client vs 30 Clients, 60 Requests, 100rpm

Look how well that performance improved, I achieved better than a 6x improvement in performance. My job here is done.

Except when I try to push this to production, I start getting a lot of errors.

30 Clients, 300 Requests, 100rpm - 67% Failure Rate

As it turns out, the external API my process saves the data to has a rate limit. When I exceed the allowable rate, I'm throttled for a short period of time. Any requests I make during that throttle period are returned with errors indicating I'm throttled.

Hmm. Luckily there are a number of common patterns available to retry these types of failures. I'll add an exponential back-off retry pattern so that when I get throttled my service will retry failed requests at slower rates until the service un-throttles me. While I've found plenty of code examples online, none of them seem to have recommendations, so I'll just use one of the sample settings they provide.

30 Clients, 300 Requests, 100rpm, Retry Policy #1 - 7% Failure Rate

Hmm, better. My failure rate has gone way down. What if I tweak the values?

30 Clients, 300 Requests, 100rpm, Retry Policy #2 - 29% Failure Rate

Oh, that was bad, I obviously was on the right track before. What if I just extend the retry amount a bit to try and knock out the last bit of errors.

30 Clients, 300 Requests, 100rpm, Retry Policy #1B - 0% Failure Rate

Ok, perfect. Now I have a system that is more than 6 times faster than the original, can be easily extended by throwing more workers at it, and is actually in a better position to handle occasional slow downs from my 3rd-party service.

Success!

Where I Went Wrong

Ok, so maybe not. Over the course of my little story I went wrong in a number of places. Even though this was a contrived example, I've watched very similar scenarios play out in a number of different organizations with real systems.

The Bottleneck

The first and most critical problem was that I didn't actually locate the bottleneck in my process, I simply tried to do more of the same. The Theory of Constraints tells us that we can improve the rate of a process by identifying and exploiting the constraints.

In this system, the constraint looked like it was the sequential execution of the tasks, but in reality the constraint was the time it took to call the 3rd-party API. Had we identified that bottleneck before starting, we could have approached the problem differently.

Process - Alternative Design

Rather than the parallel complexity, we can modify how the tasks are executed to try and take advantage of knowing where our bottleneck is. If the API allowed us to submit several requests in a batch, this redesign would net us several orders of magnitude improvement. Another option would be to run the results of the local processing into a queue and submit requests from there at a slow trickle, using only a percentage of our API limit so as not to disrupt any other real-time operations or batch processing the system supports. Another option we could take advantage of is not starting any of our expensive 3rd-party communications until we know that the entire job can actually be processed successfully through our local process.

Identifying the constraint unlocks the ability to turn the problem on it's head and achieve higher improvements, typically by orders of magnitude.

The Math Error

I concluded the scenario above by assuming I had found a good solution that also had a lot of headroom. Unfortunately what I actually did was find the ceiling. I have tuned the retry policy to 30 parallel systems, increasing that number could easily destabilize it further and cause more errors or delays. The headroom is, in fact, an illusion.

The Evolving Assumption

Somewhere along the way I found 30 clients to be a great improvement and didn't test other options. Then I made some assumptions about a retry policy. Then I tweaked that retry policy until the error rate disappeared. My assumptions made sense at the time, so I never questioned where they were leading me.

When I found a winning combination for my retry rate, I didn't realize I was missing other, better options:

30 Clients, 300 Requests, 100rpm, Retry Policy #1B vs 4A-C

What's worse, is that trail of assumptions along the way was never re-validated. I concluded with a 6.5x improvement, but is that still accurate?

1-50 Clients, 300 Requests, 100rpm, Retry Policy #1B

When I run the same settings on a range of 1 to 50 clients, we can see that I lost that original 6x improvement along the way. All I have managed to do is add complexity and some very explicit costs for those additional systems.

Note: What happened to retry policy #3? Well originally 1B was actually named 3, then I decided to update the name halfway through the post but was too lazy to update all of the completed images. Oh well.

Scaling the Wrong Thing

This post used fairly small numbers, had I applied a larger workload, higher throughput rates, different throttling windows, the whole problem would have turned out differently.

When we set out to make a system scale, we need to identify the real scenarios we are trying to scale for and the bottlenecks that stand in our way. Blindly performance tuning can look like an improvement, but is really just a poor short term investment that often entrenches the current performance problems even more deeply. There are a lot of questions that should be asked about the intended result, responsiveness fo the system, other operations it has to support while under load, potential overlap of that load, the type of load, etc. The patterns for one system may have relevance for another, but could just as easily be completely incorrect.

How hard it is to scale a system is going to depend on a lot of factors. Getting it wrong just happens to be the easiest option.

Luckily converting the existing Console project to be deployable as an Azure Worker Role project is fairly easy.

For the purposes of this post, I have created a solution with an Existing Console project, an Azure Cloud Project, and an Existing Azure Worker Role project (we have to have at least one to create the Azure Cloud Project, it can be deleted later).

Solution Explorer view of the Sample Solution

If we right click the Azure "Roles" folder, the option to use an existing Worker Role project is not available. Visual Studio enables or disables this option automatically, based on whether we have any assignable projects in the solution that aren't already associated with the Azure project.

"Worker Role Project in Solution" is Disabled

The first thing we want to do is add a RoleEntryPoint to the console application. To keep this simple, I'm going to convert the Program class to serve as the RoleEntryPoint also.

To add the RoleEntryPoint class, I need to first add some azure references. I'm using Azure 1.8, so this means I'll need the following 4 references:

Adding WindowsAzure References

Note: The Storage SDK version number for Azure SDK 1.8 is 2.0. I suspect they chose to do this because they made a number of significant changes in the storage library that were not backwards compatible with 1.7.

Note #2: I noticed that if you add a new worker role through Visual Studio, it uses StorageClient 1.7 still despite this being SDK 1.8. This may be a requirement from the Azure Diagnostics assembly, Diagnostics uses StorageClient and may still be a version behind. I would test this theory, but getting Diagnostics working properly tends to be somewhat painful.

After adding the references, we can then add RoleEntryPoint inheritance to the Program. New Worker Roles have two methods they override from RoleEntryPoint, but the important one is Run (and why this isn't abstract, I couldn't tell you).

The last thing we need to do is add some information to the project file. I can open the file in Visual Studio by right clicking and selecting "Unload Project", then right clicking again and selecting "Edit project name". In the Property Group I need to add a single property named "RoleType" to identify this as a Worker (see the last entry):

Save and reload the project, and now when we right click on the Azure Cloud Project Roles, the option to associate and existing project is available.

"Worker Role Project in Solution" is Available Now

After adding the project, we can now run it either as a local console application or in the computer emulator as an emulated worker role.

Note: Don't forget that the compute emulator must run in elevated administrator mode.

And there you have it, a Console project promoted to Azure Worker Role, preserving any history of commits that were made against the project and still runnable as either a console application or as a worker role.

5:45 PM. After logging into the service provider dashboard, I find that the database services tab is completely non-functional. Attempts to connect to the database from home also fails. Various other troubleshooting directions are followed and also meet in failure.

And then some scheduled jobs in our development and staging environments kick off and also report database timeouts. It's getting increasingly unlikely that this is us, time to contact support.

6.45 PM. Calling support leaves me in a hold queue that is experiencing "longer than normal call volume". Maybe this isn't just me? I check the service status page and everything is green. Still waiting on hold. I check twitter and don't see any mentions.

7:10 PM. Lets check the public forums...ah, there's a few other people with this problem. I add my own personal story so we can help flesh out the picture of what's going on.

7:15 PM. 30 minutes into the hold queue and I just got transferred to a ringing phone. After seeing the forum post, I suspect I'll end up getting a canned "We know there's a problem, we don't know what it is or when it will be fixed, thank you for all your money" speech. So I'm a little surprised when after two minutes of ringing the phone simply goes dead.

By now I've added a few tweets to twitter with the appropriate tags, started looking at the SLA information for the service, and generally just continue to waste my evening in front of the computer instead of spending time with my wife and two year old.

And time passes.

8:10 PM. I've been refreshing the status dashboard every 10 minutes since I first opened it but this is the first change. Red notifications show up, along with hourly messages going back in time to 5:11 PM. I add a note to the forum and a tweet for anyone else waiting for some word from our service provider. The messages are fairly generic and innocuous, admitting that some customers may experience outages with given services and regions. We apologize for the inconvenience.

We couldn't be bothered to provide an update for 3 hours, but we apologize for the inconvenience.

Your business has been down for three hours and there was no official announcement you could point your customers to, but we apologize for the inconvenience.

Admit Your Service Outages?

I would bet we have all worked with companies or products that go to great pains not to announce bugs or outages to their end users. After all, admitting an outage means that the people that didn't get affected will know about it too, and that looks bad.

Maybe we're afraid of upsetting users that weren't online at the time, of losing customers, of warning off potential users, of providing inside information to our competitors, or getting negative press.

Been There...

I once worked with a manager that screened each outage email, broadcasting only the ones that he felt were important enough to be forwarded, which tended to be none of them. Our service levels improved amazingly in the short term. Unfortunately since we weren't keeping our users informed, each outage took longer to correct as we also fielded calls from end users thinking it was just a problem with their system. Which meant our mean time to fix outages went down and affected more people. Which meant that the overall public consensus of our service level got poorer, especially as people realized we were simply not reporting our outages anymore.

We lost credibility. Our service levels were reduced to levels we had worked years to raise them from. The trust our end users extended to us was heavily impacted.

Before that slow slide back into the mud, we had originally climbed out by being open about our outages. Every outage had been announced, often before we had a fix identified, with follow-ups when information was available. Users trusted us more because they could see us jumping on problems, hear us being honest and open with them, and see the problems getting taken care of. We didn't have to tell them how much things were improving, they saw it.

But we stopped admitting our outages, and for a short time we looked even better. Then we looked far, far worse.

Pretending Doesn't Make It So

Hiding bad news to protect your image is a short term loan with major long term costs. Your statistics are broken, so justifying and performing root cause analysis is that much harder. You are limited in what you can ask or discuss with your customers, because each one has to be treated as a potential leak. Your support services are put under a lot of strain and probably built with a higher focus on cost reduction than quality of service.

In short, all of the energy you are spending to look awesome is actually moving you further away and unlikely to be fooling your end users.

Back to Saturday Night

Saturday night, a well known service provider had an outage that shut down our business for 3 hours and 31 minutes.

For 3 hours we were in the dark and then, when an update was provided, it was backdated to look like we had been kept in the loop the whole time. Countless hours were wasted as we and many other customers tried troubleshooting a problem that wasn't in our systems. The support center was overloaded (or just not responding). And at the end of the night a degree of credibility and trust was thrown away.

Someone had an opportunity to shine, and instead chose to make a bad situation that much worse.