Desktop Developer - Author(s): Eli Weinstock-Herman (tarwn)

Intellisense for custom XML in Visual Studio

Eli Weinstock-Herman (tarwn) — Mon, 06 May 2013 13:44:00 +0000

This is one of those things that I only need once every several months and always forget how to do. Working with custom XML in Visual Studio is a whole lot easier with intellisense. Schema errors are highlighted, enumerated values are displayed, and the amount of typing goes from full tag names to just a few characters followed by tab to complete.

There are a few ways to do this, but if you have a schema (XSD) for the file, then here are the steps to make the magic happen.

Sample Files

So let's assume for the moment that you have the following sample files:

AwesomeFile.xml

Code: xml

<?xml version="1.0" encoding="utf-8" ?>
<AwesomeList>
<AwesomeItem AwesomenessFactor="11">
<AwesomeType>Is Awesome</AwesomeType>
</AwesomeItem>
</AwesomeList>

Awesome.xsd

Code: xml

<?xml version="1.0" encoding="utf-8"?>
<xs:schema id="MyAwesomeSchema"
xmlns:xs="http://www.w3.org/2001/XMLSchema">

<xs:simpleType name="AwesomeLevel">
<xs:restriction base="xs:integer">
<xs:enumeration value="1"></xs:enumeration>
<xs:enumeration value="5"></xs:enumeration>
<xs:enumeration value="10"></xs:enumeration>
<xs:enumeration value="11"></xs:enumeration>
</xs:restriction>
</xs:simpleType>

<xs:complexType name="AwesomeElement">
<xs:sequence>
<xs:element name="AwesomeType" type="xs:string"></xs:element>
</xs:sequence>
<xs:attribute name="AwesomenessFactor" type="AwesomeLevel" use="required"></xs:attribute>
</xs:complexType>

<xs:element name="AwesomeList">
<xs:complexType>
<xs:sequence>
<xs:element name="AwesomeItem" type="AwesomeElement" minOccurs="0" maxOccurs="unbounded">
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>

</xs:schema>

Visual Studio gives us handy intellisense suggestions and warnings when we're writing the schema because we have specified a namespace it knows, but how do we get that usefulness when we're adding more content to our awesome XML file?

More Cowbell

Turns out, adding this functionality is pretty easy. Visual Studio is smart enough to use know schemas, so all we have to do is provide the information that makes our schema known and relevant for our XML file.

For this example I am assuming that the two files are in the same folder. I've also made the file complex enough that the additions we make should work for far more complex setups also.

Here's the steps:

Define a target namespace on the schema
Define the empty namespace of the schema as this namespace
set attributeFormDefault to unqualified so attributes in our XML file won't require namespace declarations
Add the namespace declaration to the XML file

Updating our files (and adding comments to reflect the list above), we have:

AwesomeFile.xml

Code: xml

<?xml version="1.0" encoding="utf-8" ?>
<AwesomeList xmlns="my://awesomeness"> 
<AwesomeItem AwesomenessFactor="11">
<AwesomeType>Is Awesome</AwesomeType>
</AwesomeItem>
</AwesomeList>

Awesome.xsd

Code: xml

<?xml version="1.0" encoding="utf-8"?>
<xs:schema id="MyAwesomeSchema"
xmlns:xs="http://www.w3.org/2001/XMLSchema"
targetNamespace="my://awesomeness" 
xmlns="my://awesomeness" 
attributeFormDefault="unqualified" 
elementFormDefault="qualified">

<xs:simpleType name="AwesomeLevel">
<xs:restriction base="xs:integer">
<xs:enumeration value="1"></xs:enumeration>
<xs:enumeration value="5"></xs:enumeration>
<xs:enumeration value="10"></xs:enumeration>
<xs:enumeration value="11"></xs:enumeration>
</xs:restriction>
</xs:simpleType>

<xs:complexType name="AwesomeElement">
<xs:sequence>
<xs:element name="AwesomeType" type="xs:string"></xs:element>
</xs:sequence>
<xs:attribute name="AwesomenessFactor" type="AwesomeLevel" use="required"></xs:attribute>
</xs:complexType>

<xs:element name="AwesomeList">
<xs:complexType>
<xs:sequence>
<xs:element name="AwesomeItem" type="AwesomeElement" minOccurs="0" maxOccurs="unbounded">
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>

</xs:schema>

And there we have it.

Results

Now when we start typing in the XML file we will get intellisense suggestions/completion:

We also get warnings when we forget a required attribute:

And when we use the wrong type:

Hopefully the custom color scheme isn't confusing, was feeling too lazy to switch it and switch it back

Original post blogged on LessThanDot.

Azure Worker Role - Exiting Safely

Eli Weinstock-Herman (tarwn) — Mon, 11 Feb 2013 18:59:00 +0000

The basic Azure Worker Role consists of a run method, an endless loop, and a sleep statement. Earlier this week, Magnus Martensson walked through implementing a more sophisticated wait object than the generic Thread.Sleep call. Which reminded me of a problem inherent in the basic Microsoft template.

Every exit is a crash.

The basic Worker Role is a while(true) statement that alternates between doing work and sleeping for a period of time. When it's time for Azure to recycle the instance, deploy a new one, scale...what happens to this while(true) statement?

It's killed.

The more critical it was, the higher our chances it was in the work side of the work/sleep loop.

(Ouch.)

The Basic Worker Role

Here is the worker class that Visual Studio generates on when creating a new Worker Role project:

Code: csharp

public class WorkerRole : RoleEntryPoint
{
public override void Run()
{
// This is a sample worker implementation. Replace with your logic.
Trace.WriteLine("WorkerRole1 entry point called", "Information");

while (true)
{
Thread.Sleep(10000);
Trace.WriteLine("Working", "Information");
}
}

public override bool OnStart()
{
// Set the maximum number of concurrent connections
ServicePointManager.DefaultConnectionLimit = 12;

// For information on handling configuration changes
// see the MSDN topic at <a href="http://go.microsoft.com/fwlink/?LinkId=166357">http://go.microsoft.com/fwlink/?LinkId=166357</a>.

return base.OnStart();
}
}

Azure calls the OnStart when it starts, then calls the Run method. This sample will hard crash when Azure scales it out of existence, swaps in new instances, decides it's Windows patch time, lets us press the Stop button, and so on.

Let's see it in action. I've added a DoWork() method that sleeps for 10 seconds to simulate important work being done. I've also added Trace.WriteLine calls to the existing methods and to an override of the OnStop method, so we can see what's happening.

Code: csharp

public override void Run()
{
// This is a sample worker implementation. Replace with your logic.
Trace.WriteLine("BasicWorker - Entry point called", "Information");

while (true)
{
Thread.Sleep(10000);
Trace.WriteLine("BasicWorker - Starting some work", "Information");
DoWork();
Trace.WriteLine("BasicWorker - Finished some work", "Information");
}
}

public void DoWork()
{
Thread.Sleep(10000);
}

public override void OnStop()
{
Trace.WriteLine("BasicWorker - OnStop", "Information");
}

If we run this in the emulator and suspend the worker role in the middle of our important work, it exits right on cue, in the middle of the work.

Diagnostic Trace Output

Code: text

Information: 000.1s - BasicWorker - Entry point called
Information: 010.1s - BasicWorker - Starting some work
Information: 020.1s - BasicWorker - Finished some work
Information: 030.1s - BasicWorker - Starting some work
Information: 040.1s - BasicWorker - Finished some work
Information: 050.1s - BasicWorker - Starting some work
Information: 056.1s - BasicWorker - OnStop

note: I later added timestamps to the Trace output for readability

If this were a real worker role, we could have been doing just about anything in that step when it was killed. Is our system still in a good state?

A Cancel-able Worker Role

The base class for a WorkerRole is the RoleEntryPoint. As we saw above, it offers an OnStop method that will be called when the instance is suspended. More importantly, though, we are allowed to delay that OnStop method up to 30 seconds to finish up what we are working on.

Note: Early last year (2012) this was extended to 5 minutes, though it's not reflected in the documentation above.

The first change we want to make is to replace the while(true) construct with a method that we can cancel. Using a CancellationTokenSource, we can instead loop while that token is not cancelled.

Code: csharp

private CancellationTokenSource _cancellationTokenSource;

public override void Run()
{
Trace.WriteLine("SafeWorker - Entry point called", "Information");
_cancellationTokenSource = new CancellationTokenSource();
var token = _cancellationTokenSource.Token;

while (!token.IsCancellationRequested)
{
Trace.WriteLine("SafeWorker - Starting some work", "Information");
DoWork();
Trace.WriteLine("SafeWorker - Finished some work", "Information");
token.WaitHandle.WaitOne(10000); // sleep 10s or exit early if cancellation is signalled
// ...

Replacing the Thread.Sleep with a WaitOne() call will allow us to reduce the time to cancel. Unless it receives a signal (cancellation), the token will wait the specified number of milliseconds before continuing. Moving the WaitOne to the end ensures that if a cancellation is signaled, we won't pick up one last bit of work before exiting.

The other piece of the equation is making the OnStop wait until the we have safely exited the loop. We can achieve this by creating a "Safe to exit" WaitHandle that is only set after successfully exiting the loop. The OnStop will Cancel via the CancellationToken, then wait for the "Safe to exit" token to be set before returning.

Code: csharp

private CancellationTokenSource _cancellationTokenSource;
private ManualResetEvent _safeToExitHandle;

public override void Run()
{
Trace.WriteLine("SafeWorker - Entry point called", "Information");
_cancellationTokenSource = new CancellationTokenSource();
_safeToExitHandle = new ManualResetEvent(false);
var token = _cancellationTokenSource.Token;

while (!token.IsCancellationRequested)
{
Trace.WriteLine("SafeWorker - Starting some work", "Information");
DoWork();
Trace.WriteLine("SafeWorker - Finished some work", "Information");
token.WaitHandle.WaitOne(10000); // sleep 10s or exit early if cancellation is signalled
}

Trace.WriteLine("SafeWorker - Ready to exit", "Information");
_safeToExitHandle.Set(); // cleanly exited the main loop
}

// ...

public override void OnStop()
{
Trace.WriteLine("SafeWorker - OnStop Called", "Information");
_cancellationTokenSource.Cancel();
_safeToExitHandle.WaitOne();
Trace.WriteLine("SafeWorker - OnStop Complete, Exiting Safely", "Information");
}

Now if we run this like the BasicWorker above, hitting the Suspend button in the middle of the DoWork call, we see the system takes the time to exit out safely:

Diagnostic Trace Output

Code: text

Information: 000.0s - SafeWorker - Entry point called
Information: 000.1s - SafeWorker - Starting some work
Information: 010.1s - SafeWorker - Finished some work
Information: 020.1s - SafeWorker - Starting some work
Information: 030.1s - SafeWorker - Finished some work
Information: 040.1s - SafeWorker - Starting some work
Information: 046.4s - SafeWorker - OnStop Called
Information: 050.1s - SafeWorker - Finished some work
Information: 050.1s - SafeWorker - Ready to exit
Information: 050.1s - SafeWorker - OnStop Complete, Exiting Safely

note: I later added timestamps to the Trace output for readability

We can see the OnStop call come in in the middle of our Starting some Work/Finished some work output, but instead of exiting immediately, the worker calmly finished up it's work and then announced it was ready to exit (by setting the _safeToExitHandle WaitHandle).

Wrapping Up

Letting your application die in the middle of an operation is typically not a good idea.

The example code for this post is available on github at tarwn/AzureWorkerRole_Cancellation. After finishing the code samples above, I went back and add seconds elapsed to the trace output message. I didn't update the code samples above because it would have only served to distract from the real code.

I haven't posted on Azure as much as I probably should have, given how much of my time I spend working with it. Expect to see more posts on this in the upcoming months.

Original post blogged on LessThanDot.

Testing the Not-So-Testable HttpWebResponse

Eli Weinstock-Herman (tarwn) — Mon, 10 Dec 2012 14:56:00 +0000

Recently I was working on a library to consume a REST API without exposing any of the specifics to the rest of the application. Implementing a common interface and set of custom exceptions was easy enough, but exercising the internal logic was going to be tough.

While I could use the live REST API to verify the general logic worked, I had limited options when it came to the full range of HTTP responses and communication failures. Add in the impact those live API calls would have on my build process performance, the occasional failures of my build due the imperfections of talking to a live service, and the overhead of maintaining separation between my test and live data in that service...what I really had on my hands was the beginning of years of random, painful maintenance.

If only I could mock WebRequest and have it return carefully crafted responses to test my code with, all without ever touching the real network.

MSDN: You should never directly create an instance of the HttpWebResponse class.

Hmm, ok, maybe not.

Despite having done this search before, this time around I uncovered a couple posts that helped me find a solution to this whole mess. Although not before I left some helpful feedback on the MSDN page about the difference between opinion and documentation.

Yep, making friends.

The source code for this post is located on github, with the sample service implementation and test projects: tarwn/TestableHttpWebResponse

Let's Start with the Tests

I've created a sample service implementation with two API call implementations. Each one builds a WebRequest, executes it, and analyzes the response. A retry policy wraps around the request execution, evaluating exceptions to determine whether to retry or map them to a local exception type to be rethrown.

Here is an example of one of those calls and the synchronous method it uses internally:

TestableHttpWebResponse.Sample/SampleService.cs

Code: csharp

public class SampleService
{
// ...

public ServiceResponse ListRemoteStuff(string operation)
{
var uri = new Uri(_baseUri, operation);
var request = WebRequest.Create(uri);
request.Headers.Add("version", "123-awesome");
return SendRequest(request);
}

// ...

private ServiceResponse SendRequest(WebRequest request)
{
return _retryPolicy.ExecuteAction<ServiceResponse>(() =>
{
try
{
var response = (HttpWebResponse)request.GetResponse();
var reader = new StreamReader(response.GetResponseStream());
var message = reader.ReadToEnd();
return new ServiceResponse() { IsSuccess = true, Message = message };
}
catch (WebException we)
{
throw MappedException(we);
}
});
}

// ...
}

Testing a method like this typically requires an integration test against the live service. With the provided TestableHttpWebResponse and TestableWebRequest, however, we can set up an expected request and response and verify the service reacts appropriately.

1: Register the TestableWebRequestCreateFactory
WebRequest.Create(uri) uses a factory to produce the relevant WebRequest instance of a Uri, based on the prefix. So first things first, lets register a new prefix and a factory to serve up requests:

Code: csharp

[TestFixtureSetUp]
public void TestFixtureSetup()
{
WebRequest.RegisterPrefix("test", TestableWebRequestCreateFactory.GetFactory());
}

TestableWebRequestCreateFactory.GetFactory() exposes a singleton that can be referenced from any of the tests in this class. When the WebRequest object receives a Uri starting with "test://", it will call the associated factory, giving us the opportunity to respond with a Request object of our choosing.

A common base URI will prove helpful as we write the tests:

Code: csharp

public Uri BaseUri { get { return new Uri("test://mydomain.com/api/"); } }

2: Building a Test

The easiest test to start with is one that will test the "happy path" where our API call receives a 200 Success response.

First we need to set up the request:

Code: csharp

var operation = "ListOfStuff";
var expectedRequest = new TestableWebRequest(new Uri(BaseUri, operation));

Next we need to set up the response the request will return when it is executed:

Code: csharp

expectedRequest.EnqueueResponse(HttpStatusCode.OK, "Success", "Even More Success", false);

And then add it to the Factory so it will be available when WebRequest calls it:

Code: csharp

TestableWebRequestCreateFactory.GetFactory().AddRequest(expectedRequest);

Put all of this together and we have:

TestableHttpWebResponse.Sample.Tests/SampleServiceTests.cs

Code: csharp

[Test]
public void ListRemoteStuff_ValidRequest_ReturnsSuccessfulResponse()
{
var operation = "ListOfStuff";
var expectedRequest = new TestableWebRequest(new Uri(BaseUri, operation));
expectedRequest.EnqueueResponse(HttpStatusCode.OK, "Success", "Even More Success", false);
TestableWebRequestCreateFactory.GetFactory().AddRequest(expectedRequest);
var service = new SampleService(BaseUri);

var response = service.ListRemoteStuff(operation);

Assert.IsTrue(response.IsSuccess);
}

This exercises the entire successful path of the operation without any additional abstractions in our API code or reliance on external communications and services.

Testing Http Status Codes

Another tricky part of testing a service is figuring out how to test HTTP codes other then the success case.

The sample service maps received Protocol Errors (401, 404, etc) to a local exception so the code consuming this library doesn't have to know how to parse WebExceptions.

TestableHttpWebResponse.Sample/SampleService.cs

Code: csharp

private Exception MappedException(WebException we)
{
// map to custom exceptions
if (we.Status == WebExceptionStatus.ProtocolError)
{
var reader = new StreamReader(we.Response.GetResponseStream());
var message = reader.ReadToEnd();
var httpResponse = (HttpWebResponse)we.Response;
switch (httpResponse.StatusCode)
{
case HttpStatusCode.NotFound:
if (httpResponse.StatusDescription.Contains("Dohicky"))
return new DohickyNotFoundException(message, we);
else
return new GenericNotFoundException(message, we);
default:
return new ExampleOfAnotherUsefulException(message, we);
}
}
else
return new SampleServiceOutageException(we);
}

Exercising the mapping logic is going to require the WebRequest to receive a WebException. Let's make that happen.

Yes, I know a HEAD request will break this, that's why it's called "sample" code

In the first test, we used the EnqueueResponse method of the TestableWebRequest to set up a 200 Success response. It's just as simple to return a 404 Http code with the expected message and request body:

TestableHttpWebResponse.Sample.Tests/SampleServiceTests.cs

Code: csharp

expectedRequest.EnqueueResponse(HttpStatusCode.NotFound, "Dohicky not found", "I couldn't find your dohicky because I don't like you", true);

Which allows us to create the ExpectedException test:

Code: csharp

[Test]
[ExpectedException(typeof(DohickyNotFoundException))]
public void ListRemoteStuff_404DohickeyNotFound_ThrowsDohickeyNotFoundException()
{
var operation = "ListOfStuff";
var expectedRequest = new TestableWebRequest(new Uri(BaseUri, operation));
expectedRequest.EnqueueResponse(HttpStatusCode.NotFound, "Dohicky not found", "I couldn't find your dohicky because I don't like you", true);
TestableWebRequestCreateFactory.GetFactory().AddRequest(expectedRequest);
var service = new SampleService(BaseUri);

var response = service.ListRemoteStuff(operation);

// expect exception
}

Testing Other WebExceptions

What about connection failures? Well there is another version of the EnqueueResponse method that allows us to queue up an exception to be returned from the Request, like so:

Code: csharp

expectedRequest.EnqueueResponse(new WebException("I'm broke!", WebExceptionStatus.ConnectFailure));

Just like the previous test, we can use that response to put together a full test

TestableHttpWebResponse.Sample.Tests/SampleServiceTests.cs

Code: csharp

[Test]
[ExpectedException(typeof(SampleServiceOutageException))]
public void ListRemoteStuff_ServiceOutage_ThrowsSampleServiceOutage()
{
var operation = "ListOfStuff";
var expectedRequest = new TestableWebRequest(new Uri(BaseUri, operation));
expectedRequest.EnqueueResponse(new WebException("I'm broke!", WebExceptionStatus.ConnectFailure));
TestableWebRequestCreateFactory.GetFactory().AddRequest(expectedRequest);
var service = new SampleService(BaseUri);

var response = service.ListRemoteStuff(operation);

// expect exception
}

Testing the Retry Policy

Retry policies are trickier, in that they need to be able to execute a Request multiple times and receive new responses. By enqueueing (sp?) multiple responses on the request, we can exercise the retry policy:

TestableHttpWebResponse.Sample.Tests/SampleServiceTests.cs

Code: csharp

[Test]
public void ListRemoteStuff_TimeoutOccurs_TruesASecondTime()
{
var operation = "ListOfStuff";
var expectedRequest = new TestableWebRequest(new Uri(BaseUri, operation));
expectedRequest.EnqueueResponse(new TimeoutException("took too long, so sorry"))
.EnqueueResponse(HttpStatusCode.OK, "All Good", "Nothing to see, please move along", false);
TestableWebRequestCreateFactory.GetFactory().AddRequest(expectedRequest);
var service = new SampleService(BaseUri);

var response = service.ListRemoteStuff(operation);

Assert.AreEqual("Nothing to see, please move along", response.Message);
}

And there we have it, all the various flavors of an HttpWebRequest.

The Testable Classes

The Testable classes are still under construction. At the time of this post they support the functionality above as well as the ability to set and verify Request Headers and write and verify the Request stream contents (upload). Currently the asynchronous methods (BeginGetResponse/EndGetResponse) are not implemented, but I'll be adding those soon along with SampleService calls that exercise those via TPL and async/await logic. I'll also be looking through WebRequest for other properties or methods I haven't imlpemented yet to see what's useful.

Hopefully others will find this useful as well.

Original post blogged on LessThanDot.

Converting a .Net Console Project to an Azure Worker Role Project

Eli Weinstock-Herman (tarwn) — Wed, 07 Nov 2012 11:28:00 +0000

So lets say you're building a worker role for Azure, but started it out as a Console app for faster local debugging. It comes time to deploy it to Azure for the first time, but there's no option to convert a Console application to a Worker Role project. We could copy all the code into a new project, but then we not only lose our faster debugging but also the history of commits we have made against the project.

Luckily converting the existing Console project to be deployable as an Azure Worker Role project is fairly easy.

For the purposes of this post, I have created a solution with an Existing Console project, an Azure Cloud Project, and an Existing Azure Worker Role project (we have to have at least one to create the Azure Cloud Project, it can be deleted later).

Solution Explorer view of the Sample Solution

If we right click the Azure "Roles" folder, the option to use an existing Worker Role project is not available. Visual Studio enables or disables this option automatically, based on whether we have any assignable projects in the solution that aren't already associated with the Azure project.

"Worker Role Project in Solution" is Disabled

The first thing we want to do is add a RoleEntryPoint to the console application. To keep this simple, I'm going to convert the Program class to serve as the RoleEntryPoint also.

To add the RoleEntryPoint class, I need to first add some azure references. I'm using Azure 1.8, so this means I'll need the following 4 references:

Adding WindowsAzure References

Note: The Storage SDK version number for Azure SDK 1.8 is 2.0. I suspect they chose to do this because they made a number of significant changes in the storage library that were not backwards compatible with 1.7.

Note #2: I noticed that if you add a new worker role through Visual Studio, it uses StorageClient 1.7 still despite this being SDK 1.8. This may be a requirement from the Azure Diagnostics assembly, Diagnostics uses StorageClient and may still be a version behind. I would test this theory, but getting Diagnostics working properly tends to be somewhat painful.

After adding the references, we can then add RoleEntryPoint inheritance to the Program. New Worker Roles have two methods they override from RoleEntryPoint, but the important one is Run (and why this isn't abstract, I couldn't tell you).

Code: csharp

namespace ExistingConsoleApplication
{
public class Program : RoleEntryPoint
{
static void Main(string[] args) /* Console entry point */
{
var ibl = new InterestingBusinessLogic();
ibl.DoStuff();

Console.Read();
}

public override void Run() /* Worker Role entry point */
{
var ibl = new InterestingBusinessLogic();
ibl.DoStuff();
}
}
}

The last thing we need to do is add some information to the project file. I can open the file in Visual Studio by right clicking and selecting "Unload Project", then right clicking again and selecting "Edit project name". In the Property Group I need to add a single property named "RoleType" to identify this as a Worker (see the last entry):

Code: xml

<PropertyGroup>
<Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
<Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
<ProjectGuid>{4B7C48CB-9899-47D0-95AB-2FD3C3B739CF}</ProjectGuid>
<OutputType>Exe</OutputType>
<AppDesignerFolder>Properties</AppDesignerFolder>
<RootNamespace>ExistingConsoleApplication</RootNamespace>
<AssemblyName>ExistingConsoleApplication</AssemblyName>
<TargetFrameworkVersion>v4.5</TargetFrameworkVersion>
<FileAlignment>512</FileAlignment>
<RoleType>Worker</RoleType>
</PropertyGroup>

Save and reload the project, and now when we right click on the Azure Cloud Project Roles, the option to associate and existing project is available.

"Worker Role Project in Solution" is Available Now

After adding the project, we can now run it either as a local console application or in the computer emulator as an emulated worker role.

Note: Don't forget that the compute emulator must run in elevated administrator mode.

And there you have it, a Console project promoted to Azure Worker Role, preserving any history of commits that were made against the project and still runnable as either a console application or as a worker role.

Original post blogged on LessThanDot.

Pick the Right Storage: All SQL is Not Equal

Eli Weinstock-Herman (tarwn) — Tue, 26 Jun 2012 10:38:00 +0000

As part of a long series of posts, I implemented a version of the MVC Music Store tutorial application on top of a pair of SQL Server CE databases. SQL Server CE is great for small apps, being a portable file-based database that can easily be moved to a full SQL Server instance. Last week I migrated my application to use full SQL Server instances instead of the SDF file and picked up a 3x performance improvement. It was interesting enough that I decided to share

Note: I wasn't purposefully performance tuning my sample site. This move was a necessary step to prep for doing some other SQL Server related things with the project and the performance improvement was just a neat side effect.

The Results

The graph below shows the results of the load test stage of my build over several builds.

We start with the Rate graph. On the left we see a rate of about 30 requests/second from when the system was still on the SQL Server CE file and the build and beta VMs resided on their original home (i7 920, VMWare, using Hyper-V equivalent of 2 logical CPUs). In the middle we have the move to their new home (i7 3770K, Hyper-V 2012) still on the SQL Server CE file and switching from one to two virtual CPUs assigned to the beta web server. The last third is the results of moving the primary database to a SQL Server VM, which improved the performance by roughly 3x and appears to have found a new performance constraint.

The transaction rate has gone from about 1.8 to about 4.8 per second.

We can see the response times have been fairly stable, up to the 99% mark, with the max being a little wonky. The 95% and average marks have gone from averaging 650ms and 250ms on the original server with CE to 450ms and 80ms respectively on SQL Server.

The Details

The data for these results are generated automatically as part of the build process for my application. I have 3 dedicated VMs for the build server, a web server, and a database server and they are all currently housed on a 2012 Hyper-V server. Each is assigned about 2GB of RAM and 2, 1, 1 CPUs for the web, build, and DB server respectively. The host has nothing else running on it and is an i7 3770K w/ 16GB of high speed RAM behind it and mirrored standard drives on SATA III.

The data was collected as part of the automated Load Test step of the build process. During this step, the build server deploys a copy of the application to a website on the web VM and then spins up WCAT to load test the site. There is an incremental spin up period that warms up the site and connection pool, followed by a 2.5 minute sustained test where WCAT runs as fast as it can against the site.

The load test is a full transaction that makes the same requests a real world browser makes as it selects an album, adds it to a cart, and then checks out, including static content and dynamic content. Each transaction is in a unique session and reports the appropriate headers with each HTTP request. The increase from 1.8 to 4.8 transactions per second can be restated as an increase from 1.8 to 4.8 individual purchases per second.

The site is still on SQL CE for the membership database, which is used during the checkout, and it is using Entity Framework 4 for data access to SQL Server (so more headroom for improvement if I was actually tuning for performance).

After seeing the increase I thought it would be interesting to share the numbers as well as how useful it is to have a short load test as part of the delivery pipeline.

Original post blogged on LessThanDot.

The Cloud as a Make-It-Finish-Sooner Dial

Eli Weinstock-Herman (tarwn) — Wed, 23 May 2012 10:10:00 +0000

There's been a lot of buzz about the cloud over the past years, with a lot of that attention going to IaaS and SaaS platforms, but there's a revolution (or re-revolution) that is of even more importance, and that's PaaS. What PaaS brings us is the ability to scale horizontally and treat CPU, memory, and storage as pools of resources that are as deep as our checkbooks allow.

Forget about virtual servers. Remember that 60 hour job with a 24 hour deadline? Built on a PaaS platform and equipped with a couple hundred dollars, you won't even be staying late today.

Today's post is going to share highlights from a basic file processing application. Something you would find (hopefully better written) in any random enterprise IT shop or SaaS company. It offers a web page that lets you upload files, a button to process files, a basic and poorly written list of the processed and unprocessed files, and an unattended worker. The trick is that this application was written on top of Windows Azure, so I can play tricks with time just by twisting the dial from one file processor to twenty.

The Basic File Processor

The file processing in the sample application is intended to be a sample workload. It consists of reading files completely into memory and passing them around, spinning through them one character at a time, replacing each character in the line with it's upper case variant. Very critical stuff, very performant.

In addition to running the process via the website, I also need an unattended application that will can run the same processing function. If I owned the server, this would be a scheduled task or service. As an Azure Worker the code will be remarkably similar.

Architecture of the Processor

The two front-ends access common logic in the Core library, which is responsible for both the processing logic and interacting with storage resources. This being sample code, it is certified as working on my machine and is definitely not production ready. That being said, I did write this in a few evenings, so writing a production-ready service doesn't have to take that long in normal workdays.

The Web Site

The website has a single MVC controller with 3 actions:

~/Home/Index: Displays the list of processed and unprocessed items and buttons for upload and processing
~/Home/AddFile: The post address for file uploads
~/Home/ProcessNextItem: An action to process the next queued file

Code: csharp

public class HomeController : Controller {

IStorageLocator _storageLocator;

public HomeController() {
_storageLocator = new StorageManager("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString");
}

public ActionResult Index() {
var store = new ItemStore(_storageLocator);
var model = new StatusViewModel() {
ProcessedItems = store.GetProcessedList(),
UnprocessedItems = store.GetUnprocessedList()
};

ViewData["file"] = TempData["file"];

return View(model);
}

[HttpPost]
public ActionResult AddFile(HttpPostedFileBase file) {
if (file != null && file.ContentLength > 0) {
var item = new FullItem() {
ResourceId = Guid.NewGuid(),
Received = DateTime.Now.ToUniversalTime(),
IsProcessed = false,
FileName = file.FileName
};
item.ReadFileFromStream(file.InputStream);

new ItemStore(_storageLocator).AddNewItem(item);
TempData["file"] = file.FileName + " uploaded and queued for processing.";
}
else {
TempData["file"] = "Processor ignores empty files, sorry.";
}

return RedirectToAction("Index");
}

[HttpGet]
public ActionResult ProcessNextItem() {
var store = new ItemStore(_storageLocator);
new ItemProcessor().ProcessNextItem(store);

return RedirectToAction("Index");
}
}

The key to all of these methods is the ItemStore and ItemProcessor class, all of the rest of the logic is basic presentation layer logic.

The Worker Role

The worker role consists of a roughly 6 line while(true) statement that asks the ItemStore to process the next item in the queue, then sleeps for 1 second.

Code: csharp

public override void Run() {
while (true) {
var storageLocator = new StorageManager("Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString");
var store = new ItemStore(storageLocator);
new ItemProcessor().ProcessNextItem(store);
Thread.Sleep(1000);
Trace.WriteLine("Working", "Information");
}
}

Once again, the magic happens in the ItemStore and ItemProcessor instances.

The ItemStore

The ItemStore class exposes the basic methods we need execute our process:

Code: csharp

public void AddNewItem(FullItem item) { /* More Code */ }

public FullItem RetrieveForProcessing() { /* More Code */ }

public void StoreFinishedItem(FullItem item) { /* More Code */ }

And a pair of methods for visibility:

Code: csharp

public IEnumerable<ItemBase> GetUnprocessedList() { /* More Code */ }

public IEnumerable<ItemBase> GetProcessedList() { /* More Code */ }

Windows Azure offers a number of storage options, each with their own benefits and constraints. For this process I decided to use table storage to track the summary level information about each file processing job, blob storage to store the actual file, and the queue service for managing task execution.

Code: csharp

public class ItemStore {

public static string RAW_BLOB_NAME = "RawItems";
public static string FINISHED_BLOB_NAME = "FinishedItems";
public static string QUEUE_NAME = "ToBeProcessed";
public static string TABLE_NAME = "Items";

ITableStore _table;
IBlobStore _rawBlob, _finishedBlob;
IQueueStore _queue;

public ItemStore(IStorageLocator storageLocator) {
_table = storageLocator.GetTable(TABLE_NAME);
_rawBlob = storageLocator.GetBlob(RAW_BLOB_NAME);
_finishedBlob = storageLocator.GetBlob(FINISHED_BLOB_NAME);
_queue = storageLocator.GetQueue(QUEUE_NAME);
}

public void AddNewItem(FullItem item) {
_rawBlob.Create(item.ResourceId, item.File);
_queue.Enqueue(item.AsSummary());
_table.Create(item.AsSummary());
}

public IEnumerable<ItemBase> GetUnprocessedList() {
return _table.GetUnprocessedItems().ToList();
}

public IEnumerable<ItemBase> GetProcessedList() {
// ?
return _table.GetProcessedItems().ToList();
}

public FullItem RetrieveForProcessing() {
FullItem rawItem = null;

var item = _queue.Dequeue();
if (item != null) {
rawItem = new FullItem(item);
rawItem.File = _rawBlob.Retrieve(item.ResourceId);
}
return rawItem;
}

public void StoreFinishedItem(FullItem item) {
_finishedBlob.Create(item.ResourceId, item.File);
_rawBlob.Delete(item.ResourceId);
_table.Update(item.AsSummary());
}
}

The ItemStore class is built to interact with interfaces for each of these resources, using a single IStorageLocator interface to get instances of those resource interfaces. The class (and application) was driven by the small set of unit tests that helped me define how i wanted the process to work and interact with the resources above.

Configurations

With all of the pieces defined, we use a pair of configurations to tell Azure how we want to deploy everything.

The first configuration defines the services we intend to package and deploy as well as the instance size and any endpoints:

Code: xml

The second configuration is applied when we deploy the instances above and tells Azure that I want to deploy 1 Processor_WebRole instance and 2 Processor_WorkerRole instances:

Code: xml

Note that I'm telling it to use the local development storage, which is supported by a local storage emulator. In a production configuration I would enter the service location and a generated token.

So Where's the Magic?

So where's the magic that makes this a distribute application instead of 3 days of overtime? It's sprinkled throughout the system.

The architecture of this system would work just as well outside of Azure, provided I offered it stand-ins for the 3 storage resources and deployed the instances and any necessary settings accordingly. Instead of worrying about how to manage deployments and what to use for centralized queueing and storage, I can focus on building an application that simply assumes those resources are available. Is there headroom for performance improvements? Sure, but I can also choose to throw another $15/month server at it, push data to CDNs and blob storage, add caching, or even a SQL Azure instance.

This application may be fairly basic, but nothing stops us from following this same pattern for much larger applications. PaaS has removed some of the constraints we take for granted. Even applications that have to run in-house in order to standardize against a database can now consider uploading a subset of that lookup data to a table store, performing most of the heavy lifting in the cloud, then produce a few files to import back into the on-premise system. The total execution time would be longer, but being able to scale part of the job across numerous parallel instances means the actual elapsed time can actually be much shorter.

And it means when we have a 60 hour task that's due in 24, it's no longer an impossible situation.

The source code is available on github along with requirements and links for setting up the emulators locally.

Original post blogged on LessThanDot.

PetaPoco: Mapping related objects

Eli Weinstock-Herman (tarwn) — Mon, 07 May 2012 10:16:00 +0000

In the prior PetaPoco post, I started to dig into many-to-one relationships a little. Chrissie followed up with yet more mapping behavior in his latest Simple.Data post, so I thought I would cover it in a bit more detail.

Note: Chrissie has also covered one-to-many since I wrote this post the other night and has at least one more post following that

Many flavors of Mapping Related Objects

PetaPoco doesn't offer the instrumentation for lazy loading, though it wouldn't be too hard to add it to the T4 template that is provided to automatically generate POCOs from the database. Of course the POCOs would stop being POCOs at this point and I'd be showing off my ability to write lazy loading rather than the library at hand, so lets stick to what PetaPoco does out of the box.

That said, we still have a number of ways to map data from multi-table queries into objects.

Note: I am using the same tables and insert statements I used in the prior post to create the Person and Address table, so I've left those out of the examples below to reduce the noise a bit

Automatic Mapping w/ Decorated Objects

As we saw in the prior post, using a pair of decorated objects makes it pretty easy to map a JOIN to objects:

Code: csharp

public void SelectUsingDecoratedClasses() {
using (var db = GetDatabase()) {
var results = db.Query<DecoratedPerson, DecoratedAddress>(
@"SELECT Person.*, Address.*
FROM Person
INNER JOIN Address ON Person.AddressId = Address.Id
WHERE Person.lastname=@0", "lastname1");

foreach (var person in results) {
Console.WriteLine("Person: {0} {1}", person.LastName, person.FirstName);
Console.WriteLine("Address: {0} {1}", person.Address.Street, person.Address.HouseNumber);
}
}
}

[TableName("Person")]
[PrimaryKey("Id", autoIncrement = true)]
public class DecoratedPerson {
public int Id { get; set; }
public string LastName { get; set; }
public string FirstName { get; set; }
public int AddressId { get; set; }

[Ignore]
public DecoratedAddress Address { get; set; }

public override string ToString() {
return String.Format("{0}: {1}, {2}", Id, LastName, FirstName);
}
}

[TableName("Address")]
[PrimaryKey("Id", autoIncrement = true)]
public class DecoratedAddress {
public int Id { get; set; }
public string Street { get; set; }
public string HouseNumber { get; set; }
}

Defining Mappings

While the previous example handled the mapping automatically and assigned the Address instance to the appropriate attribute in the Person, we also have the ability to define the mapping manually.

Code: csharp

public void SelectUsingMappingAndPOCO() {
using (var db = GetDatabase()) {
var results = db.Query<Person, Address, Person>(
(p, a) => { p.Address = a; return p; },
@"SELECT Person.*, Address.*
FROM Person
INNER JOIN Address ON Person.AddressId = Address.Id
WHERE Person.lastname=@0", "lastname1");

foreach (var person in results) {
Console.WriteLine("Person: {0} {1}", person.LastName, person.FirstName);
Console.WriteLine("Address: {0} {1}", person.Address.Street, person.Address.HouseNumber);
}
}
}

While this example achieves the same outcome as the prior one, the ability to provide our own mapping gives us some flexibility to add more complex logic during the mapping process, such as calculating additional field values or adding change tracking.

Dynamics

Of course PetaPoco also handles dynamics, however this is limited to outputting a single object to represent the results. This works well if we wanted to present a report view of the data and didn't have any column names that repeat:

Code: csharp

public void SelectWithDynamics() {
using (var db = GetDatabase()) {
var results = db.Query<dynamic>(
@"SELECT Person.*, Address.Street, Address.HouseNumber
FROM Person
INNER JOIN Address ON Person.AddressId = Address.Id
WHERE Person.lastname=@0", "lastname1");

foreach (var person in results) {
Console.WriteLine("Person: {0} {1}", person.LastName, person.FirstName);
Console.WriteLine("Address: {0} {1}", person.Street, person.HouseNumber);
}
}
}

Instead of a dynamic, we could just as easily create a POCO for this report view, which would then be easy to offer as a service DTO or serializable object.

One-to-Many

Switching directions for a moment, let's instead query for an address and all of it's associated persons. First we'll need an updated POCO:

Code: csharp

public class AddressWithPeople : Address {
public List<Person> Persons { get; set; }
}

Then with a slightly more complex mapping method, we can map a one-to-many to our new AddressWithPeople and existing Person POCOs:

Code: csharp

public void SelectOneToMany() {
using (var db = GetDatabase()) {
var results = db.Query<AddressWithPeople, Person, AddressWithPeople>(
new AddressToPersonRelator().MapIt,
@"SELECT Address.*, Person.*
FROM Person
INNER JOIN Address ON Person.AddressId = Address.Id
WHERE Address.Id=@0", 1);

foreach (var address in results) {
Console.WriteLine("Address: {0} {1}", address.Street, address.HouseNumber);
foreach(var person in address.Persons)
Console.WriteLine("\tPerson: {0} {1}", person.LastName, person.FirstName);
}
}
}

Of course, the magic in this case is the tricky part. In order to map the objects from the right side of the result set to the columns from my address on the left, I had to write a custom mapper that would keep track of the Address and add Person records to it while it remained the same.

Code: csharp

public class AddressToPersonRelator {
public AddressWithPeople current;

public AddressWithPeople MapIt(AddressWithPeople a, Person p) {
if (a == null)
return current;

if (a != null && current != null && a.Id == current.Id) {
current.Persons.Add(p);
return null;
}

var prev = current;
current = a;
current.Persons = new List<Person>() { p };

return prev;
}
}

Even though I mostly copied this code from the official blogs, it was still more work than I would have liked just to map the records. However it wouldn't be hard to convert this method to use generics and accept arguments for ID comparison and adding child record to the parent's collection. Another option would be to use the PetaPoco.RelationExtensions nuget package, which offers simplified methods for one-to-many and many-to-one mappings.

Conclusion

Although Chrissie and I have been posting in parallel, I think we've reached the point where the feature sets diverge. Simple.Data obviously offers a simpler looking syntax for more complex JOINs (especially if we start looking at one-to-many) and offers fluent, LINQ-based syntax. PetaPoco offers a very clean, very simple way to continue to use SQL to get our data and map it into POCOs or dynamics, with the flexibility to do it for us automatically, with help from decorators, or via specified mapping functions (not to mention the blazing performance). Hopefully seeing us play back and forth a bit will have inspired you to play with one or both of these in the future, and don't be surprised if you see them reused in some of my future projects as well.

The examples above and several related ones are available in the source on GitHub as well as examples of the methods for updates and the upserting Save() method.

Original post blogged on LessThanDot.

More PetaPoco: Id's and Multi-POCO queries

Eli Weinstock-Herman (tarwn) — Wed, 02 May 2012 08:23:00 +0000

So yesterday Chrissie and I did posts on Simple.Data and PetaPoco. Today he followed up with more complex examples, including keys and multiple table queries.

PetaPoco is built specifically with primary keys as a first class citizen, so it will be interesting to see how it compares.

Adding a column

Like Simple.Data, adding a column to our database table is no problem at all. First lets add the column to our database, then we'll look at how that affects both our existing code and an updated POCO with a matching field.

Like before, I'll use the more concise syntax of PetaPoco to add the column instead of the standard ADO logic:

Code: csharp

private void CreateTable() {
using (var db = new Database("DataSource=\"test.sdf\"; Password=\"chrissiespassword\"", "System.Data.SqlServerCe.4.0")) {
db.Execute("CREATE TABLE Person (Id int IDENTITY(1,1) PRIMARY KEY, LastName nvarchar (40) NOT NULL, FirstName nvarchar (40));");
}
}

After adding this additional column, we can still use the original POCO as PetaPoco will map the columns that are present in the POCO without complaining about leftovers. In the case where we allowed PetaPoco to build the query for us, we get more concise SQL that only queries for the columns with matching properties in that smaller POCO class:

Code: csharp

// SQL = SELECT [Person].[LastName], [Person].[FirstName] FROM [Person] WHERE lastname=@0
private void SelectDecoratedRecords() {
using (var db = GetDatabase()) {
var results = db.Query<DecoratedPerson>("WHERE lastname=@0", "lastname1");
}
}

Now let's add the new column to both our raw POCO and the decorated POCO:

Code: csharp

public class Person {
public int Id { get; set; }
public string LastName { get; set; }
public string FirstName { get; set; }

public override string ToString() {
return String.Format("{0}: {1}, {2}", Id, LastName, FirstName);
}
}

[TableName("Person")]
[PrimaryKey("Id",autoIncrement=true)]
public class DecoratedPerson : Person { }

Without changing of the logic, the function from above will populate the extra column in our non-decorated POCO and the SQL generated by the short query version above will now include the Id column:

Code: csharp

//SQL: SELECT * FROM Person WHERE lastname=@0
private void SelectRecords() {
using (var db = GetDatabase()) {
var results = db.Query<Person>("SELECT * FROM Person WHERE lastname=@0", "lastname1");
}
}

//SQL: SELECT [Person].[Id], [Person].[LastName], [Person].[FirstName] FROM [Person] WHERE lastname=@0
private void SelectDecoratedRecords() {
using (var db = GetDatabase()) {
var results = db.Query<DecoratedPerson>("WHERE lastname=@0", "lastname1");
}
}

Yesterdays third insert option, using an undecorated object and counting on reflection to match up columns to properties, will fail now because it will attempt to insert a value into that ID field, but using it in a select would still work. The first option, where we supplied the name of the table, will have to be updated to also supply the name of the ID and a boolean to indicate that it is autoincrementing. The second option, inserting the decorated object, requires no changes at all.

Adding a Table

Following Chrissie's lead, lets add an address table and an undeclared foreign key relationship from the Person table (I have to tease him about something).

Code: csharp

private void CreateTables() {
using (var db = new Database("DataSource=\"test.sdf\"; Password=\"chrissiespassword\"", "System.Data.SqlServerCe.4.0")) {
db.Execute("CREATE TABLE Person (Id int IDENTITY(1,1) PRIMARY KEY, LastName nvarchar (40) NOT NULL, FirstName nvarchar (40), AddressId int NOT NULL);");
db.Execute("CREATE TABLE Address (Id int IDENTITY(1,1) PRIMARY KEY, Street nvarchar (40) NOT NULL, HouseNumber nvarchar (10));");
}
}

And I will add an additional Person POCO to reflect the new column, as well as an Address POCO to reflect the new table. The simpler query logic is addictive, so I've decorated both POCOs (I'll explain the ResultColumn later):

Code: csharp

[TableName("Person")]
[PrimaryKey("Id", autoIncrement = true)]
public class Person {
public int Id { get; set; }
public string LastName { get; set; }
public string FirstName { get; set; }
public int AddressId { get; set; }
[ResultColumn] public Address Address { get; set; }

public override string ToString() {
return String.Format("{0}: {1}, {2}", Id, LastName, FirstName);
}
}

[TableName("Address")]
[PrimaryKey("Id", autoIncrement = true)]
public class Address {
public int Id { get; set; }
public string Street { get; set; }
public string HouseNumber { get; set; }
}

Following Chrissie's lead, I'll query for the related records separately first:

Code: csharp

public void QuerySeperately() {
// already called CreateDatabase()
// already called CreateTables()
using (var db = GetDatabase()) {
db.Insert(new Address() { Street = "street1", HouseNumber = "1" });
db.Insert(new Person() { LastName = "lastname1", FirstName = "firstname1", AddressId = 1 });
db.Insert(new Person() { LastName = "lastname2", FirstName = "firstname2", AddressId = 1 });

var results = db.Query<Person>("WHERE LastName=@0", "lastname1");
foreach (var person in results) {
Console.WriteLine("Person: {0} {1} {2}", person.Id, person.LastName, person.FirstName);
var address = db.Single<Address>("Where Id=@0", person.AddressId);
Console.WriteLine("Address: {0} {1}", address.Street, address.HouseNumber);
}

int count = db.ExecuteScalar<int>("SELECT COUNT(*) FROM Person WHERE LastName=@0", "lastname1");
Console.WriteLine("Count: " + count.ToString());
}
}

As he pointed out, this method doesn't perform well. We can replace this with a single query using the Multi-POCO support.

Code: csharp

public void QueryMultiStyle() {
// already called CreateDatabase()
// already called CreateTables()
using (var db = GetDatabase()) {
db.Insert(new Address() { Street = "street1", HouseNumber = "1" });
db.Insert(new Person() { LastName = "lastname1", FirstName = "firstname1", AddressId = 1 });
db.Insert(new Person() { LastName = "lastname1", FirstName = "firstname2", AddressId = 1 });

var results = db.Query<Person, Address>(@"
SELECT Person.*, Address.*
FROM Person
INNER JOIN Address ON Person.AddressId = Address.Id
WHERE Person.lastname=@0", "lastname1");
foreach (var person in results) {
Console.WriteLine("Person: {0} {1}", person.LastName, person.FirstName);
Console.WriteLine("Address: {0} {1}", person.Address.Street, person.Address.HouseNumber);
}
}
}

PetaPoco has the ability to map the results of JOINs to several objects, but it's kind of tricky. The simplest method is to return the fields in the same order as the generic object list. What PetaPoco then does is attempt to process each column in the result from left to right, moving to the next object in line when it reaches a column that doesn't exist in the first or has already been populated. So in this case, because both of the tables and POCOs have an "Id", when the result set reaches the second id it makes the logical conclusion that it is time to start mapping the Address object. PetaPoco uses type detection in the Person object to locate a property to assign the Address instance to.

There is also more extensive capabilities available to use lambdas to manage the multi-POCO mapping on our own, and if we wanted to we could easily define a single POCO object that had all the necessary fields for both tables. Logic for One-to-many joins is more complex and I haven't had time to dig fully into the intricacies yet.

The last trick was the ResultColumn attribute I used above. By default PetaPoco assumes that all of the properties in our POCO are going to be inserted into the database. ResultColumn properties are ignored for inserts and updates, but can still be selected into. In this case I'm using it to have PetaPoco ignore the column, but the real purpose would be to allow me to return an additional calculated column, aggregate, or other value that wouldn't have meaning in an INSERT or UPDATE.

Note: There is an Ignore attribute that would have worked just as well and been a better fit, but then I wouldn't have had a chance to talk about the ResultColumn

Conclusion

So there we go. We can add auto-incrementing IDs very easily, PetaPoco is smart enough to map partial objects, and there is some really neat stuff available for multi-POCO joins. I've continued to update the github repository, so feel free to grab a copy of the code and play around with yourself.

Original post blogged on LessThanDot.

Playing with PetaPoco

Eli Weinstock-Herman (tarwn) — Sun, 29 Apr 2012 12:40:00 +0000

Since Chrissie is playing around with Simple.Data today, I found some time to play with PetaPoco. PetaPoco is a single file micro ORM that uses MSIL generation to do it's magic. As the name suggests, it works with concrete POCOs, though support for dynamics is also being tested. It is designed to be fast and doesn't try to reimplement SQL, so double win in my book. PetaPoco was created by Brad Robinson (b | t).

It has been on my list to try for a while, so today seemed like a good day for it.

Setting it up

Setup is simple using Nuget:

Install-Package PetaPoco
Install-Package SqlServerCompact

PetaPoco comes with a T4 template to generate POCOs from a database, all you have to do is add a connection string to your config and run the transform. This is a neat feature, but I didn't plan to use it for this post so I excluded the files from my project.

PetaPoco supports SQL Server, SQL Server CE, MySQL, PostgreSQL and Oracle, and works with .Net 3.5 and Mono 2.6 forward. Documentation is available via the Main Page and blog posts.

As Chrissie did in his post, we're going to first create a Compact SQL database, except we'll be doing it in C#.

Code: csharp

private SqlCeEngine CreateDatabase() {
if (File.Exists("test.sdf")) File.Delete("test.sdf");

string connectionString = "DataSource=\"test.sdf\"; Password=\"chrissiespassword\"";
var en = new SqlCeEngine(connectionString);
en.CreateDatabase();
return en;
}

Now that we have Chrissie's database, lets add his table.

Code: csharp

private void CreateTable() {
using (var db = new Database("DataSource=\"test.sdf\"; Password=\"chrissiespassword\"", "System.Data.SqlServerCe.4.0")) {
db.Execute("CREATE TABLE Person (LastName nvarchar (40) NOT NULL, FirstName nvarchar (40))");
}
}

This statement was quite a bit shorter using PetaPoco then it was with Simple.Data. The Database object takes care of the connection and command work for us, leaving us just the bits that are specific to our individual scenario. We still have the option of providing an IDbConnection if we want, which would be handy if we were using something like Sam Saffron's MiniProfiler and wanted to pass in a profiled connection object.

Because we want to play with concrete POCOs, I'm going to add a very plain POCO and a very basic decorated POCO:

Code: csharp

public class Person {
public string LastName { get; set; }
public string FirstName { get; set; }
}

[PetaPoco.TableName("Person")]
public class DecoratedPerson : Person { }

There are also attributes for the primary key* and the ability to explicitly define columns so PetaPoco will know which columns should be included in queries and which should not.

I have to say I was surprised Chrissie didn't include a Primary Key given how many DBAs and DB Developers also blog here, brave man

So as chrissie pointed out in his post, we've got the basic setup behind us and can now move forward to interacting with our new database.

PetaPoco

With a database created, lets go ahead and add some data to play with.

Code: csharp

db.Insert("Person", null, new Person() { LastName = "lastname1", FirstName = "firstname1" });

The first overload of the Insert() method takes a table name, primary key name, and the POCO instance to insert. if we don't mind add some decoration to our Person object, we can decorate the POCO with the table name and shrink it to this:

Code: csharp

db.Insert(new DecoratedPerson() { LastName = "lastname2", FirstName = "firstname2" });

This method uses the table name attribute to generate the insert.

And last, if we want to keep our table and object names in sync, we can let reflection magically figure it out:

Code: csharp

db.Insert(new Person() { LastName = "lastname3", FirstName = "firstname3" });

Now that we have some data in our database, lets look at a few ways to get it out.

Code: csharp

// select statement
var results = db.Query<Person>("SELECT * FROM Person WHERE lastname=@0", "lastname1");
// let PetaPoco generate the SELECT portion
var results = db.Query<DecoratedPerson>("WHERE lastname=@0", "lastname1");

We can execute a parameterized SQL statement fairly easily by using numbered parameters that will line up with the additional arguments we provide. In the second case we're actually letting PetaPoco generate the SELECT portion of the statement for us, which will resolve to: SELECT [Person].[LastName], [Person].[FirstName] FROM [Person] WHERE lastname=@0.

Executing specifically for a single record instead of querying for a collection is similarly straight forward:

Code: csharp

// select statement
var result = db.Single<Person>("SELECT * FROM Person WHERE lastname=@0", "lastname1");
Console.WriteLine(String.Format("{0}: {1}", result.GetType(), result));

// let PetaPoco generate the SELECT portion
var result = db.Single<DecoratedPerson>("WHERE lastname=@0", "lastname1");
Console.WriteLine(String.Format("{0}: {1}", result.GetType(), result));

And if we examine the output we'll see they are concrete instances of our POCOs, not dynamics or proxies:
PetaPocoSample.Person: lastname1, firstname1 PetaPocoSample.DecoratedPerson: lastname1, firstname1

If we then follow Chrissie's lead and add two records into the database that will match this criteria, we'll receive an exception, as we would expect from a Single call. PetaPoco also offers a First<T> implementation we could use in this situation, a SkipTake<T> we could use to get the 2nd record, and a number of different ways to query multiple records out of the database:

Code: csharp

// T
var result = db.First<DecoratedPerson>("WHERE lastname=@0", "lastname1");
// List<T>
var results = db.SkipTake<DecoratedPerson>(1, 1, "WHERE lastname=@0", "lastname1");
//IEnumerable<T>
var results2 = db.Query<DecoratedPerson>("WHERE lastname=@0", "lastname1");
//List<T>
var results3 = db.Fetch<DecoratedPerson>("WHERE lastname=@0", "lastname1");
//Page<T> - page #2 and page size of 1
var results4 = db.Page<DecoratedPerson>(2, 1, "WHERE lastname=@0", "lastname1");

Conclusion

I haven't done much with PetaPoco yet, but just from playing with these basic queries I can tell I want to spend some more time with it. The syntax is clean and focuses on simplifying the bits that are repeated in so many projects (connection and command wrangling, mapping) while leaving me the full power of SQL and not injecting an additional layer of abstraction to try to work through. On top of that, it performs very closely to the speed of hand-coded SqlDataReader statements (results available on the dapper-dot-net page).

My sample code is up on GitHub.

Original post blogged on LessThanDot.

Unit Testing Costs Too Much - Twice The Code = Value?

Eli Weinstock-Herman (tarwn) — Mon, 15 Nov 2010 15:56:09 +0000

Last week I had the pleasure of presenting at a .Net Code Camp on the topic of Unit Testing. A key theme of the session was barriers to adoption and the values we can achieve using this tool.

The main barrier to my adoption of Unit Testing was the idea that writing twice as much code would increase the value of my work. Twice as much code, as was pointed out in a response to my post two week's ago, sounds like higher maintenance costs, higher initial development costs, and a greater opportunity for bugs.

How Does Adding Code Increase Value?

People have published numerous lists of benefits they have received from Unit Testing, but there is still this seemingly illogical starting point of creating more code to deliver the same end product. And before evaluating any other benefits we could achieve from Unit testing, we have to start by resolving that seeming contradiction.

Current Process

Consider a software process that includes developers, a QA department, a customer review, and a production deployment.

Each developer writes code to fulfill assigned features or tasks
Once enough pieces have been written, the developer can wire them together and debug the pieces they have written
Completed pieces are handed off to QA
QA tests those pieces against the requirements
Bugs (code defects or missing requirements) are returned to developers
Once QA is satisfied, the pieces are handed off to the customer for review
The customer reviews the pieces, possibly sending back instructions for refinement or corrections
Once the customer is satisfied, the pieces are deployed to production
End users use the pieces in production and interact with production data

This won't match everyone's environments but it should be simple enough to be representative of the work that happens in our environments.

Unit Testing is Testing at the Source

In Lean Manufacturing there is the concept of moving quality to the source of production. Tests are created that are quick to execute and provide the operator with a quick detection of defects instead of waiting for a Quality Control at the end of the process.

Moving testing closer to the source reduces the number of defective parts that are stored, transported, and used in later production steps. It allows for faster reaction time to correct the problem and reduces the number of interruptions from having to stop a run and re-produce product for a previous order.

All of these costs have well-defined values in manufacturing environments as well as parallels in our own environments.

Lean Development

Each step in our sample development process above increases the base cost of producing a working feature. The further a bug makes it without being detected the greater costs it has accrued and the more cost it will take to correct it.

Effort and cost from bugs increase dramatically as time passes

Costs:

Bug while writing one unit: Cheapest cost
Several pieces wired together: changes to the unit and surrounding units
During Handoff: Wasted meeting/handoff time for devs and QA personnel, potential schedule change, additional debugging time and cost changes (per prior step)
Bugs found in QA: (at minimum) Handoff to Dev, switch from new task to work on bug, investigation, development, debugging changes, handoff to QA, QA repeats some level of testing (high potential for new bug, developers tend to test direct changes only)
Bugs found in handoff to customer: (at a minimum) Prior QA steps,time for QA to duplicate problem, task-switching time for QA, cost of customer time and others involved, re-handoff to customer, timeline impact, potential impact to customer trust (depends on frequency - costs include increased micro-management, increased meetings, etc)
Bugs found by customer: Similar to last step but any testing to this point by customer is likely to be duplicated, customer schedules impacted, cost of extar customer time, timeline impacted
Bugs found in production: Prior step + developers are pulled off other tasks to detect and cleanup bad data, schedules are impacted due to developer time, potentially wasted end user time, potential to retrain + monitor users to work around problem and then retrain back afterward

Besides people-based costs, there are also hidden costs to returning to development and quality stages. The more time that passes between writing the code and returning to it, the more familiarity will be lost and the higher the potential that other pieces will have been added on top of the bug. Other details, such as the specifics of the requirements that code was fulfilling and how it was initially tested prior to QA hand-off, will rely on the memory of the developer and will have to take into account any changes that have occurred since initial development. Perhaps of greater concern is that QA will not have the same level of familiarity with the tests they initially performed and the customer is more likely to add even more 'refinements' when it comes time to re-review the changes.

Unit Testing is our tool for doing testing at the source, to catch defects earlier before they propagate costs throughout the process.

The Value of Quality at the Source

Unit Testing will not prevent all bugs but it will catch bugs earlier, reducing the costs above. Unit Testing starts earlier, allowing us to test each unit we build instead of waiting until we have 10-20 built and wired to an interface. For the situations where a bug still makes it past development, we will have a solid set of tests to run after making corrections so we can not only test changes but also the rest of the codebase, heavily reducing the risk of creating new bugs out of bug fixes.

The value we gain from Unit Testing is value to the team and the business, not to the individual developer (though there are values to be gained there as well).

Balancing Cost and Value

There is a cost to writing Unit Tests that we have to balance with the potential savings and risk reduction. I don't believe there is a universal answer as to how much Unit Testing is appropriate for a given environment, project, and team. Many teams take it on faith that 100% coverage is their best bet, but I am not convinced.

What I can tell you is that all of the benefits of this tool are out of your reach if you don’t give it a test drive.

As several people saw in my CodeCamp presentation, it doesn’t take long to add a Unit Testing project to an existing application and build a few tests. Pick a particularly complex business function or an existing bug in need of correction. If you decide to start evaluating it for greater implementation, the list of costs above should provide some pointers on where you can create some measurements to determine the level of value you are receiving.