Architecture, Design & Strategy

Scalability is Easy! (To Get Wrong)

Eli Weinstock-Herman (tarwn) — Wed, 05 Dec 2012 12:50:00 +0000

Scalability is easy, provided you don't need it to work.

Probably the number one failure of system scaling is when people dive right in and start building. No baselines, limited measurements, no analysis, just a hypothesis and a whole lot of late nights tweaking the system. With extra complexity comes extra costs, from the initial development through more expensive maintenance. Scale poorly and not only do we take on those extra complexity costs, but also the more obvious additional costs of the actual implementation (new servers, more resources, etc).

The Somewhat Contrived Example

Recently I've been working on a system to simulate parallelizing workloads, specifically workloads that depend on external resources with rate or load thresholds. Let's use it to look at a somewhat contrived example.

Note: For this post, the simulated "service" has a 100 request/minute limit and throttles individual clients for 15 second windows. Individual operations consist of 50ms of local processing and a single service request that has 50ms of latency and 100 ms for processing and response time. Similar results can be achieved with more realistic batch sizes and rates, the smaller values just allow me to more quickly produce samples for the blog post.

So the backstory is that I have a batch process that is running more and more slowly as we take on larger and more frequent workloads.

I start by testing the batch process locally so I can see how slow it is before I make changes.

1 Client, 60 Requests, 100rpm

Locally it runs pretty quickly, but I'm betting that parallelizing the process will give me a significant increase in speed.

1 Client vs 30 Clients, 60 Requests, 100rpm

Look how well that performance improved, I achieved better than a 6x improvement in performance. My job here is done.

Except when I try to push this to production, I start getting a lot of errors.

30 Clients, 300 Requests, 100rpm - 67% Failure Rate

As it turns out, the external API my process saves the data to has a rate limit. When I exceed the allowable rate, I'm throttled for a short period of time. Any requests I make during that throttle period are returned with errors indicating I'm throttled.

Hmm. Luckily there are a number of common patterns available to retry these types of failures. I'll add an exponential back-off retry pattern so that when I get throttled my service will retry failed requests at slower rates until the service un-throttles me. While I've found plenty of code examples online, none of them seem to have recommendations, so I'll just use one of the sample settings they provide.

30 Clients, 300 Requests, 100rpm, Retry Policy #1 - 7% Failure Rate

Hmm, better. My failure rate has gone way down. What if I tweak the values?

30 Clients, 300 Requests, 100rpm, Retry Policy #2 - 29% Failure Rate

Oh, that was bad, I obviously was on the right track before. What if I just extend the retry amount a bit to try and knock out the last bit of errors.

30 Clients, 300 Requests, 100rpm, Retry Policy #1B - 0% Failure Rate

Ok, perfect. Now I have a system that is more than 6 times faster than the original, can be easily extended by throwing more workers at it, and is actually in a better position to handle occasional slow downs from my 3rd-party service.

Success!

Where I Went Wrong

Ok, so maybe not. Over the course of my little story I went wrong in a number of places. Even though this was a contrived example, I've watched very similar scenarios play out in a number of different organizations with real systems.

The Bottleneck

The first and most critical problem was that I didn't actually locate the bottleneck in my process, I simply tried to do more of the same. The Theory of Constraints tells us that we can improve the rate of a process by identifying and exploiting the constraints.

In this system, the constraint looked like it was the sequential execution of the tasks, but in reality the constraint was the time it took to call the 3rd-party API. Had we identified that bottleneck before starting, we could have approached the problem differently.

Process - Alternative Design

Rather than the parallel complexity, we can modify how the tasks are executed to try and take advantage of knowing where our bottleneck is. If the API allowed us to submit several requests in a batch, this redesign would net us several orders of magnitude improvement. Another option would be to run the results of the local processing into a queue and submit requests from there at a slow trickle, using only a percentage of our API limit so as not to disrupt any other real-time operations or batch processing the system supports. Another option we could take advantage of is not starting any of our expensive 3rd-party communications until we know that the entire job can actually be processed successfully through our local process.

Identifying the constraint unlocks the ability to turn the problem on it's head and achieve higher improvements, typically by orders of magnitude.

The Math Error

I concluded the scenario above by assuming I had found a good solution that also had a lot of headroom. Unfortunately what I actually did was find the ceiling. I have tuned the retry policy to 30 parallel systems, increasing that number could easily destabilize it further and cause more errors or delays. The headroom is, in fact, an illusion.

The Evolving Assumption

Somewhere along the way I found 30 clients to be a great improvement and didn't test other options. Then I made some assumptions about a retry policy. Then I tweaked that retry policy until the error rate disappeared. My assumptions made sense at the time, so I never questioned where they were leading me.

When I found a winning combination for my retry rate, I didn't realize I was missing other, better options:

30 Clients, 300 Requests, 100rpm, Retry Policy #1B vs 4A-C

What's worse, is that trail of assumptions along the way was never re-validated. I concluded with a 6.5x improvement, but is that still accurate?

1-50 Clients, 300 Requests, 100rpm, Retry Policy #1B

When I run the same settings on a range of 1 to 50 clients, we can see that I lost that original 6x improvement along the way. All I have managed to do is add complexity and some very explicit costs for those additional systems.

Note: What happened to retry policy #3? Well originally 1B was actually named 3, then I decided to update the name halfway through the post but was too lazy to update all of the completed images. Oh well.

Scaling the Wrong Thing

This post used fairly small numbers, had I applied a larger workload, higher throughput rates, different throttling windows, the whole problem would have turned out differently.

When we set out to make a system scale, we need to identify the real scenarios we are trying to scale for and the bottlenecks that stand in our way. Blindly performance tuning can look like an improvement, but is really just a poor short term investment that often entrenches the current performance problems even more deeply. There are a lot of questions that should be asked about the intended result, responsiveness fo the system, other operations it has to support while under load, potential overlap of that load, the type of load, etc. The patterns for one system may have relevance for another, but could just as easily be completely incorrect.

How hard it is to scale a system is going to depend on a lot of factors. Getting it wrong just happens to be the easiest option.

Original post blogged on LessThanDot.

Getting Flexible With NDepend 4 and CQLinq

Alex Ullrich — Mon, 04 Jun 2012 10:45:00 +0000

At my last job we had a non-functional attribute that another team used to decorate service methods that they consumed. The other team was working on an alternative client to our WCF services, and they weren't on the same release schedule they needed to be able to target multiple versions of our services within a single version of their application. Because of this requirement, they maintained a wrapper around our services that handled some of the differences between versions. The main use for this attribute was to foster communication between the teams, so that if we changed a decorated method we would let them know. As I'm sure anyone on this other team would tell you, we weren't always that good about communicating these changes.

In an effort to make communication between teams easier we used a CQL query like this to report changes to these methods as part of our automated builds:

Code: sql

SELECT METHODS FROM NAMESPACES "Services"
WHERE HasAttribute "OPTIONAL:Services.KnownExternalClientsAttribute"
AND CodeWasChanged

This was nice, but it only got us part of the way there. This would alert us to signature changes or changes to the content of the method, but not necessarily changes to the message contracts passed in to the method. In Pseudo-CQL the query I had in mind looks something like this:

Code: sql

SELECT TYPES FROM NAMESPACES "Services"
WHERE CodeWasChanged
AND IsUsedBy (
SELECT METHODS FROM NAMESPACES "Services"
WHERE HasAttribute "OPTIONAL:Services.KnownExternalClientsAttribute"
)

This didn't work however (CQL doesn't really have support for subqueries), and I couldn't really find anything in the language that would allow us to achieve what we wanted. NDepend 4 introduces a new linq-based replacement called CQLinq that offers a lot more flexibility, so I figured I would see if I could write the query that we needed using it. It ended up being much easier than I thought - CQLinq gives us access to most (if not all) of the standard LINQ operators, and the same functions for querying code using attributes and history that we had with CQL. This is the query I came up with:

Code: csharp

// <Name>Test Query for Contract Changes</Name>
warnif count > 0

let decoratedMethods = from m in JustMyCode.Methods
where m.HasAttribute("NDependSample.TestAttribute")
&& m.ParentNamespace.Name == "NDependSample.Services"
select m

from t in JustMyCode.Types
where t.ParentNamespace.Name == "NDependSample.Contracts"
&& t.CodeWasChanged()
&& decoratedMethods.Using(t).Any()
select t

Once we have the query we can mark it as critical, so we will have a failing build after the changes are made. Only the first build after making the changes should fail, but that would be enough to trigger an investigation that would result in communicating the changes to the other team.

CQL has always been my favorite feature of NDepend, so its no surprise that CQLinq is my favorite feature in this new release. The LINQ based syntax feels much more natural to me when writing queries against a codebase than the SQL-like syntax of CQL, and still gives us all the same visualization goodies to foster quick understanding of the query results. I'm really excited to dig in a little more and see what else I can do with it.

Original post blogged on LessThanDot.

Adding User Emulation to an Application

Eli Weinstock-Herman (tarwn) — Wed, 27 Apr 2011 12:57:00 +0000

One of the tricks I picked up from my last job (and our forum software, now that I think of it) is the idea of user emulation. I could log into the application, search for a user, and, at the push of a button, temporarily become that user. The only differences between emulating them and actually logging in as them were a black bar that indicated who I am (with a link to stop emulating), all audit records continued to reflect my own user id, and I didn't need to keep track of 30 different sample accounts and passwords.

As I said, it's a neat trick.

Advantages

Implemented consistently, this stops being just a trick and becomes a very powerful tool. Developers, QA, even customer service and support see benefits from being able to quickly emulate end users.

Debugging

As developers the largest benefit for us is the ability to debug our systems from a variety of viewpoints. Rather than going through the trouble of creating and managing sample users for every impacted role, we can use existing user records with real data behind them. This not only reduces the time to start debugging and remove the time involved in ongoing maintenance of test accounts, but may actually force out a few extra bugs that we wouldn't catch with a new, vanilla user account.

Support

Duplicating a bug or answering a question becomes a lot easier when we can emulate the person on the other end of the bug report or phone. We can emulate the person in our production environment and in our development environment and verify that both environments break in the same way (or that development doesn't break after we fix it). We also get firsthand clues, which can knock hours off the bug-hunting process.

Customer Service

Just as developers and support benefit on the technical side, in some cases Customer Service representatives (or selected members of the business) can use emulation to provide business or first level user support. When an end user has a complex question about an order or report, the service representative no longer needs to imagine their way through the issue, but instead can emulate that user, walk through the process, and see exactly what their user is seeing. This can be even more critical in systems where users see only a subset of the functionality or data available to service representatives.

Implementation

So emulation is a useful tool as well as a neat trick. Like many things, it is generally easier to bake this in from the beginning than to add it after the fact. If the user information is accessed in a consistent fashion in existing software, it is possible to squeeze in emulation logic and clean up the few places people cut corners and accessed users outside the normal context. If the user information is loaded and accessed at will throughout the application, adding emulation will be much harder (though there is some additional benefit in that it forces you to clean up your architecture a bit).

Sample Session Context Class

Code: vbnet

Public Class SessionContext

Private Property EmulatedUser As User
Private Property LoggedInUser As User

Public ReadOnly Property User As User
Get
If Me.EmulatedUser IsNot Nothing Then
Return Me.EmulatedUser
Else
Return Me.LoggedInUser
End If
End Get
End Property

Public ReadOnly Property IsEmulating As Boolean
Get
Return Me.EmulatedUser IsNot Nothing
End Get
End Property

''' <summary>
''' Property used for accessing current's users information for auditing
''' </summary>
Public ReadOnly Property UserIdForAuditing As Integer
Get
If Me.LoggedInUser IsNot Nothing Then Return Me.LoggedInUser.UserID
Return 0
End Get
End Property

Public Sub LogInUser(ByVal newUser As User)
Me.LoggedInUser = newUser
Me.EmulatedUser = Nothing
' plus other login logic stuff
End Sub

Public Sub LogOutUser()
Me.LoggedInUser = Nothing
Me.EmulatedUser = Nothing
End Sub

Public Sub StartEmulating(ByVal selectedUser As User)
Me.EmulatedUser = selectedUser
End Sub

Public Sub StopEmulating()
Me.EmulatedUser = Nothing
End Sub
End Class

Altogether not that complex a code construct, although I'm sure it will grow more so over the lifetime of the application.

As long as we consistently access user information through the exposed User property in the session and user the exposed UserIdForAuditing property for auditing purposes, then most of the work is done for us. The only other piece we need is a button on the UI to start emulating and some logic to handle the danger below.

Dangers

There are two main dangers to watch for. The first danger is making sure emulation doesn't grant users the ability to promote themselves. Either emulation needs to be reserved for administrative users, or logic needs to be added to make certain levels or roles unavailable for emulation (or assignment during emulation).

The other main danger is that you now have a much greater chance (probably guarantee) that you will have the same user "logged in" in two locations at once. Most applications handle this just fine, but there are also many that cannot. Examples of this behavior are enforcing single location sign-ons, coded security that assumes multiple location means an account has been compromised, and making the mistake of storing session data keyed only to a user id instead of a unique session.

Original post blogged on LessThanDot.

Why (and How) I Model

Eli Weinstock-Herman (tarwn) — Fri, 15 Oct 2010 09:20:39 +0000

Over my years in (and before) IT, I've seen long projects, failed projects, confused projects, wildly successful projects, and even fun projects that ended far differently than we expected. The consistent take-away for me is that I am a big picture type of person, and that understanding that big, abstract picture cuts out a lot of wasted time sprinting down the wrong paths.

Don't Sprint Blindly...
(care of dmitriev.com)

Creating a model forces me to refine a concept down to it's simplest elements, forces me to face the unknowns that my mind has so casually been skipping over. When done well, a model communicates a clear idea and replaces not only the thousands words required to explain it, but the 9000 I would have wasted getting there.

I model to think through processes, question my assumptions, and provide guidance towards a solution. While it probably looks like something I threw together in about ten minutes, there are actually a lot of processes going on behind the scenes.

Purpose - What are We Drawing?

No Goal? Here's a Diagram.

As with all things, a diagram should have a goal. A model that isn't trying to communicate an idea is filler for a report no one is going to read anyway. A goal should be concise and limited to a single subject or perspective:

The data flow from the end customer to our master data system
An order-to-cash business process
The functional architecture of a software application
A graphic representation of our current state

Mess around with too many factors and at the end of the day a mess is all you'll have:

The physical network topology combined with the disaster recovery plan and data flows between the systems
The application architecture with defined user work flows and user experience elements

Or to translate: gobbledygook.

Constraint - Less is More

A goal provides me with my first constraint, and constraints are good. Defining constraints will keep my model simpler and consistent, which means the end message will be clearer. At the same time, a well-defined set of constraints will encourage creativity, providing a better end product.

Often my constraints will include things like not allowing connections to cross, only using a very simple set of shapes, restricting myself to only a few shades of color, or setting time limits. I'll define the perspective I want to use with my goal, whether it will be a topological map, a flow, or just a set of connected shapes.

This keeps me focused instead of playing with the entire palette of colors, shapes, and page sizes available in my favorite software tools.

Content - Work on a Temporary Surface

Even with constraints and a goal, I still don't know exactly where I will end up or what I will learn along the way, so I start on the whiteboard. With a whiteboard I can start diagramming out the pieces I know, add in new items or resolve question marks as I run into them, and easily combine and rearrange my thoughts. Some of my constraints will be ineffective at this stage, but natural constraints (like the number of markers I have and the board size) will replace them in helping my creativity and thought processes.

This stage is also where I figure out my wording. Because it's so easy to see the big picture (heh) on my whiteboard, I also get a good feel for when words are too specific, not specific enough, or possibly just not quite the right word for what I am trying to communicate. Instead of focusing on getting all the boxes lined up, I can focus on using clear and consistent terminology that will help support the final model rather than detract from it.

Medium - Where is it Going?

Content needs Context

As I make the transition from whiteboard to diagramming software, the last piece of the equation is to consider the medium I am going to use to communicate the model. A standalone diagram may put higher priority on further simplicity of shapes and colors, where a presentation model may put lower priority on text and higher priority on subtleties for deeper conversation.

Will there need to be a legend? Is font-size 8 going to be a waste of time or readable font? If I use subtle shades of color will it all print the same color or show up in gloriously rendered imagery on a 12 foot display? Will adding a cartoon get a chuckle in a presentation or a frown in an executive review? Can I include a picture of my cat?

The context the diagram will be communicating in will determine the last set of constraints.

Terminology - The Wrong Word Invalidates the Model

Using the right terminology,
thanks lolcaption.com

General wording may have been roughed in on the whiteboard and some really good words may have been chosen, but these now need to be examined in light of the future medium as well as the audience. In many cases, using a word out of context can distract my audience or even negate the model's message entirely.

That being said, endless anxiety over perfection is nowhere near as good as a cool beer at the end of a work day, so we need to strike a balance between working through the night and achieving good enough. So I'll be careful, in general, when using customer terminology and try to be pragmatic in my search for the perfect name for the third box from the left.

Composition - Additional Layers of Meaning

The last stage of the model, having transferred it from whiteboard to software and applied corrections to terminology, is to add some depth that supports the initial concept.

Colors, re-arranging layout to alter proximity, fonts, and even line thicknesses are all tools I use to add subtle depth to a model. If I am planning on presenting the model, I can start the discussion on the general message of the model and dive into these subtleties as the discussion progresses. A thicker line between two systems can communicate greater bandwidth or a more secure transport layer. A common shading of colors between multiple objects communicates a relationship or similarity. As with the stages before, I try to use constraint. Applying the whole palette and a different shape for each object may seem fun, but it's going to communicate confusion (and possibly a desire for medication).

While working with the composition, I will also create temporary versions to play with drastically different layouts or shapes. This gives me a fresh look at a concept that has undoubtedly been on my whiteboard for days, giving me an opportunity to catch last minute holes or simply provide alternative layout options.

Sounds Like a Lot of Work...

There are different levels of work involved in modeling. In some cases even finding the time to stop doing and try to draw an idea may seem like a waste.

Not looking ahead?
Diagram for that too...

How do I judge when it is worth spending the time?

5 Minutes: If it only takes five minutes to draw a fast diagram of what I am intending to do, then that 5 minutes didn't cost much and I can move forward with confidence.

30 Minutes: If it takes 30 minutes, I've erased and redrawn half of it, and the person I am explaining it to is still arguing with me, then it's time to draw a model.

3 Hours: If it takes 3 hours, we end with more questions than we started with, half the questions have the potential for refocusing the project, and we're still trying to figure out what to call this thing...yeah, it's definitely time to get a handle on what we're spending time on.

Jumping right into any project when we can't draw a high level summary means we're spending time and resources on something we can't adequately define. It doesn't matter how fast we're moving if we're spending that time running in random directions and ignoring cross-traffic.

Original post blogged on LessThanDot.

CQL From Visual Studio With NDepend 3

Alex Ullrich — Mon, 06 Sep 2010 20:29:00 +0000

For the last few months I've had the pleasure of working with NDepend version 3. Most of my development at home is on linux these days, so I haven't used it as much as I'd like, but I have been using it to poke around in various codebases and see what the new Visual Studio integration is all about. The last version integrated with Visual Studio, technically speaking, but it didn't seem nearly as thorough as what I've seen in version 3. I suspect the improved extensibility model in VS 2010 has a lot to do with this, but can't confirm (I haven't tried it with 2008 either).

My favorite feature of NDepend has always been CQL, the SQL-like query language that allows you to query your codebase using a variety of common metrics. This is the same as it ever was (the integration with VS is even quite similar) but with the more thorough integration it seems much more useful. I like how easy it is now to keep an eye on my CQL constraints when I rebuild.

My favorite CQL feature is the ability to set up CQL constraints from now. This is really cool for older projects, where it's unrealistic to think that your team will be able to fix everything right away. But what you can do with this feature is ensure that all new or modified code does measure up to your team's standards. You may not be able to clean up all those 1,000 line methods right away, but you can ensure that newer methods fit in a more reasonable size limit (like 975 ). This is one of the most useful features of the application, IMO. The way it works is by allowing you to establish a baseline. By comparing the code's current state to this baseline, future analyses are able to determine which methods/types/etc are new or changed, and apply the constraint to only these methods/types/etc. Sometimes I feel like this would be useful just to be able to concisely see which methods have been changed as well (version control logs aren't the most friendly things to read, especially spread throughout a large codebase). Below is a screenshot of this feature in action.

The first three queries listed are built in to NDepend. I added the fourth, just to have a listing of new/changed methods ready. The CQL for this query is simply

Code: text

SELECT METHODS WHERE WasAdded OR CodeWasChanged

Not a bad way to keep an eye on what is getting changed in the codebase. To get in this state I added three new methods to the codebase I was looking at (in a place that I could remove them easily since they are not only low quality but useless as well). Two had 7 parameters, putting them in violation of the constraint for basic quality principles. I didn't add any tests, so all three were in violation of the test coverage constraint. And finally they all showed up in the list of new methods. It's worth noting the yellow circle at the bottom right as well - the yellow means that warnings were encountered when running the CQL portion of the analysis. Green would be good, and red would mean I have some bad queries that can't be run.

Double clicking a row in the CQL Explorer will take you to the CQL Editor - from here you can view the results of the query, and the CQL it contains. From there you can easily navigate to the method definition in your source code by double-clicking.

One of the things I really like here is the comments in the built-in queries. They contain numerous links to metric definitions on the NDepend website, and sometimes even links to blog entries where the lead developer, Patrick Smacchia has explained features in greater detail. I really like this form of documentation, it makes it easier to keep up to date and also minimizes what needs to be stored on the user's computer.

What I was happiest to find in the VS integration is the ability to superimpose CQL results onto the metrics view. The metrics view consists of a grid where each block represents a unit of code (type, method, etc...) and they are sized according to their value for the metric in question. When running CQL queries, the units of code matching your criteria are highlighted, giving a great visualization of how much code exhibits the properties you are looking for.

The selected query (about types having too many efferent couplings) is the currently selected query, and I had moused over the QueryParser class to highlight it in pink and show the metrics summary on the right. I find that having this built right into visual studio really helps me figure out where to focus my refactoring energy.

It looks as if a lot of effort went into making the CQL validation phase fast:

CQL rules validation phase is fast. The performance challenge was to make this happens almost instantly to avoid slowing the developer machine. Hopefully for a large 100K Lines of Code application, code gets re-analyzed and 200 CQL rules can get checked, all within 3 seconds after the (re)compilation of one or several .NET assemblies. These fast performances were made possible thanks to the development of a new technology of incremental code analysis. With incremental code analysis, only modified code gets re-analyzed. I can attest that this was extremely challenging and complex development!

From what I've seen, the effort's been a success. I've mostly been using it to look at different versions of Lucene.net, as I've got some work to do to get some of my code to build in VS2010. For this size codebase (~23k lines) the analysis is completed very quickly, even if I disable the incremental analysis. The CQL validation portion completes almost instantly, and memory usage doesn't seem to get out of hand even when keeping VS open for days. I'd imagine if your computer can handle running Visual Studio to begin with, it won't have too much trouble with the NDepend integration. I could see some of the VS add-ins that don't play well with others causing issues, especially with a very large codebase, so I hope to go back and test with a larger codebase and some other add-ins installed eventually.

Most of the other NDepend goodies are available in VS now as well (Dependency Graphs, Test Coverage Analysis, Class Browser, etc...) but I won't get into all that here. I really see CQL as the app's killer feature, and that is what I spend the most time thinking about. There is a good overview of the app's capabilities here if you'd like to read more.

Original post blogged on LessThanDot.

You need to know a lot when you want to be a developer.

Christiaan Baes (chrissie1) — Fri, 30 Jul 2010 08:06:35 +0000

I was looking at the stack they used for the new channel9 website.

MVC 2.0, Unity (from P&P), NHibernate, Fluent NHibernate, Memcached, Enyim Managed Memcached driver, Azure – Fabric, Storage, Diagnostics, SQL Azure, xUnit (testing only), Live ID, Spark View Engine, Akismet (spam filtering service), AntiXSS, Tinymce, jQuery, and Silverlight.

You can watch the video by Mike Sampson on how they built the new site if you want to know more.

But have you counted the number of frameworks/technologies required for such a site? I was going to say simple site but I don't think that exists anymore. I count 16 (I didn't count Storage and diagnostics). And I'm sure they forgot a whole bunch that you just take for granted.

Then I started counting the stack I use at work. In no particular order.

nHibernate
WPF
Winforms
Log4net
NUnit
Rhino mocks
SQL-server
StructureMap
Microsoft reporting
ITextsharp

Seems a whole lot less than what you need for web development. But the problem remains the same. You have to know all those things well to make them work together well and you need to know the tools that you can use to make them work together well.

Here are just a few of the tools I use to make all that work well. Again in no particular order.

SSMS
Redgate toolbelt
NDepend
Visual Studio 2010
Ncover/DotCover
Resharper
Finalbuilder
An enormous amount of third party controls
NHProf
SQLCop ;-)
VisualSVN
VMWare
Windows
Oracle
dotTrace
Sharepoint

All these tools and frameworks you have to know and know well. Some you will only use once in a while but you have to know they exist and you have to know how to use them. And lets not forget all those things you tried and rejected. I think those make an even longer list.

I'm sure you could spend days and weeks just getting to grips with NDepend but you just don't have the time. And by the time you know all the above it is surely outdated.

And then you also have to know something about hardware.

But does that mean that the life of a developer is hard and harder than most other professionals? No, it isn't. I do the developing thing part time and the other part of the job takes just as much effort and commitment to do right as the development side of things. It's all part of the job. But you could do what you do with a lot less effort and commitment and some people do. But that doesn't make you better or worse then anyone else just different. Different people have different priorities.

But I can sure understand why people don't want to learn new things and just do things with what they have now.

I'm just in it because I like to do what I do. As soon as it stops being fun I will move on. But for the moment I still have lots to learn and I'm not as young as I used to be ;-) so it takes a little longer.

Original post blogged on LessThanDot.

Model-View-Presenter: Looking at Passive View

Eli Weinstock-Herman (tarwn) — Thu, 15 Jul 2010 09:44:12 +0000

Model-View-Presenter is an architecture pattern that defines a structure for behavior and logic at the UI level. M-V-P separates the logic of the presentation, such as interacting with back-end services and the business layer, from the mechanics of displaying buttons and interface components.

I often build small projects to help understand and grow my skills as a developer, architect, and all-around technologist (as may be apparent from the wide range of topics I post on). Today I worked with a combination of Visio and Visual Studio to build a sample project to play with the Passive View concept and to help grow my own understanding of the concept. This post will cover the Visio side of my learning-curve.

You can read more about Model View Presenter at Wikipedia and MSDN. Perhaps the best information can be found on Martin Fowler's site, where he has separate write-ups on Passive View and Supervising Controller.

Note: I know some people were waiting for another Virtual Lab entry this week, and here I am writing about Architecture instead. Don't worry, the virtual lab series will continue, I just felt like doing a write-up while I was playing this past weekend.

Passive View

Passive View is a subset of the Model-View-Presenter pattern. In Passive View, the interface is responsible for handling interface-specific logic, such as figuring out how to put a value in a textbox or react to events from button clicks, but all actions and logic outside of the raw UI are sent to the Presenter to execute or manage. The Presenter is responsible for calling business methods in the Business model and updating the data that is available in the View.

Basic Model-View-Presenter Diagram

From the outside in, the architecture for Passive View looks something like this:

UI - The User Interface reflects what is going on beneath it by implementing one or more View interfaces
Presenter - The Presenter receives interactions from the UI or Model and updates the Views it is attached to
Model - The model is a facade or black box in our diagram, behind which is a business logic layer and data layer

In a flat architecture we would collect data from the interface, perhaps do some business and data validation, and then save it directly to a database using stored procedures or inline SQL. Defining a data access layer (or data model like entity framework) allows our application to operate on cohesive, defined objects that are meaningful to the application and stored and retrieved consistently. Defining a business logic layer allows us to centralize business rules that operate on entities in our application in a manner that is consistent with the business and internally consistent in the application, minimizing the risk that occurs when making changes to the business flow. Separating the logic of populating inputs and responding to button presses on the UI from the information being communicated to the end user and conceptual responses to their input allows the system to interact with the user consistently across any number of interfaces into the same application.

The definition of each level increases our ability to automate testing and supports greater Separation of Concerns.

Implementing a Sample Project

My learning exercise has been the the creation of an ASP.Net search page that allows an end user (customer) to search for finished products from the AdventureWorks sample database. The architecture and design decisions were done as an exercise in Visio using simple shapes and layouts.

My example application has several functional and non-functional requirements:

Functional - Display product number, name, list price, and available quantity in tabular format
Functional - Provide a basic search input and button to search product names
Non-Functional - Implement an M-V-P pattern - Obviously the purpose of this whole exercise
Non-Functional - Use a simple model stack that can be easily replaced with a Service-Oriented one at a later time
Non-Functional - Build with the idea that we will later create a Silverlight or WPF front-end
Non-Functional - Make pretty pictures for article

My unwritten, final requirement was to finish the whole thing in half a day, though luckily I didn't define whether I intended that to mean 4 hours or 12.

Initial Architecture

To start I created a diagram of the application architecture:

More Extensive Model-View-Presenter Diagram

The purple layer is my presentation layer, which reflects the View. The blue layer is my Presenter layer which contains the logic for interacting between the end user and interface as well as a definition, or contract, of the information available in the View. The Green is the Model (or is behind the model, depending on your viewpoint) and exposes business functions and data entities for the Presenter to interact with.

Class Layout

Once the high level diagram was completed, I could approach the task of creating some base classes and interfaces to use in implementing the project.

Model.IModel - Generic Model Interface to expose business calls to Presenters
Presenter.IView - Generic View Interface that all Presenters can interact with and all screens implement
Presenter.BasePresenter - Generic Presenter class that all Presenters will implement

To keep the project to a single morning but also allow the ability to come back and build a more architecturally sound solution, I implemented the Model in a very basic fashion that was referenced locally by the Presenter project and makes direct calls to SQL Server using ADO and parametrized, inline SQL. This buys me the benefits of having a well-defined Model (via the interface) but allows me concentrate my time and effort on the learning part of the project (ie, the M-V-P interaction and structure). Defining the model interface also leaves me open to come back and replace it with better separated code and the ability to create a model that acts as a facade to a service stack, instead of local DLLs.

Model.BasicModel.Model - Basic implementation of a model that will interact with AdventureWorks on SQL Server
Model.Entities.Product - A Product Entity that can be communicated between an IModel instance and Presenter
Presenter.ProductSearchPresenter - A Presenter to manage product search interface
Presenter.IProductSearchView - A view of the data involved in a product search
ProductSearch.aspx - A web page that implements the IProductSearchView and interacts with the ProductSearchPresenter

My final Visio diagram ended up looking like this:

Diagram of Example Application

In this case the left side represents basic components (bases classes and interfaces) that are used to define common structure or contracts on the right side.

The Code

For the purposes of the example project, my view has properties for Search Text, a Search Count (number of results), Results (a generic list of the Product entity), and a boolean indicating whether there are results to display. My Web Form implements these properties, tying them to elements on the screen.

Code: csharp

public partial class WebForm1 : System.Web.UI.Page, Presenter.Views.IProductSearchView {

Presenter.ProductSearchPresenter _presenter;

protected void Page_Load(object sender, EventArgs e) {
btnSearch.Click += new EventHandler(btnSearch_Click);
rptProducts.ItemDataBound += new RepeaterItemEventHandler(rptProducts_ItemDataBound);
_presenter = new Presenter.ProductSearchPresenter(new Model.LocalModel.BasicModel(System.Configuration.ConfigurationManager.ConnectionStrings["AdventureWorks"].ConnectionString), this);
}

void btnSearch_Click(object sender, EventArgs e) {
this._presenter.ExecuteProductSearch();
}

string Presenter.Views.IProductSearchView.SearchText {
get { return tbSearch.Text; }
set { tbSearch.Text = value; }
}

int Presenter.Views.IProductSearchView.ResultCount {
set { lblResultCount.Text = value.ToString(); }
}

List<Model.Entities.Product> Presenter.Views.IProductSearchView.SearchResults {
set {
if (value != null && value.Count > 0) {
rptProducts.DataSource = value;
rptProducts.DataBind();
}
}
}

bool Presenter.Views.IProductSearchView.DisplayResults {
set { tblResults.Visible = value; }
}
...

As the presenter populates properties in the view, the information is automatically reflected on the page. The actual logic of how the business functions are called and populate those properties are neatly packaged up in the Presenter and View interface and very little logic occurs in the actual web form.

Code: csharp

public class ProductSearchPresenter : BasePresenter {

protected Views.IProductSearchView _view;

public ProductSearchPresenter(Model.IModel model, Views.IProductSearchView view) : base(model) {
this._view = view;
this._view.ResultCount = 0;
this._view.DisplayResults = false;
}

public void ExecuteProductSearch() {
List<Model.Entities.Product> results;
results = this._model.SearchProduct(this._view.SearchText);
if (results.Count > 0) {
this._view.ResultCount = results.Count;
this._view.DisplayResults = true;
this._view.SearchResults = results;
}
else {
this._view.ResultCount = 0;
this._view.DisplayResults = false;
this._view.SearchResults = null;
}
}
}

To create a unit test, we define a simple view that implements the view interface, execute the presenter logic, and verify the properties are populated the way we would expect when the same presenter calls are made from the interface.

Extending the Architecture Further

Extending the application to display product search in a different manner would only require the addition of a new interface that also implements the Product Search View. A Silverlight front-end would only require creating the basic project, implementing the product search View, and wiring the new interface controls to the view properties. To replace the direct mode reference with a service reference, we could create a service facade that implemented the IModel interface, connected to a local or remote WCF service behid the scenes to handle the real model logic. And finally, instead of counting on our QA department to test all of the application interactions, we can create unit tests directly against the Presenter and Views to ensure that all of the interactions below the top surface of the application are happening consistently and to our expectation.

Your Turn

Getting this much of the architecture working is a good first step. I took a number of shortcuts on the BasicModel class in my example, but I now have a functional Model-View-Presenter application to play with. Hopefully there was enough information in the article to interest you in trying this out on your own. I urge you to read the articles linked in the top of the post (or several more in my Model-View-Presentation bookmarks) and come up with your own diagrams and sample project. Even doing a small project will force you to run into questions and considerations you wouldn't have had by simply reading about it, not to mention unrelated tidbits you will pick up along the way (for instance, I also learned about ObservableCollections today).

Original post blogged on LessThanDot.

MSDN giveaway winners

SQLDenis — Fri, 09 Jul 2010 15:23:42 +0000

The winners of the MSDN Ultimate subscriptions are Emtucifor and Shawson. Originally we were going to select the winner based on comments, we also only had one subscription. Then Ted Krueger donated one of his subscriptions, there was only one comment that stood out and it was Emtucifor's. The comment is below

I have always dreamed of owning my own software company. A few years back I started doing some database development on the side, but then I got married, had a son, and began having some health challenges which together halted what I'd been doing.

But there is a special opportunity coming to me this Saturday to get back into the swing of things: my wife and son will be leaving the country for six weeks. I had already been planning to devote myself to developing one of many application ideas, but now:

1) I would use MSDN Ultimate Subscription to build a secure server/file transfer/fetching/archiving/processing/reconciling/user worklist managing/cross-platform system (uncannily and quite coincidentally, exactly what is desperately needed at a company I know). It will help other people's lives in several ways: filling a need that many companies are bound to have and for a low price (as initially I will need to build market presence more than profit); increasing the number of small software development businesses out there, proving again that it can be done and providing inspiration for the masses; getting a family's Dad home so he can spend more time with them. If it helps, I'll blog about the development and growth of my application and business.

2) The functionality I would use that is unique to VS Ultimate is Architecture and Modeling. I believe in "starting the way you mean to finish." So even though I'll be a one-man shop at first, that means test-driven design, agile development, automated regression testing, version control, good backups, and all the infrastructure needed to hire employees when the time comes without having to change much. I'll be eager to add the layer diagramming abilities of VS Ultimate to this mix.

3. Blogs and technical community activity:

http://blogs.lessthandot.com/index.php/All/?disp=authdir&author=71
http://stackoverflow.com/users/57611/emtucifor
http://tek-tips.com/userinfo.cfm?member=emtucifor
http://squaredthoughts.blogspot.com/ (a bit out of date but still representative of my work)
not to mention http://forum.lessthandot.com/memberlist.php?mode=viewprofile&u=98

Not only did we think that the comment was good but apparently someone else as well, he actually used parts of it as his own in Andy Leonard's giveaway here: http://sqlblog.com/blogs/andy_leonard/archive/2010/07/03/a-visual-studio-2010-msdn-seeding-card-giveaway-contest.aspx

We decided to raffle of the other subscription by vote..so I already started to put the polls together. During that time David Taylor left a great comment

1) Why do you need this - Because I am a poor, seriously underpaid SQL Server 2008 DBA/Developer who can't even afford VS2010 Pro!
What are you going to build with this - I am going to build practice apps to learn from, as I am relatively new to development
will it help other people's lives? It is my hope that training myself will get me into a job in which I am helping other people.

2) What specific functionality that is only part of Ultimate are you going to use? - Architecture Explorer

3) You need to have a technical blog and provide the URL to that blog, if you are an active member of the technical community (stackoverflow, msdn forums etc etc) then also include those links. - http://dyfhid.wordpress.com. Also, I am the Volunteer Coordinator for PASS' APplication Development Virtual Chapter, located at http://appdev.sqlpass.org

Now we had a dilemma, do we yank the poll and give the award to David or not? We proceeded with the poll. Wouldn't you know it, David won Andy's contest, because of that I emailed him and we decided that he wouldn't be eligible here.

Here are the results of the final poll: http://forum.lessthandot.com/viewtopic.php?f=121&t=11550

Congratulations to both winners, hopefully they will let us know in a couple of months if this has indeed made their developer life easier

Congratulations again from all of us at Lessthandot.

Original post blogged on LessThanDot.

Final vote for the MSDN giveaway

SQLDenis — Fri, 09 Jul 2010 10:47:59 +0000

We are giving away 2 MSDN ultimate subscriptions, for the original announcement see here: MSDN Ultimate Subscription giveaway

We did not include David Taylor because he won this morning at sqlblog: http://sqlblog.com/blogs/andy_leonard/archive/2010/07/09/and-the-winner-is.aspx and he was fine with us taking him off.

Here are the 6 finalist, I picked the top 3 from each group that people had a chance to vote on: http://forum.lessthandot.com/viewtopic.php?f=121&t=11533 and http://forum.lessthandot.com/posting.php?mode=edit&f=121&p=56403

This vote will end in 3 hours, at noon Eastern Standard Time (GMT -5)
Here are the candidates and their replies

Shawson
1) I will build games with it. I am getting into indie game development with a view to moving from my current profession of web development into games development. I'm just completing a browser game using just javascript and the new html5 media and canvas elements, to see whats possible in the near future without flash! So i will be using it for c# xna, as well as javascript and html. I also wish to do some work with the new odata stuff, to deal with the highscores, as well as a chance to use the latest iteration of the entity framework.

2) The historical debugger would be awesome, especially when debugging realtime code which is changing all the time, as you see in game loops. I'm also interested to see the designer tools letting you visually build systems with UML and have the basic structural code generated for you.

3) http://www.shawson.co.uk/codeblog
http://8weekgame.shawson.co.uk
http://stackoverflow.com/users/192305/shawson

Roger Pence
1. I need this subscription to help me simplify healthcare-related software. Simpler software is easier to use correctly. Improve record keeping improves lives.
2. Better ALM, Architecture and Modeling, and DB support would help make me a better developer.
3. http://rogerpence.com/blog

Naomi
http://blogs.lessthandot.com/index.php/All/?disp=authdir&author=218
http://beyondrelational.com/blogs/naomi/archive/2010/03/22/why-left-join-doesn-t-bring-all-records-from-the-left-table.aspx

Active answerer at many technical sites:

MSDN
ASP.NET/Forums
universalthread.com
tek-ips.com

Dave Ballantyne
1) I am now running my a user group ( http://sqlsocialkent20100818.eventbrite.com/) , so would like to speak with more authority about DataDude and such issues.
2) Intellitrace looks good as do the architecture components.
3) http://sqlblogcasts.com/blogs/sqlandthelike/ , active in SqlServerCentral and msdn online forums, User groups twitter, fingers cross for a session at SqlBits

dennydotnet
1. I am working on a startup idea. Currently using ASP.NET MVC 1, VS 2008. Would love to move up to VS 2010 / .NET 4 and latest MVC version. Idea is around helping people find great deals on products/services.

2. I could certainly use the feature packs / architecture / modeling tools and the latest Visual Studio 2010. I'd also be interested in playing with Expression Studio Ultimate. Any Windows Phone 7 stuff (if available) would be nice to experiment with as well.

3. http://www.dennydotnet.com
http://stackoverflow.com/users/113474/dennydotnet
http://forums.asp.net/members/superghost.aspx

Meile Zetstra
1) I'm currently running VS2010 Premium, but it just lacks the of all the architectural stuff that's in the Ultimate version. I'd use that to build some tooling or VS extension that revolves arround code inspection, code reflection and generations of diagrams from existing assemblies.

2) The architecture and modeling stuff. Also the use of the just released Visualization and Modeling Feature Pack is on my wish-to-use list.

3) My blog: http://blogger.xs4all.nl/mzetstra
http://sequenceviz.codeplex.com

You can vote here: http://forum.lessthandot.com/viewtopic.php?f=121&t=11550

Original post blogged on LessThanDot.

Vote for the MSDN giveaway winner

SQLDenis — Thu, 08 Jul 2010 15:17:37 +0000

As you might know we are giving away 2 MSDN ultimate subscriptions, for the original announcement see here: MSDN Ultimate Subscription giveaway

We decided on one of the winners already, for the other winner I decided to pull a Pontius Pilate on you so that nobody can blame me that they didn't win....yes I know it sounds like a cop out...that is because it is...and this is more fun.

Here is what I did..I picked the 20 best answers and put them in 2 groups of 10. In each group 3 people will advance to a final vote, you can vote for two people. The results of those two polls will be looked at tomorrow morning around 9 AM EST and then a new poll will be created with the final 6 contestants.

Make sure that you look at every person's answer, also check their links so that you can decide who should advance.

The group A people can be found here: http://forum.lessthandot.com/viewtopic.php?f=121&t=11533
The group B people can be found here: http://forum.lessthandot.com/viewtopic.php?f=121&t=11532

Good luck everyone

Original post blogged on LessThanDot.