Starting the Holy Wars on ESB? Who brings the best package deal to the table i.e IBM, MSFT...... Look at this way " A mid sized cash crunched has to implement ESB what are the options it has" or rather who should it be married to. Any thoughts

Tons of ESB from various players are in the market IBM, MSFT, Oracle , SAP but remember its a cash crunched company which requires the ESB. I can provide more details on the technical side. The idea is to throw open the debate to get various perspective.

Well, let's forget for the moment that the question was raised by a Microsoft employee, and focus more on the general principle of the question... What should companies look for in an ESB? In my brief answer, I aim to get across the point that asking which ESB vendor you should be "Married to" is very much the wrong question to ask, and to consider that there is more to solving this problem intelligently (and thus cost effectively) than simply 'buying technology'. No single technology will solve your problem out of the box, so be careful not to expect or ask it to do so.

Here's the Answer in full:

As always... understand the problem first...

Step 1: Understand the problem you are trying to solve (e.g. Imperatives, Current Pain Points, As-Is Landscape, Use Cases, etc)
Step 2: Design a Target Architecture (NOT based on a particular technology) that solves this problem, including reference models, design patterns, architecture paradigms, principles, etc
Step 3: Create a high level Roadmap to get from your current environment to your target architecture
Step 4: Now... Search for a technology that will support this need.

I would highly recommend not starting by talking to vendors about how they will solve your problem. Strangely enough, their products always have the answer :-)

ok.. let's look at what integration is all about:

Openness
Interoperability
Standardisation
Flexibility

- This is where you need to really think about why you have an integration problem in the first place... closed, insular approaches to application design. This indicates that we want an OPEN and INTEROPERABLE environment which is based on STANDARDS to give you FLEXIBILITY.

The problem with a lot of products on the market if not understood or used properly is that you can very soon become (in your words) "Married" to that vendor's solution. This destroys the core principles of what integration is all about - you are simply adding the same problem on top of the original problems.

In many ways the open source integration solutions (incl. ESBs - e.g. SOPERA (commercial and more advanced version of Eclipse Swordfish), Apache FUSE, JBoss,etc) lend themselves to these principles very well, however this does not guarantee they will fit your need, or perform to the level of some COTS/Closed Source options.

Additionally, you need to consider that properly integrating your environment is not just about purchasing / implementing an ESB. This is just one component of the overall solution, which should consider the need for MDM, Data Architecture and Canonicalisation, Portals, Business Rules, Registry/Repository, Modelling Tools, Business Process Management and Orchestration, amongst others. The commercial vendors will be more than willing to sell you complete stacks for these things, but be prepared to do things their way, and have a certain amount of lock-in to that vendor.

As you work there, I'm sure you realise that Microsoft is very Microsoft focused, so is only really sensible for Microsoft only environments. However, your recent deal with SOPERA now gives support for both native .NET and Java environments. Maybe there is hope...

Oracle and SAP have some good concepts, but unfortunately they are again very insular - if you only have SAP then XI will be wonderful for you, same with Oracle Fusion and their AIA offering - extending outside the environment is possible, but often ugly.

IBM is (imho) conceptually better at being application/solution agnostic, as it doesn't sell applications to integrate, therefore focuses on supporting a diverse range of approaches. However, they have a history of a somewhat convoluted and overlapping solution set.

Webmethods provides you with a well integrated set of tools and features, again application agnostic, though it's sweet spot is Process Management, and isn't the most open/standards compliant.

If a company is really strapped for cash... then an open source option may be best. But you may have to work harder to get a complete solution up and running.

You can get all of the components to integrate your environment for free - but you have to implement them, they may not be that pretty/easy to use, or performant, and if you choose not to pay for ongoing support then you are putting your company at risk.

You also have to ask yourself if you are really doing pure SOA, or whether you are needing to do application integration...

above all, the right architecture will win over a technology choice

So, the answer was as you would expect for a general Q&A site - relatively simple/brief (there is a post size limit!) and a little on the generic side to cater for the limited information available.

I touch briefly on a couple of points I want to highlight here.

The first point being the potential benefit but also risk of using Open Source integration technologies. Put simply, open source's core principles are very much aligned with those of integration, and as such makes an excellent candidate for the technology that glues together all of our other bits and pieces of technology. But, as it is in it's relative infancy, there are some potential risks. These are rapidly being addressed technically, and given a few more years of well rounded practical implementations, many of these issues will be addressed. Although I wont go into detail in this post, I suggest having a look at some of the open source options available, including:

SOPERA - very open modular design, with a strong plugin architecture, with integrations to many other software components, including Microsoft .NET - this is the product behind Eclipse Swordfish.
JBoss - a well known middleware/application server vendor with parent company Redhat. A well rounded offering, though slightly immature ESB.

There are others (e.g. Apache FUSE), though I'll save the details for another day - check out the above and see how it may suit your needs.

The second point is the question of whether you are really doing pure SOA or not, or whether you're either doing application integration (not too dissimilar in objective, but different in concept) or just 'web services'. Have a think about that... what exactly is SOA to you? And do you think everyone else thinks the same? Feel free to post your thoughts below...

The full Q&A from linkedin here:
http://www.linkedin.com/answers/technology/enterprise-software/TCH_ENT/441250-2894850?browseIdx=1&sik=1239049806947

• Rome did not create a great empire by having meetings, they did it by killing all those who opposed them.
  
• If you can stay calm, while all around you is chaos… then you probably haven’t completely understood the seriousness of the situation.
  
• Doing a job RIGHT the first time gets the job done. Doing the job WRONG fourteen times gives you job security.
  
• Eagles may soar, but weasels don’t get sucked into jet engines.
  
• Artificial Intelligence is no match for Natural Stupidity
  
• A person who smiles in the face of adversity… probably has a scapegoat.
  
• Plagiarism saves time.
  
• If at first you don’t succeed, try management.
  
• Never put off until tomorrow what you can avoid altogether.
  
• TEAMWORK… means never having to take all the blame yourself.
  
• The beatings will continue until morale improves.
  
• Never underestimate the power of very stupid people in large groups.
  
• We waste time, so you don’t have to.
  
• Hang in there, retirement is only thirty years away!
  
• Go the extra mile. It makes your boss look like an incompetent slacker.
  
• A snooze button is a poor substitute for no alarm clock at all.
  
• When the going gets tough, the tough take a coffee break.
  
• INDECISION is the key to FLEXIBILITY.
  
• Succeed in spite of management.
  
• Aim Low, Reach Your Goals, Avoid Disappointment.

Today I finally got round to doing some admin on my primary home server, which (amongst many things) is my main mail server. After dusting a few cobwebs away, checking nobody had broken any of the furniture and such, I decided to run yum to get the latest updates (it had been almost a month since the last time I checked but still, at least now it was about to be updated to all the shiny sparkly new things that CentOS had released for me…

And it did, it came up with about 5 or 6 updates, including an update to postfix. Great. So, it finished in about a minute or two and I went on about my evening. All was happy in the land of the ignorant.

Ignorant, that is, until I decided to send an email to my dear old mum to give her an update on the list of laptops I was recommending for her (I know, I’m so thoughtful ). At which point my squirrelmail (webmail) froze.. you see, I’m currently re-decorating my study, so have moved my main workstation into another room, and as such it’s not in use.. and my other non-work laptop is, er, recovering from, er, how do you say it? A little ‘rinse’ in the bath… so, work laptop it is, and webmail access only…

I thought maybe it was my damn windows wifi connection at first because it has been such a problem the last few days.. (I can’t wait to get back to my linux environments, this windows only stuff is giving me a serious heart condition.) Anyway, after a prod and a poke it seemed that it might be the mail server…

Determining the problem

So, I checked the mail server logs and discovered…

mmmm… now, that looks familiar.. wasn’t that a problem I faced when setting up this mail server? The problem that centos repositories only provide the standard postfix rpm which doesn’t have mysql support compiled in? But, it has been working for ages.. why is it failing now ? Hold on.. let me check what exactly was updated again recently…

Ahhhh… bugger.

Path of enlightenment

OK… now I’m thinking that some stupid numpty has released the wrong rpm in the wrong repository, because I know that I set things up originally to exclude the main repositories for postfix updates and only get it from the centosplus repository. Of course I did. I’m sure. I think. mmmmm… ok, let me check…

Ooops. Can you spot it? Yep, exactly - every entry apart from the updates was excluding “postfix", whilst the updates were excluding “posfix"..

Fixing the shame..

That, my friend, is the scourge of bugs everywhere.. the humble typo. ah well. Lesson learnt.. now just to download the previous centosplus package for postfix with mysql built in and re-install it (and fix that damn typo):

phew… mail server back up and working again. I thought for a brief second that it was going to be a major problem, but, thankfully, a download + install to rollback and a correction of a typo later.. and I’m back up and running :-) A lesson in scrutiny of important configuration data has been learnt.. until the next time..

We don’t need architects! …Do we ?

It’s important for us to remember that we develop software to “do something” that we want it to do. In which case we need to continually review the success of that software or solution by evaluating it’s ability to meet our needs. But who gets to say what those needs are, and how well they are met? More importantly, what makes IT solutions good or bad ? Is it how big they are ? Or how easy to use they are ? Or maybe how efficiently they operate ? Or what about how well measured and managed they are ? Or even how easy it is to change the way they work ? What about all of the above, and much more…?

Getting to a point where a) you understand what success looks like, and b) you have a solution that delivers that success, can be achieved by using an Architect to facilitate and coordinate the construction of a design and resulting solution. But that’s not always what happens. Apart from the many varying definitions of an architect’s role (and the different types of architects.. e.g. Enterprise Architect, Solutions Architect, Application Architect, Technical Architect,etc), there’s also often a perception from developers that architects are generally away with the fairies, dreaming up blue sky designs that are just not grounded in reality or worthwhile bothering with. Additionally, a lot of businesses (too many unfortunately), just don’t understand the value of using an architect… Why pay someone to design the solution, when you’ve got business systems analysts and developers that can work it out between themselves ?

Winchester Mystery House, San Jose, CA

Why indeed… well, I thought it may be useful to post a little analogy with the ‘real world’ - where architects and builders have been working together for many years to produce wonderful (and not so wonderful) buildings.

Firstly, let me introduce the Winchester Mystery House, of the Winchester Guns family fame:

Functionally Rich

• 160 Rooms
• 47 Fireplaces
• 17 Chimneys
• 6 kitchens
• 10,000 windows

State of the Art Technology (at the time)

• Wall Insulation
• Push button gas lights
• No-clog sink patent
• Intercoms
• Modern Heating & Sewage

Total Cost

$5.5m (~$300m today) over 38 years (1884 - 1922)

But… No Interoperability

• 65 doors to blank walls
• 13 abandoned staircases
• 24 skylights in floors
• Rooms “build-around” other rooms
• Enough keys to fill 2 large buckets

Ouch.

Due to the division of labour

• Builders: 147
• Architects: 0

Sound Familiar ?

We’ve all worked on a project with something like this. Usually because the system wasn’t really ‘designed’ it just ‘happened’. Maybe if we spent a little more time designing, and a little less building things would work a little better. I bet they had some very talented builders working on this too - but their perspective wasn’t to make a solid overall design, it was to simply get on and do what they were tasked with.. sound familiar ? How often do users come to a technical forum and ask for a solution to their problem to be faced with answers like “you need to normalise your database” etc, to which they simply retort “that’s not part of this project” ? Too many, but sadly understandably so.

A favourite Project Manager quote is “Failing to Plan is Planning to Fail". And they are generally right - not having a consistent, well thought out and planned design can lead to very complicated and difficult to manage, change or even use software. Most architects will have come across solutions or applications that weren’t ‘designed’ and just evolved, and will know just how much of a problem those applications really are. This is Software’s Dirty Little Secret - many systems have just “happened” without real thought to the design of the software. And for that matter, haven’t thought about the application as part of a composite solution either - this is actually quite rare, even today with all the SOA hype of the last few years.

The Architect’s hand

Sometimes it might seem strange to developers that an architect wants to make a project use standards compliant technologies, standardised, re-usable or existing components, such as a Business Rules Engine, Reporting & BI, Service Bus, Messaging Gateway, Portlets, etc - but this is what architecture is all about - not thinking about the one, single application or issue, but thinking about it’s co-existence in a much bigger world - not just the here and now functional requirements, but tomorrows new direction and demands. Simplifying and standardising, Rationalising and Re-Using, Conceptualising and Componentising… the best architectures are easily understood, simple solutions that achieve a kind of elegance in concept and practicality in implementation.

There are several methodologies to use for developing an architecture - personally I use TOGAF which is more for generalist Enterprise & Solutions Architecture development but the ADM can be applied in concept to designing most solutions/systems/applications.

So we do need to “do architecture” then?

In short, Yes. But how you actually “do architecture” depends greatly on the context. There is a simple lesson here - architecture is all about simplifying and standardising the problem domain to provide an efficient, re-usable and consistent answer. The important difference between a formal “architect” and a “builder” is not of technical expertise, but of perspective, so don’t assume that your most expert Java Programmer is naturally the architect to your solutions. This doesn’t mean that an architect of all types shouldn’t possess good technical abilities and experience.

There is a nice overview of the different types of Architects at the Microsoft Architecture Journal (which is often surprisingly not MS biased) written by IASA Sweden, however this misses out the technical/infrastructure architect from their Roles diagram, which is a key architect in any IT implementation. Also, another article on the Architect Journal talks about what an architect actually is, however is a little Software Architect focused. Both interesting reads, especially for non-architects to understand how to ‘use’ an architect (well, work with one effectively at least ).

Some Exceptions..

There are times when a formal architecture process is more of a hindrance than a help, and that is in R & D contexts where rapid prototyping and proof of concept solutions are all about unrestricted creativity - in these cases architecture experience and knowledge is a good backdrop to help organise and focus ideas and channel that creativity into creating something unique. (But… Remember that this is not production ready, it is a prototype in need of architectural design to create a production capable solution. This is where too many rapid prototyping approaches get a bad name.)

One thing I will write about soon is the risk of ‘over-designing’ a solution, because sometimes it is easy to get carried away with design-pattern addiction and framework-itis and actually miss the whole point of the exercise. Doing ‘too much’ architecture can be as bad, and even worse than doing none.

And Finally…

So… Yes - Sometimes it’s nice to just set out in your car and drive wherever the wind takes you until the fuel runs out. But other times, and almost always when it is on ‘business time’, wandering aimlessly in the hope of finding something that might be useful is just not acceptable. Having that vision, aspiration, direction and roadmap an architecture can bring to your project might seem like a chore and “impossible” to achieve to some people.. but how else will you get there, without ever actively and knowingly moving toward it ? Chance ? Good Luck with that…

So, the next time you think about just winging it, and not bothering with architecture, just think about your software product as your own house that you wish to build - would you really just start building ? Or even just draw up your own plans on visio and ask the builders to ’sort it out’ ?

N.B. In the Winchester House example, strange as it may seem, the design actually met the brief - the needs of the owner weren’t really to design an architecturally great building, but just to have something creative to do.. it doesn’t matter that it wasn’t consistent, or standardised or even usable - it was the pleasure of doing that was needed, not the objective of greatness in form and function. But have a think about that… when was the last time your boss suggested you design something for the fun of it, at the company’s expense, regardless of whether it results in some sort of throw-away chimera ?

A Version Control System (VCS) is an essential tool for a development team (or even individual developer) to manage their code. The more it can allow multiple developers to work together easily on the same code and manage the various states of that code, the better.

One of the most popular Version Control systems in the past was CVS, however in recent times this has been superseded by Subversion (SVN), which is a highly popular VCS that is well supported by a variety of client tools and is often embedded within software configuration management applications to manage file versioning (e.g. trac, collabNet, etc)

What is sometimes an issue with SVN is the ability to do local commits for each developer before a full commit to the main branch, making the main branch often unreliable and full of partially complete work, or local developers with limited versioning control etc.

Bazaar

One VCS tool that tackles this problem is Bazaar, which offers no less than 7 different Version Control workflows depending on your particular scenario.

Bazaar is developed and used by Canonical (the Ubuntu people), and has recently claimed Sun’s MySQL as a new user. Bazaar is one of many VCS tools, but it’s flexibility in workflow patterns is certainly something that makes it an interesting tool to consider.

Further Info

• General Information on Revision Control
• Comparison of Revision Control Systems
• Bazaar Documentation

A Canonical Data Model is most simply explained as a standardised way of representing data that generically applies to multiple systems and data models - kind of like the master data model, but for semantics aswell as content. It can be used as a data model standard template for new systems development, or as an intermediary serialised data format - which is the aspect I’ll focus on mostly here.

Here’s an Example of using the CDM as an intermediary data format to integrate six systems with different data format requirements: Canonical Data Model

In this example, where serialised data is being passed between systems, it uses the standardised format as a middle ground, meaning that each adapter only needs to be able to translate into or out of the CDM format (rather than each individual target/source) for it to be able to understand (and therefore integrate) with other systems ‘talking’ in the same context (e.g. a specific functional domain, like shipments, invoices, orders etc). This creates semantic interoperability between the system by abstracting the differences at the perimeter of the integration solution, and re-using existing components.

So what ?

It’s not always clear when first thinking about this design why it is so important to the implementation of upstream architectures such as SOA, or why this has such impact on everyday integration.. even the more traditional hub based point-to-point integration patterns. But hopefully that realisation is starting to emerge, because it really is key to building truly flexible, seamless and invisible integration solutions.

Some of the key objectives

• Maximise Re-Use of Components
• Minimise Redundant Code
• Facilitate the concept of ‘Plug-And-Play’ interfaces
• Standardise knowledge, terminology and domain concepts (talk the same language)
• Facilitate SOA through an information architecture (IOA) foundation

Some of the Key Benefits and Advantages

Connect Once, Integrate Many - once connected to the information network via the canonical adapter, you can communicate with any other connected system as long as it is in context (e.g. order requests aren’t likely to work with Proof of Delivery)… but having said that, things do become somewhat adaptable, even out of context… for example: an order quite often contains customer information and product information, as well as being an outline of the shipment. With this in mind you *could* (and that doesn’t mean *should* :-D ) feed a customer master or CRM application with the customer information, a Product Master or Stock Management application with the product information ‘just-in-time’ for the order to be processed should the information not have been provided before (unlikely for the product master, but you get the idea). You could even feed a shipment scheduling system, shipment event tracking system and plenty of others just from that single message (the most valuable might be your data warehouse).

Here’s a comparison of point to point integration versus a canonical model:

CDM v Point to Point Comparison

As you can see, the point to point method (bottom image) creates a LOT of components and complexity. Taking this example and doing the Math:

Point-To-Point Integration for 3 sending systems and 3 receiving systems:

3 (sources) * 3 (targets) = 9 (components)

CDM Based Integration for 3 sending systems and 3 receiving systems:

3 (sources) * 1 (adapter - the CDM is now the virtual target) + 3 (targets) * 1 (adapter - the CDM is now the virtual source) = 6 (components)

Or put more simply:
PTP Integration: sources MULTIPLIED by targets
CDM Integration: sources PLUS targets

This starts to demonstrate itself very quickly when you look at 10 sources and 10 targets… would you prefer to build 20 components, or 100 ?

Standardised Information Architecture Foundation - the concept and importance of an Information Oriented Architecture (IOA) should not be understated. Architecture within an IT environment is mostly about abstraction, generalisation and standardisation of concepts to deliver sustainable, re-usable solutions and technology. One of the most popular architectures in previous years has been the N-Tier architecture where layers of the solution are packaged into abstracted levels that are ‘theoretically’ interchangeable etc. Furthermore, the more recent Service Oriented Architecture (SOA) expands / explodes that concept somewhat (there’s a bit more to it, but for examples sake..) into a distributed network oriented environment, with the addition of a bus to provide the glue that binds all the interchangeable parts together. In both of these concepts, information should be standardised, or if not possible (e.g. from multiple sources, which SOA is very prone to) then to abstract the differences between the data sources and present (e.g. via the Data Access Tier, Business Data Objects, Composite Services etc) a standardised version.

There are products that will help to do this on a READ basis - generally called data federation, it allows multiple sources of data to be combined into a single ‘view’ despite the underlying technologies (e.g. RDBMS, XML, CSV, Web Service, etc). This is great, and a useful tool, but it is not enough to satisfy the needs of a true SOA environment. There are also technologies that will help do this on a more general basis, which are much more applicable, such as business data objects, which can be used to ‘mimic’ a SOA like environment without all the standards stack. Then you have the SOA concept, which implements data and functionality as services, but really is founded on those data sources, and their role within the solution (sometimes data sources are not the authority on the data). Hence, a solid information architecture is the foundation of your SOA.

Whichever way you look at it, having this standard model helps to bring your enterprise solutions together whilst minimising diversity and maximising re-use.

For example, you could implement a Data Warehouse by simply adding a canonical adapter for the data warehouse feeds, and directing existing data (coming via other canonical adapters) to this point - if you have 20 or so systems that are already in your network you can start to see how this would make your life easier !

It also helps when designing and developing new applications & data sources, as you can simply implement the CDM template, and you have 80-90% of your data model (there are always application specific data tables required by a system).

Some Things to Consider About the Design of a CDM

• Version Control needs to be closely managed
• You should be able to change the CDM Format without ‘breaking’ existing adapters - otherwise changes will become more and more of a risk/impact
• Data content (as well as the syntax and semantics) should be standardised within the CDM boundaries (transformed on entry) - this is where Master Data Management becomes essential - for example, you want to ensure that timezones are clearly included with time/dates, customer IDs are the same, country codes, currency codes and so on are all the same
• Technology needs to support then dynamic and flexible nature of this - XML is a prime candidate, coupled with XSLT, Xpath and Xquery
• Data Models should be based on REAL LIFE - not on existing applications or the buzz-de-jour - speaking to the business is an essential part of building lasting and truly representational models
• Use Standards where possible - but don’t think there is an off the shelf model and solution that will fit your business perfectly. Things like BODs are interesting and useful, but not necessarily what you need
• Consider the data serialisation approach - message centric (like an EDI Message) or entity centric (like a database) - how normalised do you want that serialised format ? For example a Customer would appear in both orders and invoices (plus many other things) - do you want ‘Customers’ to be a separate entity, or an embedded entity ? An entity centric approach is more appropriately served by developing a Semantic Ontology, using technologies such as RDF and OWL.

So, what next ?

Delivering a CDM can be challenging - especially when delivered as part of a full information architecture overhaul. My advice ? Take little steps.

Start with a vision - think about (and ask) what the business really needs from information, then conceptualise, agree and communicate it.

Then (ideally) you want to define an information architecture strategy as thoroughly as possible upfront, so you have a good framework to work from. Try to be as far reaching (in terms of your business units etc) as possible with the analysis for this, but don’t get bogged down in specifics - just get the framework for how things should be done and the strategy around managing, governing, delivering etc.

Once you have a good framework and strategy, you can start analysing the detail and pulling together a Requirements Analysis Spec and the Architecture/Design for the solution. BUT.. this bit should be done in small manageable chunks - that way a) you minimise risk of failure b) minimise impact if things do go wrong and c) show fast, measurable results to encourage people to continue to take-up this strategy. I’ll post again soon with more detailed steps to this approach

I hope this has either inspired you to look at implementing a Canonical Data Model as part of your enterprise Information Architecture strategy, or enraged you enough to think of some better way…

Adding a range of numbers

Adding numbers is easy. Very easy for programs, right? How about adding up a range of numbers? 1 to 5 maybe ? 1+2+3+4+5 = 15 … easy !

What about adding up 1 to 100 ? not so easy in your head, but with a little code this shouldn’t be a problem. In fact a lot of programmers would approach it like the following:

Great - it gets you the answer that you wanted. Now what about 25 to 25,000,000 ? OK, that takes a while to run…. What if I tried 234 to 435,657,123 ? Mmmm… ouch. This doesn’t work too well does it ?

Improving Performance

If we want to be able to scale our code then we need to use a little math to help us get the answer, instead of relying on brute force.

In this excellent and clearly explained article about number range summation, you will see that there are many ways to visualise the process to make it easier to calculate the sum of a range - even in our head, and how they all lead toward a single formula. Within the comments you will notice that discussion identifies a more generic formula that allows for the starting position to be more than 1, and also to allow evenly spaced ’steps’.

This runs almost instantly, compared to the hung browser effect of the loop approach for this scale of numbers.

Performance Benchmarks

To see just how different the performance is, here is a complete working example testing both methods over multiple iterations of the same moderately sized range

Results:

Testing Calc: 0.001 seconds
Testing Loop: 7.342 seconds

In Conclusion

So, this simply means that you should use the right tool for the job. The Brute Force method doesn’t scale very well - there are simple equations that can answer this question, so.. use them. Here is a self contained function that returns the sum of a range from x to y with stepping z. You’ll need to validate the rangeStep to ensure it is valid with the rangeStart and rangeEnd.