So what is a CDM ?…

A Canonical Data Model is most simply explained as a standardised way of representing data that generically applies to multiple systems and data models - kind of like the master data model, but for semantics aswell as content. It can be used as a data model standard template for new systems development, or as an intermediary serialised data format - which is the aspect I’ll focus on mostly here.

Here’s an Example of using the CDM as an intermediary data format to integrate six systems with different data format requirements: Canonical Data Model

In this example, where serialised data is being passed between systems, it uses the standardised format as a middle ground, meaning that each adapter only needs to be able to translate into or out of the CDM format (rather than each individual target/source) for it to be able to understand (and therefore integrate) with other systems ‘talking’ in the same context (e.g. a specific functional domain, like shipments, invoices, orders etc). This creates semantic interoperability between the system by abstracting the differences at the perimeter of the integration solution, and re-using existing components.

So what ?

It’s not always clear when first thinking about this design why it is so important to the implementation of upstream architectures such as SOA, or why this has such impact on everyday integration.. even the more traditional hub based point-to-point integration patterns. But hopefully that realisation is starting to emerge, because it really is key to building truly flexible, seamless and invisible integration solutions.

Some of the key objectives

• Maximise Re-Use of Components
• Minimise Redundant Code
• Facilitate the concept of ‘Plug-And-Play’ interfaces
• Standardise knowledge, terminology and domain concepts (talk the same language)
• Facilitate SOA through an information architecture (IOA) foundation

Some of the Key Benefits and Advantages

Connect Once, Integrate Many - once connected to the information network via the canonical adapter, you can communicate with any other connected system as long as it is in context (e.g. order requests aren’t likely to work with Proof of Delivery)… but having said that, things do become somewhat adaptable, even out of context… for example: an order quite often contains customer information and product information, as well as being an outline of the shipment. With this in mind you *could* (and that doesn’t mean *should* :-D ) feed a customer master or CRM application with the customer information, a Product Master or Stock Management application with the product information ‘just-in-time’ for the order to be processed should the information not have been provided before (unlikely for the product master, but you get the idea). You could even feed a shipment scheduling system, shipment event tracking system and plenty of others just from that single message (the most valuable might be your data warehouse).

Here’s a comparison of point to point integration versus a canonical model:

CDM v Point to Point Comparison

As you can see, the point to point method (bottom image) creates a LOT of components and complexity. Taking this example and doing the Math:

Point-To-Point Integration for 3 sending systems and 3 receiving systems:

3 (sources) * 3 (targets) = 9 (components)

CDM Based Integration for 3 sending systems and 3 receiving systems:

3 (sources) * 1 (adapter - the CDM is now the virtual target) + 3 (targets) * 1 (adapter - the CDM is now the virtual source) = 6 (components)

Or put more simply:
PTP Integration: sources MULTIPLIED by targets
CDM Integration: sources PLUS targets

This starts to demonstrate itself very quickly when you look at 10 sources and 10 targets… would you prefer to build 20 components, or 100 ?

Standardised Information Architecture Foundation - the concept and importance of an Information Oriented Architecture (IOA) should not be understated. Architecture within an IT environment is mostly about abstraction, generalisation and standardisation of concepts to deliver sustainable, re-usable solutions and technology. One of the most popular architectures in previous years has been the N-Tier architecture where layers of the solution are packaged into abstracted levels that are ‘theoretically’ interchangeable etc. Furthermore, the more recent Service Oriented Architecture (SOA) expands / explodes that concept somewhat (there’s a bit more to it, but for examples sake..) into a distributed network oriented environment, with the addition of a bus to provide the glue that binds all the interchangeable parts together. In both of these concepts, information should be standardised, or if not possible (e.g. from multiple sources, which SOA is very prone to) then to abstract the differences between the data sources and present (e.g. via the Data Access Tier, Business Data Objects, Composite Services etc) a standardised version.

There are products that will help to do this on a READ basis - generally called data federation, it allows multiple sources of data to be combined into a single ‘view’ despite the underlying technologies (e.g. RDBMS, XML, CSV, Web Service, etc). This is great, and a useful tool, but it is not enough to satisfy the needs of a true SOA environment. There are also technologies that will help do this on a more general basis, which are much more applicable, such as business data objects, which can be used to ‘mimic’ a SOA like environment without all the standards stack. Then you have the SOA concept, which implements data and functionality as services, but really is founded on those data sources, and their role within the solution (sometimes data sources are not the authority on the data). Hence, a solid information architecture is the foundation of your SOA.

Whichever way you look at it, having this standard model helps to bring your enterprise solutions together whilst minimising diversity and maximising re-use.

For example, you could implement a Data Warehouse by simply adding a canonical adapter for the data warehouse feeds, and directing existing data (coming via other canonical adapters) to this point - if you have 20 or so systems that are already in your network you can start to see how this would make your life easier !

It also helps when designing and developing new applications & data sources, as you can simply implement the CDM template, and you have 80-90% of your data model (there are always application specific data tables required by a system).

Some Things to Consider About the Design of a CDM

• Version Control needs to be closely managed
• You should be able to change the CDM Format without ‘breaking’ existing adapters - otherwise changes will become more and more of a risk/impact
• Data content (as well as the syntax and semantics) should be standardised within the CDM boundaries (transformed on entry) - this is where Master Data Management becomes essential - for example, you want to ensure that timezones are clearly included with time/dates, customer IDs are the same, country codes, currency codes and so on are all the same
• Technology needs to support then dynamic and flexible nature of this - XML is a prime candidate, coupled with XSLT, Xpath and Xquery
• Data Models should be based on REAL LIFE - not on existing applications or the buzz-de-jour - speaking to the business is an essential part of building lasting and truly representational models
• Use Standards where possible - but don’t think there is an off the shelf model and solution that will fit your business perfectly. Things like BODs are interesting and useful, but not necessarily what you need
• Consider the data serialisation approach - message centric (like an EDI Message) or entity centric (like a database) - how normalised do you want that serialised format ? For example a Customer would appear in both orders and invoices (plus many other things) - do you want ‘Customers’ to be a separate entity, or an embedded entity ? An entity centric approach is more appropriately served by developing a Semantic Ontology, using technologies such as RDF and OWL.

So, what next ?

Delivering a CDM can be challenging - especially when delivered as part of a full information architecture overhaul. My advice ? Take little steps.

Start with a vision - think about (and ask) what the business really needs from information, then conceptualise, agree and communicate it.

Then (ideally) you want to define an information architecture strategy as thoroughly as possible upfront, so you have a good framework to work from. Try to be as far reaching (in terms of your business units etc) as possible with the analysis for this, but don’t get bogged down in specifics - just get the framework for how things should be done and the strategy around managing, governing, delivering etc.

Once you have a good framework and strategy, you can start analysing the detail and pulling together a Requirements Analysis Spec and the Architecture/Design for the solution. BUT.. this bit should be done in small manageable chunks - that way a) you minimise risk of failure b) minimise impact if things do go wrong and c) show fast, measurable results to encourage people to continue to take-up this strategy. I’ll post again soon with more detailed steps to this approach

I hope this has either inspired you to look at implementing a Canonical Data Model as part of your enterprise Information Architecture strategy, or enraged you enough to think of some better way…

From Fax to SOA, a journey to success ?

We’ve come a long way since the days of fax machines and typewriters - the objectives have remained pretty similar, and with each iteration of the solution we’ve found resolution to issues in the former, yet we continue to unveil new aspects of the objective that need addressing. Will we ever get to a perfect solution ? What is a perfect solution ? Were any of the previous solutions wrong when they were implemented ? Or were they just right for the time ?

People argue, even today, about how much better previous methods were to those of today - ok, these people generally aren’t integration experts.. but they are people that use, need or implement integration solutions - and they far outnumber the ‘experts’. The most amusing examples are those that shout and stomp about costs, and threaten to go back to faxing their orders… yes, we actually had a customer threaten to do this because they didn’t like our (very competitive - we know we’ve done a lot of benchmarking) prices. In these cases a simple bit of math will help clarify things for them.. especially when they are a global business that has about 100 million B2B transactions (messages) a year. Shall we do the math ? Ah, go on…

100m orders with 5 minutes to read/type/correct each one, would take 8 hours a day, for 250 (working) days a year, for about 21 millenia for one person. Or you could hire 20,833 administrators, who could do this for you in one year (so they don’t get a pile up in their In Tray). At, let’s say, £15k per admin, that would be about £312m per year. Now, I can assure you that our entire budget for all of our customers integration solutions doesn’t come quite to that amount (I wish it did!). Suffice to say, this is where the discussion would end. (Oh, and let’s hope they don’t intend on posting those orders.. that would be another £25m just for the stamps). Of course, when we look at the entirety of all customers, and internal interactions we start to move away from the ‘millions’ and start looking at ‘billions’, which just makes the need for a strong automated integration foundation all the more clear.

Now, that’s an easy sell isn’t it ? Manual versus automated is dead easy - cost benefits are very obvious, even to the laymen. But what about more subtle changes in the design - ones which are really more technically focused, and yield more than just ‘cost reduction’.. for example, revenue opportunity and long term cost avoidance, as well as business agility and so on. That can be difficult - not for those who understand these things, but for those that hold the purse strings and let the black and red on their current years budget dictate their views. This is where we need to help them understand why these benefits will make their longevity in success a reality.

So… here’s an introduction for those that want to get to grips with how things have been progressing and, of course, why..

Simple Requirements

OK, so here are some diagramatic examples based on the following very simple set of requirements

• 1 Message Type (e.g. Orders)
• Content Consumers (e.g. Customers)
• Each using 3 Content Providers (e.g. Visibility, Warehousing, Finance etc)

The Manual Approach…

The Direct Point-to-Point Approach…

The Aggregated Point-To-Point Approach…

The Information Oriented Approach…

The Service Oriented Approach…

The Maturity Graph…

The Future Holds Untold Adventures…

Are you up for it ? Or would you prefer to spend the next 21 millenia manually entering order data ?

I know I would like to make things ‘just work’ - automatic integration of information, services, communities and, well, pretty much everything. But getting there will be a long journey, and not one stifled through lack of technical ability, but more by social acceptance and understanding.

Is SOA our end goal ? I doubt it. But it is our foreseeable path to success, and a stepping stone to better things. There are next steps, which I will continue to discuss here, but for now, let’s just try to get the majority of people to at least the 3rd or 4th step in this pathway to integration maturity… because like it or not, MOST ’solutions’ in existence today are in the first two steps..

Just in case you wanted to use this for your own integration discussions, here is a PowerPoint Show of the Evolution of Integration.