Governance
Top 10 reasons why MDM might be the next dead buzzword
After several days of mourning the death of SOA, one of our closest friends, I see a lot of chatter shifting to MDM (Master Data Management).
Like SOA, properly architected MDM provides businesses with flexibility and agility. In addition, it can add a higher level of quality by presenting a cleansed federated view of corporate data so that there is only “one truth” to the questions that are asked by the consumers of data. Like SOA, I am a huge fan of MDM and will have some upcoming posts about how my company is leveraging it. But also like SOA, MDM can be another transformational initiative that can have disastrous consequences if we focus only on the technology side of the house. I wrote an article called the Top 10 Reasons Why People Make SOA Fail for CIO.com last year. Here is a summary:
- Fail to explain the business value
- Underestimate the impact of organizational change
- Fail to secure executive sponsorship
- Try to do it “on the cheap”
- Lack the required skills
- Poor project management
- Think of it as a project instead of an architecture
- Underestimate the complexity
- Fail to implement and/or adhere to governance
- Let vendors drive the architecture
Guess what? All of these same reasons apply to MDM. Granted the scope of MDM is typically smaller than SOA. In fact many SOA initiatives have a MDM component as part of the deliverable. But large MDM initiatives are extremely challenging especially since it targets the corporation’s crown jewel, the data! So here is my advice for those getting ready to start a MDM initiative.
- Start with the problem not the solution – Make sure MDM solves a key business problem and explain it as a business solution. Don’t use the words MDM with business people!
- Do an organizational change management assessment - Many techies scoff at this notion. This is critical to the success of large transformational projects. I highly recommend using a consultant because they can see things that those deeply rooted in the culture cannot.
- Secure strong executive sponsorship – Find a champion on the business side or leverage your C-Level IT folks if they have the integrity and respect of the leadership team. They need push this initiative down through the ranks and remove all obstacles that interfere with the success of the initiative.
- Don’t cut out steps to save money - When I say don’t do it on the cheap I am not saying you have to buy expensive tools. There are some great open source tools that may work for you. What I am saying is don’t cut out steps, don’t refrain from bringing in experts because they are expensive, don’t skip architecture and planning steps, etc.
- Partner with experts, acquire resources with the proper skills – Why reinvent the wheel? Hire an expert who has implemented MDM many times and leverage his/her expertise. Leverage some experienced consultants to work side by side with your people to get real on the job training. Hire some full time employees with MDM experience. If your budget does not allow for new hires, get rid of the dead weight or those resistant to change and replace them with people who have the necessary skills. Don’t put all of your eggs in the consulting basket!
- Put your best project manager(s) on the project – It still comes down to controlling scope, getting the right requirements, mitigating risks, and communicating effectively. This is one series of projects that you cannot afford to have fail.
- Think of MDM as an evolution, not a project – Define the future state and break up the initiative into a series of manageable deliverables. I recommend an architectural assessment and well defined strategy as an early deliverable. Why does the first deliverable always have to be production ready code? Get a plan in place first!
- Don’t underestimate the complexity – Take time up front to fully grasp what you are about to undertake. There is nothing more embarassing than telling your users that you need more time because you didn’t anticipate numerous tasks. In many cases they won’t care and will hold you to your original dates. Then you start to cut out steps (see #4).
- Implement and adhere to governance – There is nothing worse than devoting huge amounts of time building out a great architectural solution, only to see its value degrade over time due to lack of control and standards. If you are not going to govern your MDM, why bother go through the effort?
- Don’t let vendors drive the architecture – There are plenty of wonderful tools in the Magic Quadrants and in the Wave, but that doesn’t mean you need to buy them. You may already have everything you need. You may not need all of the features in commercial products and can find a reliable open source solution. Whatever you do, make sure you come to that decision by analyzing your business and technical requirements, evaluating tools based on these requirements, and performing a proof of concept before committing your precious capital. Don’t let the Power Point slides blind you! Anybody can make a nice Power Point deck. Only a few tools will actually meet your needs at your price!
Lessons Learned
Now that SOA is dead, let’s learn from this tragedy and not kill off another dear ally. Maybe we should even stop using the term MDM now and start talking about data services. Maybe we should discuss the concepts of “One view of the Truth” instead of talking to the business about data transformations and federated views. Maybe if we can get MDM right, we can bring SOA back to life. After all, I would argue that you are not really getting the benefits out of SOA without abstracting your data layer. Regardless, we need to focus on architecture. Define the future state and create a plan on how to get there one step at a time. It will be a long journey that pays dividends with every milestone that gets completed. Let’s get this one right!
Enterprise Mashups – The Icing on your SOA
The Dilemma
I have been spending a lot of time recently learning about enterprise mashups. The new startup that I am working for is building a SaaS solution that will be consumed by various types of customers and partners. These customers and partners may want to consume our data services as a RSS feed, gadget, SMS message, web page, within a portal or portlet, or a number of different ways. I do not want to spend the rest of my life developing new output mediums for our services. Instead, I would rather spend my time adding new business services to enhance our product and service offerings hence contributing to the bottom line.
Enterprise Mashups to the rescue
Enterprise mashups will allow me to offer my partners and customers the ultimate flexibility to access our products and services in ways that are convenient for them without having to wait on my IT shop to decide if (a) we think the request is important enough in our priority list, (b) if we have the time and resources to work on it, and (c) how much we will charge them. On the IT side of the house, with an enterprise mashup strategy in place we can be assured that whatever mashups our customers and partners create, they will be subject to the same security and governance as the services we have developed. The diagram below shows a logical view of how our SOA will be designed.
As you can see, we have clearly abstracted the various layers within the architecture and they all inherit our overall security policies. SOA governance is applied to this architectural approach to enforce our standards and design principles. Overall IT governance provides oversight over the entire enterprise which includes legacy systems (we don’t have any legacy yet), third party software, etc.
Now let’s add the enterprise mashup layer. We want to hide the complexity of our architecture from the end user and expose data services to them to consume. At the same time we want these mashups to be equally secure as the services we write and adhere to the same governing principles. Enterprise mashup products provide tools to make managing this layer easy and efficient. The diagram below shows the enterprise mashup layer inserted into the architecture as a layer on top of SOA.
 |
| From SOA Slides |
Enterprise Mashups in simple terms
If the concept of Enterprise Mashups is still not clear, check out this white board discussion with JackBe’s CTO John Crupi.
I spent some time talking to JackBe’s VP of Engineering Deepak Alur a few week’s back. Deepak discussed how enterprises have been focusing more on infrastructure and technology and not on the consumers of data. As he coined it, many shops are “developing horizontally and not addressing the needs of the users”. He talked about how users were doing their own brute force mashups by cutting and pasting data from various places into Excel. This creates various issues within the enterprise due to lack of data integrity, security, and governance. It is ironic how corporations spend huge amounts of money on accounting software and ERP systems, yet they still run the business out of user created Excel spreadsheets! The concept of enterprise mashups addresses this by shifting the focus back to the user consumption of data. Here are some of the requirements for mashups that Deepak pointed out to me:
- User driven & user focused
- Both visual & non-visual
- Client & server side (although most are server)
- Plug-n-Play
- Dynamic, Adhoc, Situational
- Secure & Governed
- Sharable & Customizable
- Near zero cost to the consumer
Jackbe’s enterprise mashup tool is called Presto. Presto is an Enterprise Mashup Platform that allows consumers to create “mashlets” or virtual services. IT’s role is to provide the security and governance for each data service that will be exposed for consumer use.
Presto Wires is a user friendly tool to allow users to create their own mashups by joining, filtering, and merging various data services (as shown in the picture below).
In this example the user is combing multiple data points from many different organizations in an automated fashion. They could then present this data to multiple different user interfaces and devices. All without waiting on IT.
Here is a simple example of a mashup of Google Maps with a Carpool data service.
The folks who create the Carpool data service where able to leverage the Google Maps service without having to write their own mapping software and without having to know the underlying architecture behind Google Maps.
How this solves my Dilemma
Back to my dilemma. By leveraging a tool like Jackbe’s Presto or WSO2’s Mashup Server, I can now present various data services in a secured and governed fashion to my customers and partners without being concerned on how they want to consume it. Whether they want the mashup on their own intranet, as a desktop gadget, as an application on Facebook, or what ever they dream of, all I need to be concerned with is the SLA of my data services. This also makes my product offering more competitive than my competitors who have proprietary user interfaces that do not provide the flexibility and customization that the customers desire. As I said in the title, this is the Icing on your SOA cake. For those organizations who are disciplined enough to implement SOA and follow the best practices of design and governance, the reward can be an simple addition of an Enterprise Mashup Platform on top of your SOA stack. This is the ultimate flexibility and agility that SOA promises.
Book Review: SOA Governance by Todd Biske
My friend and colleague Todd Biske has recently published a must read book about SOA Governance. Most books that discuss process can require a ton of caffeine in order to make it from cover to cover. Not this book! Todd uses a unique style of combining a story of a fictional company along with his well served advice. He walks us through a multi year SOA project with a fictional insurance company called Advasco. Advasco, like many of the companies that we work for, has years of silo legacy applications that are the direct result of acquisitions, mergers, and years of developing in silos. The goal of their initiative was to “provide sales agents and marketing staff for the insurance division with a single view of the customer”.
Todd takes us through the evolution which spans five years. Over this time, you can see the company make continual strides towards the overall vision but not without challenges. With each challenge comes another opportunity to mature their governance model. At the end of the book, Advasco’s IT staff have made a major impact to the company’s bottom line and ability to react to market changes. This is one of the few books that actually shows us what success looks like. This is why the book is so good. Many books talk about processes, structure, and controls but fail provide us with a glimpse into what the fruits of that labor looks like. Todd takes us from project inception, to successes, to set backs, to mitigation strategies, and ultimately to enterprise wide success. By taking us through this journey we can get a better understanding of the impact of the recommendations that he makes for SOA Governance. I won’t go into details on what he recommends. I will let you read it yourself. But I can guarantee that you will enjoy the book and will be able to relate your real world experiences with the experiences of the fictional characters who helped move Advasco forward by leveraging SOA.
What made Advasco successful?
If you follow everything that Todd recommends in this book, you still are not guaranteed success. There were some specific events and characteristics that made this journey a success. Here is a short list:
- Initiative was driven top down
- Very little resistance to change
- Great working relationship with business and IT
- Very talented staff that was willing to learn
- Effective EA team
- Project was business focused
Final Thoughts
SOA Governance is a must read for any company preparing for SOA or for companies struggling to make their SOA initiative successful. Follow the advice and examples that Todd provides but also address the bullets I listed above. If you have the great staff that Advasco has you will find success. But in the real world, the effort to promote this type of change in the enterprise will usually require much more focus on change. At the SOA Consortium meeting last month in Orlando, Todd provided us with a great presentation about SOA Governance. I followed that up with this presentation on change.
If you focus on Organizational Change Management and heed Todd’s advice on SOA governance, your odds of succeeding with SOA will be greatly enhanced. Buy the book, it is the best book on governance that I have read so far.
SOA Skillsets: Part 4 – The Testers
I have discussed the SOA Evangelist, the Architects, and the Developers. Today I will discuss the role of the Testers and the characteristics required to contribute to a successful SOA implementation.
One of the most important roles and one that I probably should have included in the Architects post is the Testing Architect. As I have written on CIO.com in my article called Six Questions to Consider Before Building a SOA Testing Team, SOA testing requires a much deeper knowledge of technical skills, including development skills. It might be unrealistic to expect your entire testing team to possess the desired technical skills required to successfully test SOA, but your test architect and your leads should be able to understand concepts like statefullness, distributed computing, and data services, to be able to properly validate the underlying architecture. They also need to be able to take developer test harnesses and update them with their own test scripts.
A successful SOA testing strategy starts with the test architect. This person must have a very in depth knowledge of SOA and should work closely with the EA team. I actually recommend that this person is a member of the EA team, but every business and culture is different. The goal of the test architect should be to set up a framework and a core group of policies and procedures (part of IT Governance) so that the rest of the test team has the tools and the guidance to successfully test SOA. Without an established testing architecture, the company will have to rely heavily on expert knowledge from the entire testing team. I have seen three scenarios through my personal experiences and through my research.
Scenario 1: Underestimating SOA
The first scenario is out right failure caused by not having the tools and knowledge required. This is caused by a company not realizing that their current methodology and internal personnel are not well equipped for testing SOA. These companies do not invest enough in tools, training, and governance and usually can only test the presentation layer and possibly the interfaces. By not understanding the concepts of SOA, they are unable to validate the architecture which leads to poor performing and fragile services.
Scenario 2: Paying through the nose
The second scenario I have seen is relying too heavily on “expert” consulting firms for testing. In this scenario the company bets the farm on an experienced SOA consulting firm and pay rates that far exceed $100/hr. This model cannot be sustained for any length of time unless the company is willing to burn huge amounts of capital (which is not a popular thing to do these days).
Scenario 3: Good balance of internal/external expertise
A more desirable scenario is to train or hire a SOA testing architect, build a solid testing platform tailored for the needs of the organization, and govern the testing process while training the other members of the team. Companies should hire one or more experienced SOA tester, find an experienced consultant or two, or train an experienced and credible internal candidate to take the lead for creating the testing architecture. At the same time the testing experts are charged with transferring knowledge over to the rest of the internal team members. This is critical because a highly experienced SOA tester is in great demand and is a flight risk since they are a rare breed. It is critical to grow the knowledge base internally.
Testing needs
The needs can be broken down into these categories: People, Tools, Governance. So what are the characteristics of a successful SOA tester. The answer is dependent on the architecture that is implemented, which is related to the tools and policies that are put in place. Below is a diagram I often use to discuss the typical layers of an architecture.
 |
| From SOA Slides |
As I mentioned in the previous discussion about the developers, I see the need and desire to specialize within the layers. The same holds true for testers. Work within the layers happens simultaneously in development. I recommend an iterative development and testing approach which means there should be testers working within the layers simultaneously as well.
Here is what I would strive to put in place, keeping in mind that these are roles and some people may fill multiple roles:
SOA Test Architect
Courageous leader with extensive working knowledge of SOA, distributed computing, integration testing experience, coding/scripting experience, and understands the business. It is likely that you may have to go outside to hire this person or bring in a consultant to assist your top level tester.
SOA testing leads
This person or persons must understand all layers of the architecture (let’s not forget security either) and should be able to design test plans that validate both the architecture and the governance model. They should understand how to perform black, white, and gray box tests. Testing abstracted services requires extensive testing in the areas of security, performance, and regression. Throw in the versioning capabilities of services and the fact that service consumers can use services in ways that were not anticipated and the permutations of test cases start skyrocketing. The test leads need to practice risk based testing and balance risks, timelines, and costs. There is just so much more at stake here than in the traditional n-tier architectures.
Business process testers
The business processes should be modeled within some tool and will likely call one or many services. The process flow requires a series of decision points based on variables and constants and can trigger events such as notifications, alerts, other processes, error handling routines, services, and a variety of other possibilities. The testers need to validate the business process as a stand alone entity. For example, if the business process is “validate credit card”, the tester needs to ensure that this process handles the inputs correctly, properly runs the validation process, and generates the appropriate output. At this point, the tester need not be concerned with any other processes or services. These testers must work closely with the business and/or business analysts and do not need the breadth of technical knowledge that the leads and architects must have (although it would help). They should be approaching the testing from a business standpoint.
Integration testers
These testers must be much more technical and understand how to work with XML, SOAP, WSDLs, networking & telecommunication concepts, statefullness, and various platforms and technologies (Java vs .Net, Windows vs Unix/Linux vs. mainframe, etc.).
User Interface testers
The company most likely already has an abundance of people who can test the UI. If your company is leveraging mashups, wireless devices, or consumer facing UIs, the complexity of the testing will be greatly increased. These testers may be need to become familiar with AJAX, RIA, portal technology, RSS, and a variety of social software.
Data services testers
These testers must understand concepts of data modeling, database CRUD operations, transformations, security and roles, authentication, and much more. Everything starts with the data and if errors are introduced in this layer, everything else is doomed to fail. You must have a very strong testing lead working in the data services layer since data is the foundation of any successful implementation.
Other areas of focus
The name of the game is speed to market and test automation is a critical component for making that a reality. Performance testing is extremely critical and organizations should practice simulations to try to anticipate future performance of all services. Validating the security model and the governance model should also be part of the overall test strategy. Whoever is responsible for designing the security test plan should read (and fully understand) this book.
Closing comments
I am by no means a testing expert (but I did stay at a Holiday Inn Express last night). I do read a number of testing blogs listed below:
- Randy Rice’s Software Testing & Quality Blog
- Frank Cohen’s blog
- The ITKO Lisa Soapbox: SOA testing, validation & virtualization
- SOA Testing
- Mike Kelly’s testing blog
Make sure your organization funds the necessary training and tools for testing. From my own personal lessons learned, make sure you invest as much effort in testing and security up front as you do in architecture and development. On my project a few years ago, I wore my developer hat too much in the research phase and did not properly budget for what was necessary for testing. Do not make this same mistake. As hard as it is to sell the business on SOA, it is much harder to go back for more money, especially before any business value is realized!
Are you Ready for SOA?
I frequently discuss SOA with potential clients, fellow architects, and business partners. Each audience requires a discussion that is tailored to the appropriate level of technical and business expertise. Today I had to present to a CEO and a few of his top staffers about what it takes to launch into a SOA initiative. So I put together a presentation similar to the one below and felt like sharing it with all of you.
Before you look at it let me put the discussion into context. The target audience is hi level management, business and IT, who are already bought into SOA and are trying to understand what is involved before they launch into a full blown SOA initiative. This is not a presentation for architects, rather it is to inform C-Level stakeholders what is required from a business, technology, and culture standpoint.
I also had to sell to them that I know what I am talking about, hence the first few slides about my background. One last note: Some of the slides are mostly visual and only get the point across when backed up with talking points.
I hope this adds value to some of you. Enjoy!
Preventing SOA Failures
Here is a slide show for those who need to explain to upper management the risks involved trying to implement SOA and the importance of SOA governance. This must be sold early on and the necessary resources must be brought in. Trying to implement governance after implementation is risky, expensive, and hard to do. Like SOA, governance should be implemented one step at a time and will take years to evolve to a high level of maturity. Make sure your SOA governance maturity level matches your SOA maturity level. Don’t try to implement all of your governance up front because you will never have any time left to implement any SOA projects, hence you will not deliver value to the business. I hope the slides help. Enjoy!
Free SOA Governance Pocket Guide
One of the many websites I visit, SOAGovSource, published a free pocket guide today for all of their registered users. This guide has quotes from numerous articles about SOA Governance taken from a variety of popular blog posts including yours truly. You can see the tables of content on the right hand side of this post.
To download this guide, go here. This PDF file gives you great quotes from various authors and links to articles written by industry experts, analysts, and practitioners.
Enjoy!
SOA Governance case studies
I attended Zapthink’s Practical SOA event on Tuesday in New Jersey. There were many great presentations from both vendors and practitioners. There were three different case studies presented including mine which was on selling SOA to the Business (the video will be posted soon). What was interesting to me was the three different paths the companies in these case studies took. The first case study was from The Hartford. They have a well established governance model and are very far along into the maturity model. If you read a text book on what SOA governance should look like, it will look like the Hartford. They have their processes so buttoned up that they even tie developers’ annual review process to how they adhere to Hartford’s best practices in reusability. A key take away is the fact that this took several years and a firm commitment from their leadership team to put this in place.
The second case study was from Lehman Brothers which was implemented in a totally different fashion. In this scenario, the governing body had very little control and power over the distributed development groups. So instead of just giving in and giving up, they took a different approach. They leveraged tools and some custom code to discover web services and usage of services in the repository. They would receive a page anytime a new service was discovered. Then they would study the service to see if it was constructed the proper way. If there were opportunities for improvement, the governance team would engage in conversations with the service owners to coach and mentor them on the proper design and deployment of their services.
In my scenario, we have a totally different story. Because we were so successful in selling the business on BPM and SOA, they funded the initiative and targeted very aggressive delivery dates for key functionality. That did not leave us enough time to plan and implement a governance model. So we are establishing a road map for delivering governance one step at a time. We understand the risks of not having enough governance up front and expect some headaches along the way, but the business was not going to wait a year for us to put those processes in place.
So theory and text books say that you must start with governance. The reality is you evolve governance over time. I highly recommend attending these Practical SOA conferences if you get a chance. I jokingly refer to it as group therapy because we are all sharing our lessons learned.
If you missed the conference, there are two more scheduled. The next one is in London on April 25 and the last one is in Las Vegas on May 16. I also highly recommend the LZA SOA Bootcamp which I recapped a while back. Ron at Zapthink told me that they are targeting May 5-8 in New Jersey for the next boot camp if they get enough people interested in it. Send Ron an email at info@zapthink.com if you want to attend.
Dumb SOA Guys
I just read an entertaining post from Dave Linthicum called Do you Have a DSG (Dumb SOA Guy) Issue? A DSG refers to a person leading the SOA effort who has the political clout but has no clue what SOA is. If you have one of these, you could be in for a long painful and expensive journey. Here is a key point from the article:
Again, the technology around SOA is simple, and I never worry about how we’re going to leverage technology to solve a problem. However, the people issues are more concerning and more difficult to fix.
My head was nodding when I read this. I can tell you from experience over the last 18 months that the hard part is culture change, change management, and establishing governance. If you have a few smart architects and a good implementation partner like I have, the technology part can be easily conquered. The hardest part I have experienced from the technology standpoint is getting all of the tools in the stack completely integrated and stable. But even these issues are a one time hit that you have to deal with on your first implementation.
Back to the DSGs. The person leading the SOA initiative needs to be strong in the following areas:
- Technology – must understand SOA at a conceptual level or better
- Political Clout – must have a high level of influence to remove roadblocks
- Business – must align technically with business drivers
- People Skills – must know how to motivate and direct people
If your DSG is lacking in any one of those four areas, your SOA initiative’s chance of success will decrease dramatically. If your DSG is lacking in more then one area, cancel the project now or get a new DSG!
SOA Governance – Balancing process and agility
We are in the process of defining and implementing our SOA governance processes. We are a medium size company so having armies of process cops is not an option, not to mention totally undesirable. At the same time, we need some level of process to ensure that we architect business services in a consistent manner and that we design for the enterprise as opposed to the old application silo approach. To make governance even more challenging, we want to deploy in cycles of 12 weeks or less. So how can we be agile and enforce SOA governance at the same time?
One way is to shift from text heavy documentation to visual documentation. In other words, stop generating hundred page Word documents and start creating UML models, business process models using BPMN, use case diagrams, and architecture diagrams. These artifacts are like blue prints for a building architect. If you were building your dream house would you type up the specs for your house in a Word document and hand it to your builder or would you give him blue prints?
Another way to stay agile and still enforce governance is to tightly control scope and requirements. Users are used to asking for the sun and the moon because they typically have to wait several months to get the next release. It is similar to the way politicians load up a bill with pet projects because they know it might be their only chance to get their stuff done. We must train the user community that in an agile approach, we will deploy more frequently and in shorter time spans. This means that you don’t need to get every last requirement jammed into the project because another release is coming on the heals of the current release.
But the most critical way to stay agile while having a solid governance model in place is to use processes where it makes sense. Focus on artifacts that mean something to those who have to develop software. Don’t focus on documents that are never used once they get signed off. One of my favorites is the team structure document. I have seen teams waste days creating a document that nobody other then the PM uses that describes all the roles, who the team members are, what their contact info is, how meetings will be held, when they will be held, and so on. Who cares? The PM needs to know this but why waste the rest of the teams time with reviewing this stuff. I can look up people’s information on the portal faster then I can even find this document. Over the years I have seen teams create documents for the sake of satisfying a check list. This is a total waste of time. SOA Governance is all about managing services and the service life cycle. SOA governance can plug into your existing methodology (if you so desire). Focus on the processes that add value and discard everything else.
For small and medium sized businesses, we must be careful not to get bogged down in process. Unlike large enterprises, we don’t have the luxury of dedicating several full time employees to enforcing processes and procedures. Instead, we must put a solid framework in place and rely on our people to enforce the necessary processes. Setting up a SOA Center of Excellence and an IT Steering committee under the watchful eye of a strong executive sponsor (CIO, Chief Architect, etc.) is one way to approach it.
There is no silver bullet here. Each company must decide for themselves how much process to put in place. If they put too much process in place, they can get bogged down and never meet expectations. If they don’t implement enough process, then they will eventually spiral into mass chaos and won’t realize the benefits of SOA like reuse, reduced maintenance costs, and increased flexibility.
