testing
How the cloud has changed testing forever
While many companies still debate whether it makes sense for them to look to the cloud for their application and data center needs, there should be no debate that the cloud is the place to go for testing. In this post I discuss three examples of how my company (M-Dot Network) has leveraged the cloud for testing in ways that are not even feasible in the on-premise world.
Prototyping/Discovery
Problem: We were building a high speed transaction processing system and did not know how much memory and CPU would be optimal for the architecture.
Pre Cloud – In most of the previous projects I worked on in other companies, we typically had to purchase our hardware early in the life-cycle due to long procurement cycles and setup time. Usually we had to order this way before we started prototyping which means we had to take our best guess. The consequences would normally be that we would have to tailor our solution around the hardware that we purchased or existing hardware that we were forced to use. The end result was usually a less than optimal performing solution due to hardware constraints. Instant legacy, how nice!
Cloud - The procurement and setup issues go away because resources can be provisioned in minutes in the cloud. This allowed us to get deep into the prototyping phase and then test our design against numerous combinations of CPU, memory, and database subsystems. The result was the complete opposite of the old days. Instead of rigging our design to meet the constraints of hardware, we were able to provision the virtual servers that met our performance requirements based on actual test results. That is nirvana!
Performance Testing
Problem: It is very time consuming and expensive to create performance tests that can simulate millions of transactions from multiple touchpoints.
Pre Cloud – Unless you were testing on a soon to be production system, you had no way to do true performance testing without having a replica of your production environment, plus enough hardware/infrastructure to simulate loads that far exceed your current projections. After all, it makes no sense to only test the amount of traffic that you expect to see in production without testing spikes and testing for future growth. I would also argue that one should test the system to see at what load the system starts to break or perform sub-optimally. In the on-premise world, buying additional capacity just to do performance tests coupled with the incredibly high costs of software licensing from tools like Loadrunner is just not feasible. So companies rarely are able to simulate with large enough loads to test scale and simply can’t afford to provision test systems that are equivalent to their production environments.
Cloud - We leveraged Testing as a Service. We spent 2 weeks with our technology partner SOASTA and were able to script and deploy a 1 million concurrent user performance test. SOASTA fired up 587 servers and we fired up over 50, that is 600+ servers “procured” and installed in an hour. We ran our simulation through out the day testing bursts and sustaining a load of 1 million concurrent transactions and were able to prove that our network could handle transactions for over 200,000 stores concurrently! After we finished the test we tore down all 600+ servers and only paid for the time we used them. Imagine trying that on-premise! A 2 week setup and test of 1 million concurrent transactions per minute leveraging 600+ servers at a very low cost without adding headcount, infrastructure, software licenses, etc. Without a solution like SOASTA’s Cloudtest, we would not even had attempted a test of this magnitude. We actually discovered and fixed a few bottlenecks along the way. Without this test, those bottlenecks would have been discovered in production and would have impacted our customers.
Resources and Mobile Devices
Problem:
We recently completed our social media manager tool to allow our retailers to manage their SMS groups, Twitter, & Facebook updates from one place (including issuing coupons). To test this we needed to have access to numerous types of mobile devices, carriers, geographical locations, and an army of temporary testing resources.
Pre-Cloud:
In the old days, I would have to purchase, lease, or borrow a large collection of mobile devices across a wide variety of carriers. I would also need to put people in various areas across the country to truly test the experience as the retailers’ customers would experience it. After all, I would rather encounter coverage/carrier issues before consumers do. I could also outsource this to a technology provider for some heavy costs. Either way, this is an expensive undertaking using traditional methods.
Cloud:
Instead, we partnered with cloud based crowd sourcing testing partner, UTest, who has testers scattered throughout the world with every handset/carrier combination imaginable. They also have a web based defect tracking and collaboration platform. So with a simple phone call and a very low cost, we set up and executed a three day test that covered most of the major phones, from smart phones to basic mobile devices, and across numerous carriers. In addition, we loaded up on testers who lived in the areas close to the retailers who would be using this service. This gave us mass coverage of devices and carriers for an extremely low cost and over a very short window of time. Agile, cost effective, and efficient!
Summary
These are just three examples of how the cloud is changing testing forever. All three of these scenarios can be tested in the cloud regardless if the actual solution to be tested was built in the cloud or on-premise. Even if you are building your solution on-premise, you can still do the initial prototyping on the cloud to figure our how much CPU and memory you need to buy. Many of both SOASTA’s and UTest’s customers have on-premise solutions. So even if your company fears the cloud as a solution for deploying applications and building data centers, there is no excuse for continuing to spend way too much time and money testing things the “way we have always done it”.
SOA Testing: A discussion with iTKO’s John Michelsen
A few days ago I had a chance to catch up with iTKO’s co-founder and “Chief Scientist” John Michelsen to discuss SOA Testing. For those of you unfamiliar with iTKO, they produce one of the industry leading SOA testing products known as Lisa.
I have been writing about SOA testing a lot lately and the folks at iTKO contacted me after I wrote Six Questions to Consider before Building a SOA Testing Team on CIO.com. Since this article was fresh on my mind, I asked John some of the same questions that were posed in the article.
My first question to John was to identify some of the characteristics of successful SOA implementations he has seen from his customer base. Without hesitation, John pointed to customers that integrated testing into the development cycle. Successful SOA implementations tend to shy away from old waterfall methodologies and involve QA early in the life cycle. In many cases, testers work side by side with developers. John mentioned that “development builds things from scratch, QA does not“. He pointed to this three step process which is the trademark of successful SOA testing:
- Development builds the testing framework
- Development builds initial test cases
- QA enhances the test cases
Steps 2 & 3 is an iterative process and requires good collaboration between development and QA. This advice is similar to what I recommend in my article SOA Skillsets: PArt 4 – The Testers. In this article I recommend that a Test Architect be part of the EA team and design the testing framework and strategy. This person needs to be extremely technical and have a deep understanding of SOA concepts. The test leads also need to have a thorough understanding of SOA while the remainder of the testing team does not need to be as strong technically. However, they do need to be able to write some code as the developers hand over their test cases for refinement.
I asked John how technical the testers need to be. He stated that “most QA folks are still business analyst types and it should stay that way. If QA gets too technical then they lose the focus on testing“. I agree which is why I advocate having the Test Architect as a member of the EA team focusing on building out the framework, evangelizing, and training the QA team.
I asked John about the importance of testing tools. He stated that the “Testing tools need to let less technical people perform technical tasks“. This is a good point because your testers need to more focused on validating that the business and technical requirements are met than figuring out how to master a new tool.
This last point led us to a discussion about iTKO’s Lisa products. John mentioned that they have just released version 4 which has many new features. In addition to the core product, Lisa Enterprise, the Lisa Server product has two new important modules, Continuous Validation Service (Lisa CVS) and Virtual Service Environment (Lisa VSE). Lisa CVS addresses the challenges of integration testing, deployment issues, and validation of governance compliance. Anybody with experience in SOA will tell you that the distributed and loosely coupled nature of SOA adds a whole level of complexity in the areas of builds and deployments.
Lisa CVS helps manage and automate some of those complexities and takes a more proactive approach to management and governance by allowing you to “revalidate expected behavior“. Lisa VSE takes advantage of virtualization technologies so that you can create images that can simulate behavior of your heterogeneous environment. They accomplish this by installing agents on your various environments (mainframes, database servers, web servers, etc.) which allows VSE to monitor and emulate performance by creating a server side model of your production environment. This model can then be deployed in a virtual testing environment and be tweaked to simulate various scenarios such as abnormal spikes in traffic, simulating new customer loads, or whatever scenario you wish to entertain.
John was real excited when he talked about version 4.6 that is due out towards the end of the year. In this release, Lisa will add functionality to allow customers to test stateful services. This has been the missing link in SOA testing up to this point. John also mentioned that Lisa is now integrated with all major registry and repository tools. In addition, Lisa is also being used for Web 2.0 testing due to the work they have done with web services and WS-* standards.
In summary, testing is a critical component in the quest to launch and implement a successful SOA. By creating a well thought out testing strategy that includes tools to automate and simplify testing, organizations can focus more on the business side of SOA and less time focusing on the technology side of the equation.
SOA Skillsets: Part 4 – The Testers
I have discussed the SOA Evangelist, the Architects, and the Developers. Today I will discuss the role of the Testers and the characteristics required to contribute to a successful SOA implementation.
One of the most important roles and one that I probably should have included in the Architects post is the Testing Architect. As I have written on CIO.com in my article called Six Questions to Consider Before Building a SOA Testing Team, SOA testing requires a much deeper knowledge of technical skills, including development skills. It might be unrealistic to expect your entire testing team to possess the desired technical skills required to successfully test SOA, but your test architect and your leads should be able to understand concepts like statefullness, distributed computing, and data services, to be able to properly validate the underlying architecture. They also need to be able to take developer test harnesses and update them with their own test scripts.
A successful SOA testing strategy starts with the test architect. This person must have a very in depth knowledge of SOA and should work closely with the EA team. I actually recommend that this person is a member of the EA team, but every business and culture is different. The goal of the test architect should be to set up a framework and a core group of policies and procedures (part of IT Governance) so that the rest of the test team has the tools and the guidance to successfully test SOA. Without an established testing architecture, the company will have to rely heavily on expert knowledge from the entire testing team. I have seen three scenarios through my personal experiences and through my research.
Scenario 1: Underestimating SOA
The first scenario is out right failure caused by not having the tools and knowledge required. This is caused by a company not realizing that their current methodology and internal personnel are not well equipped for testing SOA. These companies do not invest enough in tools, training, and governance and usually can only test the presentation layer and possibly the interfaces. By not understanding the concepts of SOA, they are unable to validate the architecture which leads to poor performing and fragile services.
Scenario 2: Paying through the nose
The second scenario I have seen is relying too heavily on “expert” consulting firms for testing. In this scenario the company bets the farm on an experienced SOA consulting firm and pay rates that far exceed $100/hr. This model cannot be sustained for any length of time unless the company is willing to burn huge amounts of capital (which is not a popular thing to do these days).
Scenario 3: Good balance of internal/external expertise
A more desirable scenario is to train or hire a SOA testing architect, build a solid testing platform tailored for the needs of the organization, and govern the testing process while training the other members of the team. Companies should hire one or more experienced SOA tester, find an experienced consultant or two, or train an experienced and credible internal candidate to take the lead for creating the testing architecture. At the same time the testing experts are charged with transferring knowledge over to the rest of the internal team members. This is critical because a highly experienced SOA tester is in great demand and is a flight risk since they are a rare breed. It is critical to grow the knowledge base internally.
Testing needs
The needs can be broken down into these categories: People, Tools, Governance. So what are the characteristics of a successful SOA tester. The answer is dependent on the architecture that is implemented, which is related to the tools and policies that are put in place. Below is a diagram I often use to discuss the typical layers of an architecture.
 |
| From SOA Slides |
As I mentioned in the previous discussion about the developers, I see the need and desire to specialize within the layers. The same holds true for testers. Work within the layers happens simultaneously in development. I recommend an iterative development and testing approach which means there should be testers working within the layers simultaneously as well.
Here is what I would strive to put in place, keeping in mind that these are roles and some people may fill multiple roles:
SOA Test Architect
Courageous leader with extensive working knowledge of SOA, distributed computing, integration testing experience, coding/scripting experience, and understands the business. It is likely that you may have to go outside to hire this person or bring in a consultant to assist your top level tester.
SOA testing leads
This person or persons must understand all layers of the architecture (let’s not forget security either) and should be able to design test plans that validate both the architecture and the governance model. They should understand how to perform black, white, and gray box tests. Testing abstracted services requires extensive testing in the areas of security, performance, and regression. Throw in the versioning capabilities of services and the fact that service consumers can use services in ways that were not anticipated and the permutations of test cases start skyrocketing. The test leads need to practice risk based testing and balance risks, timelines, and costs. There is just so much more at stake here than in the traditional n-tier architectures.
Business process testers
The business processes should be modeled within some tool and will likely call one or many services. The process flow requires a series of decision points based on variables and constants and can trigger events such as notifications, alerts, other processes, error handling routines, services, and a variety of other possibilities. The testers need to validate the business process as a stand alone entity. For example, if the business process is “validate credit card”, the tester needs to ensure that this process handles the inputs correctly, properly runs the validation process, and generates the appropriate output. At this point, the tester need not be concerned with any other processes or services. These testers must work closely with the business and/or business analysts and do not need the breadth of technical knowledge that the leads and architects must have (although it would help). They should be approaching the testing from a business standpoint.
Integration testers
These testers must be much more technical and understand how to work with XML, SOAP, WSDLs, networking & telecommunication concepts, statefullness, and various platforms and technologies (Java vs .Net, Windows vs Unix/Linux vs. mainframe, etc.).
User Interface testers
The company most likely already has an abundance of people who can test the UI. If your company is leveraging mashups, wireless devices, or consumer facing UIs, the complexity of the testing will be greatly increased. These testers may be need to become familiar with AJAX, RIA, portal technology, RSS, and a variety of social software.
Data services testers
These testers must understand concepts of data modeling, database CRUD operations, transformations, security and roles, authentication, and much more. Everything starts with the data and if errors are introduced in this layer, everything else is doomed to fail. You must have a very strong testing lead working in the data services layer since data is the foundation of any successful implementation.
Other areas of focus
The name of the game is speed to market and test automation is a critical component for making that a reality. Performance testing is extremely critical and organizations should practice simulations to try to anticipate future performance of all services. Validating the security model and the governance model should also be part of the overall test strategy. Whoever is responsible for designing the security test plan should read (and fully understand) this book.
Closing comments
I am by no means a testing expert (but I did stay at a Holiday Inn Express last night). I do read a number of testing blogs listed below:
- Randy Rice’s Software Testing & Quality Blog
- Frank Cohen’s blog
- The ITKO Lisa Soapbox: SOA testing, validation & virtualization
- SOA Testing
- Mike Kelly’s testing blog
Make sure your organization funds the necessary training and tools for testing. From my own personal lessons learned, make sure you invest as much effort in testing and security up front as you do in architecture and development. On my project a few years ago, I wore my developer hat too much in the research phase and did not properly budget for what was necessary for testing. Do not make this same mistake. As hard as it is to sell the business on SOA, it is much harder to go back for more money, especially before any business value is realized!
