How the cloud has changed testing forever
popping..While many companies still debate whether it makes sense for them to look to the cloud for their application and data center needs, there should be no debate that the cloud is the place to go for testing. In this post I discuss three examples of how my company (M-Dot Network) has leveraged the cloud for testing in ways that are not even feasible in the on-premise world.
Prototyping/Discovery
Problem: We were building a high speed transaction processing system and did not know how much memory and CPU would be optimal for the architecture.
Pre Cloud – In most of the previous projects I worked on in other companies, we typically had to purchase our hardware early in the life-cycle due to long procurement cycles and setup time. Usually we had to order this way before we started prototyping which means we had to take our best guess. The consequences would normally be that we would have to tailor our solution around the hardware that we purchased or existing hardware that we were forced to use. The end result was usually a less than optimal performing solution due to hardware constraints. Instant legacy, how nice!
Cloud - The procurement and setup issues go away because resources can be provisioned in minutes in the cloud. This allowed us to get deep into the prototyping phase and then test our design against numerous combinations of CPU, memory, and database subsystems. The result was the complete opposite of the old days. Instead of rigging our design to meet the constraints of hardware, we were able to provision the virtual servers that met our performance requirements based on actual test results. That is nirvana!
Performance Testing
Problem: It is very time consuming and expensive to create performance tests that can simulate millions of transactions from multiple touchpoints.
Pre Cloud – Unless you were testing on a soon to be production system, you had no way to do true performance testing without having a replica of your production environment, plus enough hardware/infrastructure to simulate loads that far exceed your current projections. After all, it makes no sense to only test the amount of traffic that you expect to see in production without testing spikes and testing for future growth. I would also argue that one should test the system to see at what load the system starts to break or perform sub-optimally. In the on-premise world, buying additional capacity just to do performance tests coupled with the incredibly high costs of software licensing from tools like Loadrunner is just not feasible. So companies rarely are able to simulate with large enough loads to test scale and simply can’t afford to provision test systems that are equivalent to their production environments.
Cloud - We leveraged Testing as a Service. We spent 2 weeks with our technology partner SOASTA and were able to script and deploy a 1 million concurrent user performance test. SOASTA fired up 587 servers and we fired up over 50, that is 600+ servers “procured” and installed in an hour. We ran our simulation through out the day testing bursts and sustaining a load of 1 million concurrent transactions and were able to prove that our network could handle transactions for over 200,000 stores concurrently! After we finished the test we tore down all 600+ servers and only paid for the time we used them. Imagine trying that on-premise! A 2 week setup and test of 1 million concurrent transactions per minute leveraging 600+ servers at a very low cost without adding headcount, infrastructure, software licenses, etc. Without a solution like SOASTA’s Cloudtest, we would not even had attempted a test of this magnitude. We actually discovered and fixed a few bottlenecks along the way. Without this test, those bottlenecks would have been discovered in production and would have impacted our customers.
Resources and Mobile Devices
Problem:
We recently completed our social media manager tool to allow our retailers to manage their SMS groups, Twitter, & Facebook updates from one place (including issuing coupons). To test this we needed to have access to numerous types of mobile devices, carriers, geographical locations, and an army of temporary testing resources.
Pre-Cloud:
In the old days, I would have to purchase, lease, or borrow a large collection of mobile devices across a wide variety of carriers. I would also need to put people in various areas across the country to truly test the experience as the retailers’ customers would experience it. After all, I would rather encounter coverage/carrier issues before consumers do. I could also outsource this to a technology provider for some heavy costs. Either way, this is an expensive undertaking using traditional methods.
Cloud:
Instead, we partnered with cloud based crowd sourcing testing partner, UTest, who has testers scattered throughout the world with every handset/carrier combination imaginable. They also have a web based defect tracking and collaboration platform. So with a simple phone call and a very low cost, we set up and executed a three day test that covered most of the major phones, from smart phones to basic mobile devices, and across numerous carriers. In addition, we loaded up on testers who lived in the areas close to the retailers who would be using this service. This gave us mass coverage of devices and carriers for an extremely low cost and over a very short window of time. Agile, cost effective, and efficient!
Summary
These are just three examples of how the cloud is changing testing forever. All three of these scenarios can be tested in the cloud regardless if the actual solution to be tested was built in the cloud or on-premise. Even if you are building your solution on-premise, you can still do the initial prototyping on the cloud to figure our how much CPU and memory you need to buy. Many of both SOASTA’s and UTest’s customers have on-premise solutions. So even if your company fears the cloud as a solution for deploying applications and building data centers, there is no excuse for continuing to spend way too much time and money testing things the “way we have always done it”.
Did you enjoy this post? Why not leave a comment below and continue the conversation, or subscribe to my feed and get articles like this delivered automatically to your feed reader.
Comments
Good evening.
I remember a few years ago, when people deployed virtualization for their test environments. Knowing what I know from working in Information Systems for the last ten years, your data is actually no safer at the data center inside your building than it would be in the cloud, technically.
It is all about a level of “risk” that you are willing to take. I like to use the comparison to driving down the highway, where there is about an inch or two of paint separating your car from the next one driving down the road. In this case, it is actually very dangerous, but you’ve built up a tolerance of the level of risk.
Now, this isn’t necessarily an apples to apples comparison, but I must assert that, knowing what I technically know, the data isn’t any more secure inside your on-premises datacenter, than it is with a cloud provider.
The only issue I may have with the cloud is how low one can get the latency on the network connection. If latency is an issue, you can still transfer most line of business apps that aren’t that latency-sensitive to the cloud. As more and more vendors design their apps to be less sensitive to latency, and as clients get lower-latency connections to their clouds, then this will be a non-issue.
If I was not clear before, I technically believe that data is not more or less secure in the cloud than on-premises.
If an organization can make a cost-effective deployment to the cloud, then that is what they should do.
[...] This post was mentioned on Twitter by . said: [...]