Welcome!

@CloudExpo Authors: Elizabeth White, Jyoti Bansal, Yeshim Deniz, Greg Schulz, ManageEngine IT Matters

Related Topics: @CloudExpo, Java IoT, Microservices Expo, Microsoft Cloud, Containers Expo Blog

@CloudExpo: Article

Hip? Or Hype? Load and Performance Testing from the Cloud

Some common approaches in choosing a cloud solution

Assuming you haven't spent the last couple of years living under a rock, you're bound to have been bombarded with all sorts of propaganda about "The Cloud." "The Cloud," according to the marketing types, is the greatest thing since the invention of bread, surely able to solve all of our needs, whether technology-related or not. While the hype for the cloud might be frequently and frustratingly overstated and confusingly applied in odd places (can someone please explain to me the Microsoft Cloud commercial where the woman goes "to the cloud!!" so she can generate a family photo? What does this have to do with the cloud? Isn't this just Photoshop?), I'd like to discuss one place where the cloud adds a great deal of value: Load and Performance testing.

In this article, I'm going to talk about the benefits of using the cloud as part of your load and performance testing practices, as well as point out some common approaches in choosing a cloud solution.

In a future article, I'll discuss some of the challenges the cloud presents and some best practices to deal with these challenges.

For those of you who have ever heard me present on load testing, you are well aware of how fixated I am on the need for load testing that truly emulates the real world behavior of your end users. And for those that haven't, understand that, in my opinion, if you are not doing accurate and realistic load testing, you might as well stop now. Inaccurate load testing, in my view, is often worse than doing no testing at all. I've seen far too many organizations perform inaccurate testing, that is testing that falls short of mirroring what their end users are really doing, and use results of this testing to lull themselves into a false sense of security. Then they go live with their web application, which in turn experiences end-user behavior that wasn't fully tested for, and are surprised when their web application falls down.

In my mind, The Cloud immediately brings two distinct advantages to our load and performance procedures that help us better model this realistic behavior. The first is what I call "Instant Infrastructure."

In today's economy, customer-facing web applications are experiencing vastly increased user loads. These loads might increase consistently over time due the success of your business, or they might be more sporadic in nature, perhaps due to a seasonal sales or new advertising campaign. As we know, if your application is unable to handle this increased load - whether the application experiences poor performance or downtime (or both) - you run the risk of losing business or experiencing damage to your brand.

To avoid these potential problems, performance testing should include load tests with very large user loads. Unfortunately, trying to maintain a test environment that generates this kind of load can be both costly and challenging. Creating an environment of this size can typically require tens or even hundreds of machines. Purchasing and configuring these systems requires a significant investment of time and money. After the machines have been acquired, configured and used for the immediate load testing need, they may end up sitting unused for long stretches until they are needed for the next large-scale load testing project. Not a very cost-effective approach in a lot of situations.

Here's where the "Instant Infrastructure" comes in. By accessing a test infrastructure in the cloud, you can rapidly access as many load generating machines as you need, on demand. With this approach there is no need to spend weeks setting up and configuring dozens of real machines. The cloud testing provider will automate this process as well as keep all the machines updated. This obviously saves you from the substantial up-front costs of purchasing and maintaining the load-generating machines. Most cloud testing providers also use a pay-as-you-go model, for which you rapidly access the testing infrastructure you need, when you need it, and only for as long as you need it. From a business standpoint, the cloud lowers total cost of ownership, while increasing flexibility.

The second key way the Cloud helps us achieve more realistic load tests is its ability to allow testing from disparate locations. In many cases your real-world users are not sitting inside your firewall; instead they are probably accessing your location from their office, their home - locations around the country or around the globe. As such, for performance testing to be accurate it must include load generated from these locations as well. If your testing only uses load-generating machines inside your firewall, you're not testing the entire delivery chain. With the cloud, you can execute load tests that access your web application just as your users will - from outside of your firewall - and validate all components of the delivery chain, including the firewall, DNS, network equipment, and ISP. These tests are not only more realistic, but they also allow you to understand the impact of third-party components, such as content delivery networks, analytics servers, and ad servers. And you know how I feel about realistic tests.

Now that I've discussed the advantages that the cloud can bring to your load testing, I'd like to talk a bit about some of the limitations that relying solely on a cloud solution might present. Like any other part of our testing effort, understanding the tools we have at our disposal and how to use them is the key to being successful. Just because we have some great tools in our toolbox doesn't necessarily mean they will help us everywhere, especially if they aren't used correctly. The cloud is no exception. First of all, load testing from the cloud should not replace internal testing; that is, testing with load generated from within the firewall. Too often, I've seen organizations complete their load testing scripting and then move right away to running large-scale tests with the cloud. To me, this approach misses a key step. Load testing cycles should include a combination of tests run from both inside and outside the firewall. You may be wondering if I'm contradicting myself - based on my earlier cloud discussions, the cloud seems like the way to go with all of our load testing, doesn't it? Why would I want to do anything else? There are a couple of reasons why "inside the firewall" testing should still be considered.

First, as we've already seen, driving the test load from cloud load generators allows us to test the whole delivery chain. So our performance measures will include not only the performance of the application but of also all the networking pieces that exist between the application and the end user. Running a load test from inside the firewall will provide us with performance counters of really just the application itself. Combining the results of these two tests can yield some very powerful analysis. If we correlate the same measures gathered from "inside the firewall" tests with those gathered from tests run from the cloud, we can get a much better sense of the impact the delivery chain itself has on the measured performance. Without these combined tests, it can be very difficult, if not impossible, to understand the true root cause of performance issues - whether they are impacted by things under our control (inside the firewall) or those we may not have control over (outside the firewall).

As you can see, the cloud is opening new opportunities to improve the scale and realism of load testing as well as saving time and lowering costs. But, in order to realize these benefits of the cloud, a proper approach is required. Part of this approach is the actual selection of a cloud load testing solution. When selecting a cloud testing solution, keep in mind that it's not just about moving to the cloud itself but also making sure you choose the right solution. In my next article, I will discuss some ideas of how to best approach the selection of a cloud testing solution.

More Stories By Steve Weisfeldt

Steve Weisfeldt is a Senior Performance Engineer at Neotys, a provider of load testing software for Web applications. Previously, he has worked as the President of Engine 1 Consulting, a services firm specializing in all facets of test automation. Prior to his involvement at Engine 1 Consulting, he was a Senior Systems Engineer at Aternity. Prior to that, Steve spent seven years at automated testing vendor Segue Software (acquired by Borland). While spending most of his time at Segue delivering professional services and training, he was also involved in pre-sales and product marketing efforts.

Being in the load and performance testing space since 1999, Steve has been involved in load and performance testing projects of all sizes, in industries that span the retail, financial services, insurance and manufacturing sectors. His expertise lies in enabling organizations to optimize their ability to develop, test and launch high-quality applications efficiently, on-time and on-budget. Steve graduated from the University of Massachusetts-Lowell with a BS in Electrical Engineering and an MS in Computer Engineering.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
DevOps and microservices are permeating software engineering teams broadly, whether these teams are in pure software shops but happen to run a business, such Uber and Airbnb, or in companies that rely heavily on software to run more traditional business, such as financial firms or high-end manufacturers. Microservices and DevOps have created software development and therefore business speed and agility benefits, but they have also created problems; specifically, they have created software securi...
All clouds are not equal. To succeed in a DevOps context, organizations should plan to develop/deploy apps across a choice of on-premise and public clouds simultaneously depending on the business needs. This is where the concept of the Lean Cloud comes in - resting on the idea that you often need to relocate your app modules over their life cycles for both innovation and operational efficiency in the cloud. In his session at @DevOpsSummit at19th Cloud Expo, Valentin (Val) Bercovici, CTO of Soli...
Almost two-thirds of companies either have or soon will have IoT as the backbone of their business. Though, IoT is far more complex than most firms expected with a majority of IoT projects having failed. How can you not get trapped in the pitfalls? In his session at @ThingsExpo, Tony Shan, Chief IoTologist at Wipro, will introduce a holistic method of IoTification, which is the process of IoTifying the existing technology portfolios and business models to adopt and leverage IoT. He will delve in...
Cloud Expo, Inc. has announced today that Aruna Ravichandran, vice president of DevOps Product and Solutions Marketing at CA Technologies, has been named co-conference chair of DevOps at Cloud Expo 2017. The @DevOpsSummit at Cloud Expo New York will take place on June 6-8, 2017, at the Javits Center in New York City, New York, and @DevOpsSummit at Cloud Expo Silicon Valley will take place Oct. 31-Nov. 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
Bert Loomis was a visionary. This general session will highlight how Bert Loomis and people like him inspire us to build great things with small inventions. In their general session at 19th Cloud Expo, Harold Hannon, Architect at IBM Bluemix, and Michael O'Neill, Strategic Business Development at Nvidia, discussed the accelerating pace of AI development and how IBM Cloud and NVIDIA are partnering to bring AI capabilities to "every day," on-demand. They also reviewed two "free infrastructure" pr...
In his session at @ThingsExpo, Steve Wilkes, CTO and founder of Striim, will delve into four enterprise-scale, business-critical case studies where streaming analytics serves as the key to enabling real-time data integration and right-time insights in hybrid cloud, IoT, and fog computing environments. As part of this discussion, he will also present a demo based on its partnership with Fujitsu, highlighting their technologies in a healthcare IoT use-case. The demo showcases the tracking of pati...
Tricky charts and visually deceptive graphs often make a case for the impact IT performance has on business. The debate isn't around the obvious; of course, IT performance metrics like website load time influence business metrics such as conversions and revenue. Rather, this presentation will explore various data analysis concepts to understand how, and how not to, assert such correlations. In his session at 20th Cloud Expo, Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Sys...
Stratoscale, the software company developing the next generation data center operating system, exhibited at SYS-CON's 18th International Cloud Expo®, which took place at the Javits Center in New York City, NY, in June 2016.Stratoscale is revolutionizing the data center with a zero-to-cloud-in-minutes solution. With Stratoscale’s hardware-agnostic, Software Defined Data Center (SDDC) solution to store everything, run anything and scale everywhere, IT is empowered to take control of their data ce...
The buzz continues for cloud, data analytics and the Internet of Things (IoT) and their collective impact across all industries. But a new conversation is emerging - how do companies use industry disruption and technology enablers to lead in markets undergoing change, uncertainty and ambiguity? Organizations of all sizes need to evolve and transform, often under massive pressure, as industry lines blur and merge and traditional business models are assaulted and turned upside down. In this new da...
It is one thing to build single industrial IoT applications, but what will it take to build the Smart Cities and truly society changing applications of the future? The technology won’t be the problem, it will be the number of parties that need to work together and be aligned in their motivation to succeed. In his Day 2 Keynote at @ThingsExpo, Henrik Kenani Dahlgren, Portfolio Marketing Manager at Ericsson, discussed how to plan to cooperate, partner, and form lasting all-star teams to change the...
For organizations that have amassed large sums of software complexity, taking a microservices approach is the first step toward DevOps and continuous improvement / development. Integrating system-level analysis with microservices makes it easier to change and add functionality to applications at any time without the increase of risk. Before you start big transformation projects or a cloud migration, make sure these changes won’t take down your entire organization.
What are the new priorities for the connected business? First: businesses need to think differently about the types of connections they will need to make – these span well beyond the traditional app to app into more modern forms of integration including SaaS integrations, mobile integrations, APIs, device integration and Big Data integration. It’s important these are unified together vs. doing them all piecemeal. Second, these types of connections need to be simple to design, adapt and configure...
To manage complex web services with lots of calls to the cloud, many businesses have invested in Application Performance Management (APM) and Network Performance Management (NPM) tools. Together APM and NPM tools are essential aids in improving a business's infrastructure required to support an effective web experience... but they are missing a critical component - Internet visibility.
Microservices are a very exciting architectural approach that many organizations are looking to as a way to accelerate innovation. Microservices promise to allow teams to move away from monolithic "ball of mud" systems, but the reality is that, in the vast majority of organizations, different projects and technologies will continue to be developed at different speeds. How to handle the dependencies between these disparate systems with different iteration cycles? Consider the "canoncial problem" ...
Both SaaS vendors and SaaS buyers are going “all-in” to hyperscale IaaS platforms such as AWS, which is disrupting the SaaS value proposition. Why should the enterprise SaaS consumer pay for the SaaS service if their data is resident in adjacent AWS S3 buckets? If both SaaS sellers and buyers are using the same cloud tools, automation and pay-per-transaction model offered by IaaS platforms, then why not host the “shrink-wrapped” software in the customers’ cloud? Further, serverless computing, cl...
“We're a global managed hosting provider. Our core customer set is a U.S.-based customer that is looking to go global,” explained Adam Rogers, Managing Director at ANEXIA, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
The speed of software changes in growing and large scale rapid-paced DevOps environments presents a challenge for continuous testing. Many organizations struggle to get this right. Practices that work for small scale continuous testing may not be sufficient as the requirements grow. In his session at DevOps Summit, Marc Hornbeek, Sr. Solutions Architect of DevOps continuous test solutions at Spirent Communications, explained the best practices of continuous testing at high scale, which is rele...
Hardware virtualization and cloud computing allowed us to increase resource utilization and increase our flexibility to respond to business demand. Docker Containers are the next quantum leap - Are they?! Databases always represented an additional set of challenges unique to running workloads requiring a maximum of I/O, network, CPU resources combined with data locality.
"Matrix is an ambitious open standard and implementation that's set up to break down the fragmentation problems that exist in IP messaging and VoIP communication," explained John Woolf, Technical Evangelist at Matrix, in this SYS-CON.tv interview at @ThingsExpo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
"A lot of times people will come to us and have a very diverse set of requirements or very customized need and we'll help them to implement it in a fashion that you can't just buy off of the shelf," explained Nick Rose, CTO of Enzu, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.