Welcome!

@CloudExpo Authors: Yeshim Deniz, Elizabeth White, Pat Romanski, Liz McMillan, Aruna Ravichandran

Related Topics: @CloudExpo, Java IoT, Microservices Expo, Microsoft Cloud, Containers Expo Blog

@CloudExpo: Article

Hip? Or Hype? Load and Performance Testing from the Cloud

Some common approaches in choosing a cloud solution

Assuming you haven't spent the last couple of years living under a rock, you're bound to have been bombarded with all sorts of propaganda about "The Cloud." "The Cloud," according to the marketing types, is the greatest thing since the invention of bread, surely able to solve all of our needs, whether technology-related or not. While the hype for the cloud might be frequently and frustratingly overstated and confusingly applied in odd places (can someone please explain to me the Microsoft Cloud commercial where the woman goes "to the cloud!!" so she can generate a family photo? What does this have to do with the cloud? Isn't this just Photoshop?), I'd like to discuss one place where the cloud adds a great deal of value: Load and Performance testing.

In this article, I'm going to talk about the benefits of using the cloud as part of your load and performance testing practices, as well as point out some common approaches in choosing a cloud solution.

In a future article, I'll discuss some of the challenges the cloud presents and some best practices to deal with these challenges.

For those of you who have ever heard me present on load testing, you are well aware of how fixated I am on the need for load testing that truly emulates the real world behavior of your end users. And for those that haven't, understand that, in my opinion, if you are not doing accurate and realistic load testing, you might as well stop now. Inaccurate load testing, in my view, is often worse than doing no testing at all. I've seen far too many organizations perform inaccurate testing, that is testing that falls short of mirroring what their end users are really doing, and use results of this testing to lull themselves into a false sense of security. Then they go live with their web application, which in turn experiences end-user behavior that wasn't fully tested for, and are surprised when their web application falls down.

In my mind, The Cloud immediately brings two distinct advantages to our load and performance procedures that help us better model this realistic behavior. The first is what I call "Instant Infrastructure."

In today's economy, customer-facing web applications are experiencing vastly increased user loads. These loads might increase consistently over time due the success of your business, or they might be more sporadic in nature, perhaps due to a seasonal sales or new advertising campaign. As we know, if your application is unable to handle this increased load - whether the application experiences poor performance or downtime (or both) - you run the risk of losing business or experiencing damage to your brand.

To avoid these potential problems, performance testing should include load tests with very large user loads. Unfortunately, trying to maintain a test environment that generates this kind of load can be both costly and challenging. Creating an environment of this size can typically require tens or even hundreds of machines. Purchasing and configuring these systems requires a significant investment of time and money. After the machines have been acquired, configured and used for the immediate load testing need, they may end up sitting unused for long stretches until they are needed for the next large-scale load testing project. Not a very cost-effective approach in a lot of situations.

Here's where the "Instant Infrastructure" comes in. By accessing a test infrastructure in the cloud, you can rapidly access as many load generating machines as you need, on demand. With this approach there is no need to spend weeks setting up and configuring dozens of real machines. The cloud testing provider will automate this process as well as keep all the machines updated. This obviously saves you from the substantial up-front costs of purchasing and maintaining the load-generating machines. Most cloud testing providers also use a pay-as-you-go model, for which you rapidly access the testing infrastructure you need, when you need it, and only for as long as you need it. From a business standpoint, the cloud lowers total cost of ownership, while increasing flexibility.

The second key way the Cloud helps us achieve more realistic load tests is its ability to allow testing from disparate locations. In many cases your real-world users are not sitting inside your firewall; instead they are probably accessing your location from their office, their home - locations around the country or around the globe. As such, for performance testing to be accurate it must include load generated from these locations as well. If your testing only uses load-generating machines inside your firewall, you're not testing the entire delivery chain. With the cloud, you can execute load tests that access your web application just as your users will - from outside of your firewall - and validate all components of the delivery chain, including the firewall, DNS, network equipment, and ISP. These tests are not only more realistic, but they also allow you to understand the impact of third-party components, such as content delivery networks, analytics servers, and ad servers. And you know how I feel about realistic tests.

Now that I've discussed the advantages that the cloud can bring to your load testing, I'd like to talk a bit about some of the limitations that relying solely on a cloud solution might present. Like any other part of our testing effort, understanding the tools we have at our disposal and how to use them is the key to being successful. Just because we have some great tools in our toolbox doesn't necessarily mean they will help us everywhere, especially if they aren't used correctly. The cloud is no exception. First of all, load testing from the cloud should not replace internal testing; that is, testing with load generated from within the firewall. Too often, I've seen organizations complete their load testing scripting and then move right away to running large-scale tests with the cloud. To me, this approach misses a key step. Load testing cycles should include a combination of tests run from both inside and outside the firewall. You may be wondering if I'm contradicting myself - based on my earlier cloud discussions, the cloud seems like the way to go with all of our load testing, doesn't it? Why would I want to do anything else? There are a couple of reasons why "inside the firewall" testing should still be considered.

First, as we've already seen, driving the test load from cloud load generators allows us to test the whole delivery chain. So our performance measures will include not only the performance of the application but of also all the networking pieces that exist between the application and the end user. Running a load test from inside the firewall will provide us with performance counters of really just the application itself. Combining the results of these two tests can yield some very powerful analysis. If we correlate the same measures gathered from "inside the firewall" tests with those gathered from tests run from the cloud, we can get a much better sense of the impact the delivery chain itself has on the measured performance. Without these combined tests, it can be very difficult, if not impossible, to understand the true root cause of performance issues - whether they are impacted by things under our control (inside the firewall) or those we may not have control over (outside the firewall).

As you can see, the cloud is opening new opportunities to improve the scale and realism of load testing as well as saving time and lowering costs. But, in order to realize these benefits of the cloud, a proper approach is required. Part of this approach is the actual selection of a cloud load testing solution. When selecting a cloud testing solution, keep in mind that it's not just about moving to the cloud itself but also making sure you choose the right solution. In my next article, I will discuss some ideas of how to best approach the selection of a cloud testing solution.

More Stories By Steve Weisfeldt

Steve Weisfeldt is a Senior Performance Engineer at Neotys, a provider of load testing software for Web applications. Previously, he has worked as the President of Engine 1 Consulting, a services firm specializing in all facets of test automation. Prior to his involvement at Engine 1 Consulting, he was a Senior Systems Engineer at Aternity. Prior to that, Steve spent seven years at automated testing vendor Segue Software (acquired by Borland). While spending most of his time at Segue delivering professional services and training, he was also involved in pre-sales and product marketing efforts.

Being in the load and performance testing space since 1999, Steve has been involved in load and performance testing projects of all sizes, in industries that span the retail, financial services, insurance and manufacturing sectors. His expertise lies in enabling organizations to optimize their ability to develop, test and launch high-quality applications efficiently, on-time and on-budget. Steve graduated from the University of Massachusetts-Lowell with a BS in Electrical Engineering and an MS in Computer Engineering.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
SYS-CON Events announced today that SourceForge has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SourceForge is the largest, most trusted destination for Open Source Software development, collaboration, discovery and download on the web serving over 32 million viewers, 150 million downloads and over 460,000 active development projects each and every month.
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities – ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups. As a result, many firms employ new business models that place enormous impor...
SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
As popularity of the smart home is growing and continues to go mainstream, technological factors play a greater role. The IoT protocol houses the interoperability battery consumption, security, and configuration of a smart home device, and it can be difficult for companies to choose the right kind for their product. For both DIY and professionally installed smart homes, developers need to consider each of these elements for their product to be successful in the market and current smart homes.
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
In his general session at 21st Cloud Expo, Greg Dumas, Calligo’s Vice President and G.M. of US operations, will go over the new Global Data Protection Regulation and how Calligo can help business stay compliant in digitally globalized world. Greg Dumas is Calligo's Vice President and G.M. of US operations. Calligo is an established service provider that provides an innovative platform for trusted cloud solutions. Calligo’s customers are typically most concerned about GDPR compliance, applicatio...
Companies are harnessing data in ways we once associated with science fiction. Analysts have access to a plethora of visualization and reporting tools, but considering the vast amount of data businesses collect and limitations of CPUs, end users are forced to design their structures and systems with limitations. Until now. As the cloud toolkit to analyze data has evolved, GPUs have stepped in to massively parallel SQL, visualization and machine learning.
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...
As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp emp...
SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.
In his session at 21st Cloud Expo, Michael Burley, a Senior Business Development Executive in IT Services at NetApp, will describe how NetApp designed a three-year program of work to migrate 25PB of a major telco's enterprise data to a new STaaS platform, and then secured a long-term contract to manage and operate the platform. This significant program blended the best of NetApp’s solutions and services capabilities to enable this telco’s successful adoption of private cloud storage and launchi...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It’s clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. Tha...