Welcome!

@CloudExpo Authors: Yeshim Deniz, Elizabeth White, Pat Romanski, Liz McMillan, Aruna Ravichandran

Related Topics: @CloudExpo, Microservices Expo, Containers Expo Blog, Agile Computing

@CloudExpo: Blog Post

Considerations for Choosing a Cloud Load Testing Solution

How to pick a tool to load test in the cloud

In a previous post, I've discussed how load testing with the cloud can enable your team to conduct more efficient and more realistic large scale tests while saving you (time and) money. With this all in mind, how do we go about choosing the correct cloud load testing solution?

There are many cloud load testing solutions that will enable you to make use of the cloud. However, very few enable you to capitalize on the opportunities of load testing in the cloud. As I've discussed, load testing with the cloud offers clear advantages over traditional load testing in certain circumstances, but the tools you use are even more important to the quality of your tests.

When considering a cloud testing solution, ask the following questions:

  • To what extent does the solution integrate with the cloud?
  • Will the solution enable us to conduct realistic tests?
  • Does the solution support unified tests inside and outside the firewall?
  • Is the solution easy to use, or will we spend weeks learning and configuring it?
  • Does the solution include full-featured reporting and decision-making modules to help our team make the most of the results?
  • Does the solution support the technologies we used to build the application?

Much of this is outlined in the Neotys Cloud Whitepaper, but for the purposes of this blog, let's briefly talk about each of these:

Integration with the Cloud Platform
If you choose a solution that is not integrated with one or more cloud platforms, there are several manual steps you'll need to perform on your own. First, you'll need to learn how each platform you'll be using works, including its limitations and constraints. Second, you'll need to build, maintain and access the machines you wish to use.

Load testing solutions that offer integration with the cloud simplify and accelerate the steps needed to use the cloud infrastructure. With the proper solution, connecting to, and utilizing the cloud should be possible with just a few mouse clicks.

Realistic Load Tests
Simply moving to the cloud is not always enough to ensure the most realistic tests. Real users often have access to less bandwidth than a load generator in a cloud data center. With a slower connection, the real user will have to wait longer than the load generator to download all the data needed for a web page or application. This has some major implications as your performance results won't be fully reflective of what your real users will experience (here we go again with the inaccurate load testing).

When choosing a load testing solution, look for one that provides a bandwidth simulation feature that limits bandwidth to ensure that the virtual users download the content of the web application at a realistic rate. This capability is particularly important when testing mobile applications, because mobile devices typically operate with less bandwidth than laptops and desktops.

Unified Lab Testing and Cloud Testing
While including the cloud in our load testing is obviously a huge advantage, it does not mean that we should exclude testing from within our own firewall. Both types of testing should be included - if we are only doing one or the other, we are again running into a disadvantage. Likewise if we use different solutions for these different tests.

When choosing a load testing solution, you should make sure that it supports lab testing so you can confirm the performance on your application(s) internally, before you introduce additional variables by testing over the internet. This allows you to shake out any early performance issues with smaller loads before you execute tests from the cloud. Finding these issues earlier in the application's lifecycle will lower costs associated with not only the cloud but also those attributed to "late stage bugs".

Making sure that you use a single solution for both "internal" (within the lab and/or firewall) and "external" (via The Cloud) testing allows your engineers to reuse scripts across both types of tests. If you choose different solutions, you may have to create a new set of scripts for your external testing, costing you both time and money.

Ease of Use
As every test engineer knows (not just the performance testers, the functional testers can relate too!), testing is almost always performed under tight time constraints. Delays in development or other phases earlier in the lifecycle result in less time for the test engineers to do their jobs. The pressure is on to deliver results as quickly as possible. With this kind of stress, there is no place for a solution that is hard to use. Finding a tool that is easy to use is imperative. Not only does it save2 time and money, but it will allow you to get more testing done in the time you have - ensuring a higher quality application. While an exhaustive list of "ease of use" features might be best suited for another blog article, some key things you should look for are:

  • Easy recording of a virtual user profile (preferably in one click).
  • The ability to easily define advanced behaviors (with structures such as conditions and loops) via a graphical interface, without a need for writing code or test scripts.
  • Automatic handling of session parameters. This is probably the most challenging and time-consuming part of in the development of load testing user profiles. Find a tool that does this for you automatically.
  • Easy results comparisons. Generating graphs and charts based on a lot of raw data can be excruciating - a tool that automates this ability is a must.

Analysis, Monitoring, and Reporting
Creating realistic scenarios and running comprehensive load tests is only the part of the story. The final steps of course are to understand how your application behaved while it was subjected to the test load. Did it meet performance requirements? If not, why? To do this analysis, you need a powerful set of analysis tools to analyze the data generated during the test.

Make sure you choose a solution that allows you to easily analyze the collected data, creating actionable reports that describe the performance of the tested application. I've seen too many a test engineer spend hours, if not days taking raw data from a load testing tool and pushing it through Excel or a statistical tool to create graphs, charts and reports. A tool that allows for easy analysis not only makes this process pain-free but can increase the collaboration among the different stakeholders of the application's infrastructure.

Making sure your load testing solution contains a comprehensive monitoring system is also essential when you need to find the root causes of a problem. If you are only looking at performance measure collected from the end-users point of view, you're missing half the picture. Being able to use a monitoring system allows you to understand what was happening within the infrastructure of your web application is critical in identifying any root issues. Moreover, the selected load testing solution should allow for easy correlation between data collected from the end-users point of view and data collected from any back-end servers. Without this ability, results analysis becomes extremely cumbersome as well as potentially inaccurate. Too many times in my load testing career I've tried to correlate performance data collected from a load testing tool with that of data collected from separate application server and database server monitoring tools. An almost impossible task.

Support for Web Technologies
This one is pretty simple. If your application has been developed using more advanced web technologies, make sure the solution you choose has the proper support. With applications built with Adobe Flex, Microsoft Silverlight, Real-Time Messaging Protocol (RTMP), AJAX push technologies, etc. becoming more and more prevalent, the proper technology support is critical. Without this support, effectively testing the performance of your application is next to impossible. I'm sure that any of you that have tried to load test a Siebel application are nodding your heads as you read this.

While considering the above topics to search for a cloud provider, you may also want to consider using MULTIPLE cloud computing providers. There are several advantages to this approach.

First, multiple providers may allow you to test from more geographical regions. Earlier, we've talked about how testing from multiple regions can provide a more realistic test scenario. With this in mind, combine the regions available from multiple providers can lead to an even MORE realistic scenario. In other words, if a single provider does not give you the geographical coverage you need to emulate (based on where the real world users of your application "live"), aggregating multiple regions can be very useful.

Second, if you are executing exceptionally large scale tests, engaging multiple providers simultaneously allows you to bypass any limitations that a single provider may place on bandwidth or the number of machines in use.

Finally, using multiple cloud providers enables you to detect potential network issues at the cloud provider level. Let's say that during results analysis of particular test run you noting that there is significantly worse performance measure from load generators from a particular cloud provider while all other load generator show acceptable performance. With this data, you can safely conclude that there is a problem (temporary or not) with that provider and not your application. If you had locked yourself into a single cloud provider, you might be limited in your ability to conduct accurate large-scale tests.

More Stories By Steve Weisfeldt

Steve Weisfeldt is a Senior Performance Engineer at Neotys, a provider of load testing software for Web applications. Previously, he has worked as the President of Engine 1 Consulting, a services firm specializing in all facets of test automation. Prior to his involvement at Engine 1 Consulting, he was a Senior Systems Engineer at Aternity. Prior to that, Steve spent seven years at automated testing vendor Segue Software (acquired by Borland). While spending most of his time at Segue delivering professional services and training, he was also involved in pre-sales and product marketing efforts.

Being in the load and performance testing space since 1999, Steve has been involved in load and performance testing projects of all sizes, in industries that span the retail, financial services, insurance and manufacturing sectors. His expertise lies in enabling organizations to optimize their ability to develop, test and launch high-quality applications efficiently, on-time and on-budget. Steve graduated from the University of Massachusetts-Lowell with a BS in Electrical Engineering and an MS in Computer Engineering.

@CloudExpo Stories
SYS-CON Events announced today that SourceForge has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SourceForge is the largest, most trusted destination for Open Source Software development, collaboration, discovery and download on the web serving over 32 million viewers, 150 million downloads and over 460,000 active development projects each and every month.
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities – ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups. As a result, many firms employ new business models that place enormous impor...
SYS-CON Events announced today that TidalScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale is the leading provider of Software-Defined Servers that bring flexibility to modern data centers by right-sizing servers on the fly to fit any data set or workload. TidalScale’s award-winning inverse hypervisor technology combines multiple commodity servers (including their ass...
As popularity of the smart home is growing and continues to go mainstream, technological factors play a greater role. The IoT protocol houses the interoperability battery consumption, security, and configuration of a smart home device, and it can be difficult for companies to choose the right kind for their product. For both DIY and professionally installed smart homes, developers need to consider each of these elements for their product to be successful in the market and current smart homes.
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
In his general session at 21st Cloud Expo, Greg Dumas, Calligo’s Vice President and G.M. of US operations, will go over the new Global Data Protection Regulation and how Calligo can help business stay compliant in digitally globalized world. Greg Dumas is Calligo's Vice President and G.M. of US operations. Calligo is an established service provider that provides an innovative platform for trusted cloud solutions. Calligo’s customers are typically most concerned about GDPR compliance, applicatio...
Companies are harnessing data in ways we once associated with science fiction. Analysts have access to a plethora of visualization and reporting tools, but considering the vast amount of data businesses collect and limitations of CPUs, end users are forced to design their structures and systems with limitations. Until now. As the cloud toolkit to analyze data has evolved, GPUs have stepped in to massively parallel SQL, visualization and machine learning.
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, will lead you through the exciting evolution of the cloud. He'll look at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering ...
As hybrid cloud becomes the de-facto standard mode of operation for most enterprises, new challenges arise on how to efficiently and economically share data across environments. In his session at 21st Cloud Expo, Dr. Allon Cohen, VP of Product at Elastifile, will explore new techniques and best practices that help enterprise IT benefit from the advantages of hybrid cloud environments by enabling data availability for both legacy enterprise and cloud-native mission critical applications. By rev...
SYS-CON Events announced today that Dasher Technologies will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Dasher Technologies, Inc. ® is a premier IT solution provider that delivers expert technical resources along with trusted account executives to architect and deliver complete IT solutions and services to help our clients execute their goals, plans and objectives. Since 1999, we'v...
SYS-CON Events announced today that NetApp has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. NetApp is the data authority for hybrid cloud. NetApp provides a full range of hybrid cloud data services that simplify management of applications and data across cloud and on-premises environments to accelerate digital transformation. Together with their partners, NetApp emp...
SYS-CON Events announced today that TidalScale, a leading provider of systems and services, will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. TidalScale has been involved in shaping the computing landscape. They've designed, developed and deployed some of the most important and successful systems and services in the history of the computing industry - internet, Ethernet, operating s...
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
SYS-CON Events announced today that Massive Networks, that helps your business operate seamlessly with fast, reliable, and secure internet and network solutions, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. As a premier telecommunications provider, Massive Networks is headquartered out of Louisville, Colorado. With years of experience under their belt, their team of...
Widespread fragmentation is stalling the growth of the IIoT and making it difficult for partners to work together. The number of software platforms, apps, hardware and connectivity standards is creating paralysis among businesses that are afraid of being locked into a solution. EdgeX Foundry is unifying the community around a common IoT edge framework and an ecosystem of interoperable components.
Infoblox delivers Actionable Network Intelligence to enterprise, government, and service provider customers around the world. They are the industry leader in DNS, DHCP, and IP address management, the category known as DDI. We empower thousands of organizations to control and secure their networks from the core-enabling them to increase efficiency and visibility, improve customer service, and meet compliance requirements.
In his session at 21st Cloud Expo, Michael Burley, a Senior Business Development Executive in IT Services at NetApp, will describe how NetApp designed a three-year program of work to migrate 25PB of a major telco's enterprise data to a new STaaS platform, and then secured a long-term contract to manage and operate the platform. This significant program blended the best of NetApp’s solutions and services capabilities to enable this telco’s successful adoption of private cloud storage and launchi...
SYS-CON Events announced today that IBM has been named “Diamond Sponsor” of SYS-CON's 21st Cloud Expo, which will take place on October 31 through November 2nd 2017 at the Santa Clara Convention Center in Santa Clara, California.
Amazon is pursuing new markets and disrupting industries at an incredible pace. Almost every industry seems to be in its crosshairs. Companies and industries that once thought they were safe are now worried about being “Amazoned.”. The new watch word should be “Be afraid. Be very afraid.” In his session 21st Cloud Expo, Chris Kocher, a co-founder of Grey Heron, will address questions such as: What new areas is Amazon disrupting? How are they doing this? Where are they likely to go? What are th...
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It’s clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. Tha...