Welcome!

@CloudExpo Authors: Elizabeth White, John Rauser, Mark Leake, Liz McMillan, Derek Weeks

Related Topics: @CloudExpo, Microservices Expo, Containers Expo Blog, Agile Computing

@CloudExpo: Blog Post

Considerations for Choosing a Cloud Load Testing Solution

How to pick a tool to load test in the cloud

In a previous post, I've discussed how load testing with the cloud can enable your team to conduct more efficient and more realistic large scale tests while saving you (time and) money. With this all in mind, how do we go about choosing the correct cloud load testing solution?

There are many cloud load testing solutions that will enable you to make use of the cloud. However, very few enable you to capitalize on the opportunities of load testing in the cloud. As I've discussed, load testing with the cloud offers clear advantages over traditional load testing in certain circumstances, but the tools you use are even more important to the quality of your tests.

When considering a cloud testing solution, ask the following questions:

  • To what extent does the solution integrate with the cloud?
  • Will the solution enable us to conduct realistic tests?
  • Does the solution support unified tests inside and outside the firewall?
  • Is the solution easy to use, or will we spend weeks learning and configuring it?
  • Does the solution include full-featured reporting and decision-making modules to help our team make the most of the results?
  • Does the solution support the technologies we used to build the application?

Much of this is outlined in the Neotys Cloud Whitepaper, but for the purposes of this blog, let's briefly talk about each of these:

Integration with the Cloud Platform
If you choose a solution that is not integrated with one or more cloud platforms, there are several manual steps you'll need to perform on your own. First, you'll need to learn how each platform you'll be using works, including its limitations and constraints. Second, you'll need to build, maintain and access the machines you wish to use.

Load testing solutions that offer integration with the cloud simplify and accelerate the steps needed to use the cloud infrastructure. With the proper solution, connecting to, and utilizing the cloud should be possible with just a few mouse clicks.

Realistic Load Tests
Simply moving to the cloud is not always enough to ensure the most realistic tests. Real users often have access to less bandwidth than a load generator in a cloud data center. With a slower connection, the real user will have to wait longer than the load generator to download all the data needed for a web page or application. This has some major implications as your performance results won't be fully reflective of what your real users will experience (here we go again with the inaccurate load testing).

When choosing a load testing solution, look for one that provides a bandwidth simulation feature that limits bandwidth to ensure that the virtual users download the content of the web application at a realistic rate. This capability is particularly important when testing mobile applications, because mobile devices typically operate with less bandwidth than laptops and desktops.

Unified Lab Testing and Cloud Testing
While including the cloud in our load testing is obviously a huge advantage, it does not mean that we should exclude testing from within our own firewall. Both types of testing should be included - if we are only doing one or the other, we are again running into a disadvantage. Likewise if we use different solutions for these different tests.

When choosing a load testing solution, you should make sure that it supports lab testing so you can confirm the performance on your application(s) internally, before you introduce additional variables by testing over the internet. This allows you to shake out any early performance issues with smaller loads before you execute tests from the cloud. Finding these issues earlier in the application's lifecycle will lower costs associated with not only the cloud but also those attributed to "late stage bugs".

Making sure that you use a single solution for both "internal" (within the lab and/or firewall) and "external" (via The Cloud) testing allows your engineers to reuse scripts across both types of tests. If you choose different solutions, you may have to create a new set of scripts for your external testing, costing you both time and money.

Ease of Use
As every test engineer knows (not just the performance testers, the functional testers can relate too!), testing is almost always performed under tight time constraints. Delays in development or other phases earlier in the lifecycle result in less time for the test engineers to do their jobs. The pressure is on to deliver results as quickly as possible. With this kind of stress, there is no place for a solution that is hard to use. Finding a tool that is easy to use is imperative. Not only does it save2 time and money, but it will allow you to get more testing done in the time you have - ensuring a higher quality application. While an exhaustive list of "ease of use" features might be best suited for another blog article, some key things you should look for are:

  • Easy recording of a virtual user profile (preferably in one click).
  • The ability to easily define advanced behaviors (with structures such as conditions and loops) via a graphical interface, without a need for writing code or test scripts.
  • Automatic handling of session parameters. This is probably the most challenging and time-consuming part of in the development of load testing user profiles. Find a tool that does this for you automatically.
  • Easy results comparisons. Generating graphs and charts based on a lot of raw data can be excruciating - a tool that automates this ability is a must.

Analysis, Monitoring, and Reporting
Creating realistic scenarios and running comprehensive load tests is only the part of the story. The final steps of course are to understand how your application behaved while it was subjected to the test load. Did it meet performance requirements? If not, why? To do this analysis, you need a powerful set of analysis tools to analyze the data generated during the test.

Make sure you choose a solution that allows you to easily analyze the collected data, creating actionable reports that describe the performance of the tested application. I've seen too many a test engineer spend hours, if not days taking raw data from a load testing tool and pushing it through Excel or a statistical tool to create graphs, charts and reports. A tool that allows for easy analysis not only makes this process pain-free but can increase the collaboration among the different stakeholders of the application's infrastructure.

Making sure your load testing solution contains a comprehensive monitoring system is also essential when you need to find the root causes of a problem. If you are only looking at performance measure collected from the end-users point of view, you're missing half the picture. Being able to use a monitoring system allows you to understand what was happening within the infrastructure of your web application is critical in identifying any root issues. Moreover, the selected load testing solution should allow for easy correlation between data collected from the end-users point of view and data collected from any back-end servers. Without this ability, results analysis becomes extremely cumbersome as well as potentially inaccurate. Too many times in my load testing career I've tried to correlate performance data collected from a load testing tool with that of data collected from separate application server and database server monitoring tools. An almost impossible task.

Support for Web Technologies
This one is pretty simple. If your application has been developed using more advanced web technologies, make sure the solution you choose has the proper support. With applications built with Adobe Flex, Microsoft Silverlight, Real-Time Messaging Protocol (RTMP), AJAX push technologies, etc. becoming more and more prevalent, the proper technology support is critical. Without this support, effectively testing the performance of your application is next to impossible. I'm sure that any of you that have tried to load test a Siebel application are nodding your heads as you read this.

While considering the above topics to search for a cloud provider, you may also want to consider using MULTIPLE cloud computing providers. There are several advantages to this approach.

First, multiple providers may allow you to test from more geographical regions. Earlier, we've talked about how testing from multiple regions can provide a more realistic test scenario. With this in mind, combine the regions available from multiple providers can lead to an even MORE realistic scenario. In other words, if a single provider does not give you the geographical coverage you need to emulate (based on where the real world users of your application "live"), aggregating multiple regions can be very useful.

Second, if you are executing exceptionally large scale tests, engaging multiple providers simultaneously allows you to bypass any limitations that a single provider may place on bandwidth or the number of machines in use.

Finally, using multiple cloud providers enables you to detect potential network issues at the cloud provider level. Let's say that during results analysis of particular test run you noting that there is significantly worse performance measure from load generators from a particular cloud provider while all other load generator show acceptable performance. With this data, you can safely conclude that there is a problem (temporary or not) with that provider and not your application. If you had locked yourself into a single cloud provider, you might be limited in your ability to conduct accurate large-scale tests.

More Stories By Steve Weisfeldt

Steve Weisfeldt is a Senior Performance Engineer at Neotys, a provider of load testing software for Web applications. Previously, he has worked as the President of Engine 1 Consulting, a services firm specializing in all facets of test automation. Prior to his involvement at Engine 1 Consulting, he was a Senior Systems Engineer at Aternity. Prior to that, Steve spent seven years at automated testing vendor Segue Software (acquired by Borland). While spending most of his time at Segue delivering professional services and training, he was also involved in pre-sales and product marketing efforts.

Being in the load and performance testing space since 1999, Steve has been involved in load and performance testing projects of all sizes, in industries that span the retail, financial services, insurance and manufacturing sectors. His expertise lies in enabling organizations to optimize their ability to develop, test and launch high-quality applications efficiently, on-time and on-budget. Steve graduated from the University of Massachusetts-Lowell with a BS in Electrical Engineering and an MS in Computer Engineering.

@CloudExpo Stories
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), provided an overview of various initiatives to certify the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldwide re...
It is ironic, but perhaps not unexpected, that many organizations who want the benefits of using an Agile approach to deliver software use a waterfall approach to adopting Agile practices: they form plans, they set milestones, and they measure progress by how many teams they have engaged. Old habits die hard, but like most waterfall software projects, most waterfall-style Agile adoption efforts fail to produce the results desired. The problem is that to get the results they want, they have to ch...
The current age of digital transformation means that IT organizations must adapt their toolset to cover all digital experiences, beyond just the end users’. Today’s businesses can no longer focus solely on the digital interactions they manage with employees or customers; they must now contend with non-traditional factors. Whether it's the power of brand to make or break a company, the need to monitor across all locations 24/7, or the ability to proactively resolve issues, companies must adapt to...
Wooed by the promise of faster innovation, lower TCO, and greater agility, businesses of every shape and size have embraced the cloud at every layer of the IT stack – from apps to file sharing to infrastructure. The typical organization currently uses more than a dozen sanctioned cloud apps and will shift more than half of all workloads to the cloud by 2018. Such cloud investments have delivered measurable benefits. But they’ve also resulted in some unintended side-effects: complexity and risk. ...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
In 2014, Amazon announced a new form of compute called Lambda. We didn't know it at the time, but this represented a fundamental shift in what we expect from cloud computing. Now, all of the major cloud computing vendors want to take part in this disruptive technology. In his session at 20th Cloud Expo, Doug Vanderweide, an instructor at Linux Academy, discussed why major players like AWS, Microsoft Azure, IBM Bluemix, and Google Cloud Platform are all trying to sidestep VMs and containers wit...
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities. In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, posited that disruption is inevitable for comp...
No hype cycles or predictions of zillions of things here. IoT is big. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, Associate Partner at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He discussed the evaluation of communication standards and IoT messaging protocols, data analytics considerations, edge-to-cloud tec...
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists examined how DevOps helps to meet the de...
When growing capacity and power in the data center, the architectural trade-offs between server scale-up vs. scale-out continue to be debated. Both approaches are valid: scale-out adds multiple, smaller servers running in a distributed computing model, while scale-up adds fewer, more powerful servers that are capable of running larger workloads. It’s worth noting that there are additional, unique advantages that scale-up architectures offer. One big advantage is large memory and compute capacity...
"When we talk about cloud without compromise what we're talking about is that when people think about 'I need the flexibility of the cloud' - it's the ability to create applications and run them in a cloud environment that's far more flexible,” explained Matthew Finnie, CTO of Interoute, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Cloud applications are seeing a deluge of requests to support the exploding advanced analytics market. “Open analytics” is the emerging strategy to deliver that data through an open data access layer, in the cloud, to be directly consumed by external analytics tools and popular programming languages. An increasing number of data engineers and data scientists use a variety of platforms and advanced analytics languages such as SAS, R, Python and Java, as well as frameworks such as Hadoop and Spark...
The Internet giants are fully embracing AI. All the services they offer to their customers are aimed at drawing a map of the world with the data they get. The AIs from these companies are used to build disruptive approaches that cannot be used by established enterprises, which are threatened by these disruptions. However, most leaders underestimate the effect this will have on their businesses. In his session at 21st Cloud Expo, Rene Buest, Director Market Research & Technology Evangelism at Ara...
Join us at Cloud Expo June 6-8 to find out how to securely connect your cloud app to any cloud or on-premises data source – without complex firewall changes. More users are demanding access to on-premises data from their cloud applications. It’s no longer a “nice-to-have” but an important differentiator that drives competitive advantages. It’s the new “must have” in the hybrid era. Users want capabilities that give them a unified view of the data to get closer to customers and grow business. The...
"We are a monitoring company. We work with Salesforce, BBC, and quite a few other big logos. We basically provide monitoring for them, structure for their cloud services and we fit into the DevOps world" explained David Gildeh, Co-founder and CEO of Outlyer, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"Loom is applying artificial intelligence and machine learning into the entire log analysis process, from start to finish and at the end you will get a human touch,” explained Sabo Taylor Diab, Vice President, Marketing at Loom Systems, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"Tintri focuses on the Ops side of the DevOps, which basically is pushing more and more of the accessibility of the infrastructure to the developers and trying to get behind the scenes," explained Dhiraj Sehgal of Tintri in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
@DevOpsSummit at Cloud Expo taking place Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center, Santa Clara, CA, is co-located with the 21st International Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is ...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
A look across the tech landscape at the disruptive technologies that are increasing in prominence and speculate as to which will be most impactful for communications – namely, AI and Cloud Computing. In his session at 20th Cloud Expo, Curtis Peterson, VP of Operations at RingCentral, highlighted the current challenges of these transformative technologies and shared strategies for preparing your organization for these changes. This “view from the top” outlined the latest trends and developments i...