Welcome!

Cloud Expo Authors: Gary Kaiser, Liz McMillan, Elizabeth White, Kevin Benedict, Cloud Ventures

Related Topics: SOA & WOA, Java, .NET, AJAX & REA, Web 2.0, Cloud Expo

SOA & WOA: Article

Load Testing: Same Old, Same Old or a Whole New Ball Game?

How can we be prepared for the future of load and performance testing?

Bonjour Foxes!

I started my career as a Telecom Engineer for Rational Software in the load testing space back in the late '90s, and when I look back on the last decade, there were enormous advances in the broader IT world including development methodologies, processing speeds, network speeds, mobile devices (it's hard to believe the first iPhone was only released 6 years ago). But the question is: Has the world of load and performance testing really changed all that much? Are the missions the same? Are the challenges different? And how can we be prepared for the future of load and performance testing?

Let's start with what hasn't changed:

Websites crash.
Crashes are a reality today just as much as they were 10 years ago. While many companies have become aware of big events (e.g., Cyber Monday, big ad campaigns) that cause traffic spikes, exact traffic levels are still difficult to predict. One big difference between now and then is the sheer number of websites on the Internet. According to Netcraft, the number of active sites on the Internet has increased by almost 1000% over the past ten years. More websites = more crashes.

Developers still think their code is bug-free.
Being a load tester can sometimes come with the occasional power trip that results from crashing an application written by developers who think their code is perfect. While I'm sure their code is very good, load testing can still quickly uncover any flaws in the code or architecture that can cause performance issues.

Load testing is still pushed off until late in the development cycle.
This is a reality most testers just have to deal with. Testing in general, but particularly load and performance testing, are held off until the end of the development cycle, which can be particularly frustrating for testers who can be blamed for holding up a release.

HTTP is still a connectionless protocol.
Furthermore, it's being used to drive a world that is becoming increasingly connected. In fact, the vast majority of advances in web technologies (cookies, sessions, AJAX, WebSockets, SPDY, etc.) have been created in order to overcome the HTTP connectionless limitations. For load testers, this means it still causes complications.

What Has Changed?
Load testing is becoming a mandatory step in the development process.

I've been talking to more and more companies these days that are requiring that all applications go through load and performance testing before they're deployed to production. This is especially true for new online services that know that they only have one chance to show their best or likely lose that customer forever.

The performance of applications is becoming more important than the breadth of functionality.
For some companies like insurance and banks, the importance of application performance is much higher than the pure number of functions their apps can perform. This means that load and performance testers are playing more prominent roles within these development organizations.

The number of technologies built over HTTP is growing each month.
More and more technologies are being developed to make the web faster, more secure and more reactive, and more and more development organizations are adopting them at faster rates. This means load testers are required to test apps containing complex technologies they've never seen before, and it's no easy task to test apps utilizing AJAX, SPDY, WebSockets, video streaming, etc.

App developers have to consider the performance of their apps on mobile devices and networks.
Morgan Stanley
has predicted that mobile Internet users will surpass desktop Internet users by the end of the year. With this in mind, performance testers need to be able to re-create the use cases and network conditions actual users will experience with several different network types and several different devices.

What can you do to handle the realities of today and be prepared for load testing world of tomorrow?

1. Don't avoid load and performance testing.
If you're one of those people who think your apps will be fine and users will do the testing in production, I hope for your sake that your developers actually do write bug-free code. Remember, your end-users will not be as forgiving or as patient as virtual users are when your application is under high load if you recall the old Amazon.com statistic about losing 1% of sales for every 100 ms delay in response times.

2. Make your tests as realistic as possible.
Load testing is much easier these days with the tools available, but don't think it's a "point & click" operation. I see too many companies running "load tests" that do not simulate the number of users observed in production nor the conditions under which the apps will be used.

3. Make sure your load testing tool can match the rhythm of the technical "dance".
Developers and architects are going to want to take advantage of the latest technologies, even some that are still in beta. As a tester, you and your tool shouldn't be the bottleneck for the product launch. Make sure your tool supports your needs as well as the needs of your development organization.

With all of the advances in web and mobile application technologies and the instant response times end-users expect these days, the performance of applications is only going to grow in importance. My advice to load testers is to stay on top of these trends because they move quickly. It's certainly an exciting time to be a load tester.

More Stories By Hervé Servy

Hervé Servy is a Senior Performance Engineer at Neotys. He has spent 10 years working for IBM-Rational and Microsoft pre-sales and marketing in France and the Middle East. During the past 3 years, as a personal project, Hervé founded a nonprofit organization in the health 2.0 area. If that isn’t techie enough, Hervé was also born on the very same day Apple Computer was founded.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Cloud Expo Breaking News
Cloud Computing is evolving into a Big Three of Amazon Web Services, Google Cloud, and Microsoft Azure. Cloud 360: Multi-Cloud Bootcamp, being held Nov 4–5, 2014, in conjunction with 15th Cloud Expo in Santa Clara, CA, delivers a real-world demonstration of how to deploy and configure a scalable and available web application on all three platforms. The Cloud 360 Bootcamp, led by Janakiram MSV, an analyst with Gigaom Research, is the first bootcamp that introduces the core concepts of Infrastructure as a Service (IaaS) based on the workings of the Big Three platforms – Amazon EC2, Google Compute Engine, and Azure VMs. Bootcamp attendees will get to see the big picture and also receive the knowledge needed to make the best cloud decisions for their business applications and entire enterprise IT organization.
Scott Jenson leads a project called The Physical Web within the Chrome team at Google. Project members are working to take the scalability and openness of the web and use it to talk to the exponentially exploding range of smart devices. Nearly every company today working on the IoT comes up with the same basic solution: use my server and you'll be fine. But if we really believe there will be trillions of these devices, that just can't scale. We need a system that is open a scalable and by using the URL as a basic building block, we open this up and get the same resilience that the web enjoys.
The Internet of Things is a natural complement to the cloud and related technologies such as Big Data, analytics, and mobility. In his session at Internet of @ThingsExpo, Joe Weinman will lay out four generic strategies – digital disciplines – to exploit emerging digital technologies for strategic advantage. Joe Weinman has held executive leadership positions at Bell Labs, AT&T, Hewlett-Packard, and Telx, in areas such as corporate strategy, business development, product management, operations, and R&D.
SYS-CON Events announced today that DevOps.com has been named “Media Sponsor” of SYS-CON's “DevOps Summit at Cloud Expo,” which will take place on June 10–12, 2014, at the Javits Center in New York City, New York. DevOps.com is where the world meets DevOps. It is the largest collection of original content relating to DevOps on the web today Featuring up-to-the-minute news, feature stories, blogs, bylined articles and more, DevOps.com is where the thought leaders of the DevOps movement make their ideas known.
There are 182 billion emails sent every day, generating a lot of data about how recipients and ISPs respond. Many marketers take a more-is-better approach to stats, preferring to have the ability to slice and dice their email lists based numerous arbitrary stats. However, fundamentally what really matters is whether or not sending an email to a particular recipient will generate value. Data Scientists can design high-level insights such as engagement prediction models and content clusters that allow marketers to cut through the noise and design their campaigns around strong, predictive signals, rather than arbitrary statistics. SendGrid sends up to half a billion emails a day for customers such as Pinterest and GitHub. All this email adds up to more text than produced in the entire twitterverse. We track events like clicks, opens and deliveries to help improve deliverability for our customers – adding up to over 50 billion useful events every month. While SendGrid data covers only abo...
SYS-CON Events announced today that the Web Host Industry Review has been named “Media Sponsor” of SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Since 2000, The Web Host Industry Review has made a name for itself as the foremost authority of the Web hosting industry providing reliable, insightful and comprehensive news, reviews and resources to the hosting community. TheWHIR Blogs provides a community of expert industry perspectives. The Web Host Industry Review Magazine also offers a business-minded, issue-driven perspective of interest to executives and decision-makers. WHIR TV offers on demand web hosting video interviews and web hosting video features of the key persons and events of the web hosting industry. WHIR Events brings together like-minded hosting industry professionals and decision-makers in local communities. TheWHIR is an iNET Interactive property.
SYS-CON Events announced today that O'Reilly Media has been named “Media Sponsor” of SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. O'Reilly Media spreads the knowledge of innovators through its books, online services, magazines, and conferences. Since 1978, O'Reilly Media has been a chronicler and catalyst of cutting-edge development, homing in on the technology trends that really matter and spurring their adoption by amplifying "faint signals" from the alpha geeks who are creating the future. An active participant in the technology community, the company has a long history of advocacy, meme-making, and evangelism.
SYS-CON Events announced today that Verizon has been named “Gold Sponsor” of SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Verizon Enterprise Solutions creates global connections that generate growth, drive business innovation and move society forward. With industry-specific solutions and a full range of global wholesale offerings provided over the company's secure mobility, cloud, strategic networking and advanced communications platforms, Verizon Enterprise Solutions helps open new opportunities around the world for innovation, investment and business transformation. Visit verizonenterprise.com to learn more.
SYS-CON Events announced today that TMCnet has been named “Media Sponsor” of SYS-CON's 15th International Cloud Expo®, which will take place on November 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA. Technology Marketing Corporation (TMC) is the world's leading business to business and integrated marketing media company, servicing niche markets within the communications and technology industries.
"In my session I spoke about enterprise cloud analytics and how we can leverage analytics as a service," explained Ajay Budhraja, CTO at the Department of Justice, in this SYS-CON.tv interview at the 14th International Cloud Expo®, held June 10-12, 2014, at the Javits Center in New York City. Cloud Expo® 2014 Silicon Valley, November 4–6, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading Cloud industry players in the world.
“We are starting to see people move beyond the commodity cloud and enterprises need to start focusing on additional value added services in order to really drive their adoption," explained Jason Mondanaro, Director of Product Management at MetraTech, in this SYS-CON.tv interview at the 14th International Cloud Expo®, held June 10-12, 2014, at the Javits Center in New York City. Cloud Expo® 2014 Silicon Valley, November 4–6, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading Cloud industry players in the world.
"We are automated capacity control software, which basically looks at all the supply and demand and running a virtual cloud environment and does a deep analysis of that and says where should things go," explained Andrew Hillier, Co-founder & CTO of CiRBA, in this SYS-CON.tv interview at the 14th International Cloud Expo®, held June 10-12, 2014, at the Javits Center in New York City. Cloud Expo® 2014 Silicon Valley, November 4–6, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading Cloud industry players in the world.
Almost everyone sees the potential of Internet of Things but how can businesses truly unlock that potential. The key will be in the ability to discover business insight in the midst of an ocean of Big Data generated from billions of embedded devices via Systems of Discover. Businesses will also need to ensure that they can sustain that insight by leveraging the cloud for global reach, scale and elasticity. In his session at Internet of @ThingsExpo, Mac Devine, Distinguished Engineer at IBM, will discuss bringing these three elements together via Systems of Discover.
The Internet of Things promises to transform businesses (and lives), but navigating the business and technical path to success can be difficult to understand. In his session at 15th Internet of @ThingsExpo, Chad Jones, Vice President, Product Strategy of LogMeIn's Xively IoT Platform, will show you how to approach creating broadly successful connected customer solutions using real world business transformation studies including New England BioLabs and more.
All too many discussions about DevOps conclude that the solution is an all-purpose player: developer and operations guru, complete with pager for round-the-clock duty. For most organizations that is not the way forward. In his session at DevOps Summit, Bernard Golden, Vice President of Strategy at ActiveState, will discuss how to achieve the agility and speed of end-to-end automation without requiring an organization stocked with Supermen and Superwomen.