Welcome!

Cloud Expo Authors: Carmen Gonzalez, Elizabeth White, Michael Jannery, Tim Hinds, Ian Khan

Related Topics: Cloud Expo, Java, SOA & WOA, Linux, Virtualization, Web 2.0, Big Data Journal

Cloud Expo: Article

Understanding Application Performance on the Network | Part 4

Packet Loss

We know that losing packets is not a good thing; retransmissions cause delays. We also know that TCP ensures reliable data delivery, masking the impact of packet loss. So why are some applications seemingly unaffected by the same packet loss rate that seems to cripple others? From a performance analysis perspective, how do you understand the relevance of packet loss and avoid chasing red herrings?

In Part II, we examined two closely related constraints - bandwidth and congestion. In Part III, we discussed TCP slow-start and introduced the Congestion Window (CWD). In Part IV, we'll focus on packet loss, continuing the concepts from these two previous entries.

TCP Reliability
TCP ensures reliable delivery of data through its sliding window approach to managing byte sequences and acknowledgements; among other things, this sequencing allows a receiver to inform the sender of missing data caused by packet loss in multi-packet flows. Independently, a sender may detect packet loss through the expiration of its retransmission timer. We will look at the behavior and performance penalty associated with each of these cases; generally, the impact of packet loss will depend on both the characteristics of the flow and the position of the dropped packet within the flow.

The Retransmission Timer
Each packet a node sends is associated with a retransmission timer; if the timer expires before the sent data has been acknowledged, it is considered lost and retransmitted. There are two important characteristics of the retransmission timer that relate to performance. First, the default value for the initial retransmission timeout (RTO) is almost always 3000 milliseconds; this is adjusted to a more reasonable value as TCP observes actual path round-trip times. Second, the timeout value is doubled for subsequent retransmissions of a packet.

In small flows (a common characteristic of chatty operations - like web pages), the retransmission timer is the method used to detect packet loss. Consider a request or reply message of just 1000 bytes, sent in a single packet; if this packet is dropped, there will of course be no acknowledgement; the receiver has no idea the packet was sent. If the packet is dropped early in the life of a TCP connection - perhaps one of the SYN packets during the TCP 3-way handshake, or an initial GET request or a 304 Not Modified response - the dropped packet will be retransmitted only after 3 seconds have elapsed.

Triple Duplicate ACK
Within larger flows, a dropped packet may be detected before the retransmission time expires if the sender receives three duplicate ACKs; this is generally more efficient (faster) than waiting for the retransmission timer to expire. As the receiving node receives packets that are out of sequence (i.e., after the missing packet data should have been seen), it sends duplicate ACKs, the acknowledgement number repeatedly referencing the expected (missing) packet data. When the sending node receives the third duplicate ACK, it assumes the packet was in fact lost (not just delayed) and retransmits it. This event causes the sender to assume network congestion, reducing its congestion window by 50% to allow congestion to subside. Slow-start begins to increase the CWD from that new value, using a relatively conservative congestion avoidance ramp.

As an example, consider a server sending a large file to a client; the sending node is ramping up through slow-start. As the CWD reaches 24, earlier packet loss is detected via a triple duplicate ACK; the lost data is retransmitted, and the CWD is reduced to 12. Slow-start resumes from this point in its congestion avoidance mode.

While arguments abound about the inefficiency of existing congestion avoidance approaches, especially on high-speed networks, you can expect to see this behavior in today's networks.

Transaction Trace Illustration
Identifying retransmission timeouts using merged trace files is generally quite straightforward; we have proof the packet has been lost (because we see it on the sending side and not on the receiving side), and we know the delay between the dropped and retransmitted packets at the sending node. The Delta column in the Error Table indicates the retransmission delay.

Error Table entry showing a 3-second retransmission delay caused by a retransmission timeout (RTO)

For larger flows, you can illustrate the effect of dropped packets on the sender's Congestion Window by using the Time Plot view. For Series 1, graph the sender's Frames in Transit; this is essentially the CWD. For Series 2, graph the Cumulative Error Count in both directions. As errors (retransmitted packets or out-of-sequence packets) occur, the CWD will be reduced by about 50%.

Time Plot view showing the impact of packet loss (blue plot) on the Congestion Window (brown plot)

For more networking tips click here for the full article

More Stories By Gary Kaiser

Gary Kaiser is a Subject Matter Expert in Network Performance Analysis at Compuware APM. He has global field enablement responsibilities for performance monitoring and analysis solutions embracing emerging and strategic technologies, including WAN optimization, thin client infrastructures, network forensics, and a unique performance management maturity methodology. He is also a co-inventor of multiple analysis features, and continues to champion the value of software-enabled expert network analysis.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
Want to enable self-service provisioning of application environments in minutes that mirror production? Can you automatically provide rich data with code-level detail back to the developers when issues occur in production? In his session at DevOps Summit, David Tesar, Microsoft Technical Evangelist on Microsoft Azure and DevOps, will discuss how to accomplish this and more utilizing technologies such as Microsoft Azure, Visual Studio online, and Application Insights in this demo-heavy session.
Entuity®, a provider of enterprise-class network management solutions, today announced that it solidifies its position as a market leader through global enterprise customer acquisitions and a refined channel strategy. In 2014, Entuity increased new license revenues in EMEA by over 75 percent, and LATAM by over 125 percent as customers embraced Entuity for its highly automated solution and unified architecture. Entuity’s refined channel strategy focuses on even deeper strategic alignment with ke...
We are all here because we are sold on the transformative promise of The Cloud. But what good is all of this ephemeral, on-demand infrastructure if your usage doesn't actually improve the agility and speed of your business? How must Operations adapt in order to avoid stifling your Cloud initiative? In his session at DevOps Summit, Damon Edwards, co-founder and managing partner of the DTO Solutions, will highlight the successful organizational, process, and tooling patterns of high-performing c...
The 4th International DevOps Summit, co-located with16th International Cloud Expo – being held June 9-11, 2015, at the Javits Center in New York City, NY – announces that its Call for Papers is now open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's large...
Cloud Expo 2014 TV commercials will feature @ThingsExpo, which was launched in June, 2014 at New York City's Javits Center as the largest 'Internet of Things' event in the world.
“We help people build clusters, in the classical sense of the cluster. We help people put a full stack on top of every single one of those machines. We do the full bare metal install," explained Greg Bruno, Vice President of Engineering and co-founder of StackIQ, in this SYS-CON.tv interview at 15th Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
"People are a lot more knowledgeable about APIs now. There are two types of people who work with APIs - IT people who want to use APIs for something internal and the product managers who want to do something outside APIs for people to connect to them," explained Roberto Medrano, Executive Vice President at SOA Software, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
In this demo at 15th Cloud Expo, John Meza, Product Engineer at Esri, showed how Esri products hook into Hadoop cluster to allow you to do spatial analysis on the spatial data within your cluster, and he demonstrated rendering from a data center with ArcGIS Pro, a new product that has a brand new rendering engine.
"Blue Box has been around for 10-11 years, and last year we launched Blue Box Cloud. We like the term 'Private Cloud as a Service' because we think that embodies what we are launching as a product - it's a managed hosted private cloud," explained Giles Frith, Vice President of Customer Operations at Blue Box, in this SYS-CON.tv interview at DevOps Summit, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
The Software Defined Data Center (SDDC), which enables organizations to seamlessly run in a hybrid cloud model (public + private cloud), is here to stay. IDC estimates that the software-defined networking market will be valued at $3.7 billion by 2016. Security is a key component and benefit of the SDDC, and offers an opportunity to build security 'from the ground up' and weave it into the environment from day one. In his session at 16th Cloud Expo, Reuven Harrison, CTO and Co-Founder of Tufin,...
CloudBees, Inc., has announced a $23.5 million financing round, led by longtime CloudBees investor Lightspeed Venture Partners. Existing investors Matrix Partners, Verizon Ventures and Blue Cloud Ventures also participated in the round. The latest funding announcement follows earlier rounds of $4 million, $10.5 million and $10.8 million, bringing the total investment in CloudBees to just under $50 million since the company’s inception in 2010. Previous venture investment rounds were led by Ma...
“We are strong believers in the DevOps movement and our staff has been doing DevOps for large enterprise environments for a number of years. The solution that we build is intended to allow DevOps teams to do security at the speed of DevOps," explained Justin Lundy, Founder & CTO of Evident.io, in this SYS-CON.tv interview at DevOps Summit, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Media announced that Splunk, a provider of the leading software platform for real-time Operational Intelligence, has launched an ad campaign on Big Data Journal. Splunk software and cloud services enable organizations to search, monitor, analyze and visualize machine-generated big data coming from websites, applications, servers, networks, sensors and mobile devices. The ads focus on delivering ROI - how improved uptime delivered $6M in annual ROI, improving customer operations by minin...
The move in recent years to cloud computing services and architectures has added significant pace to the application development and deployment environment. When enterprise IT can spin up large computing instances in just minutes, developers can also design and deploy in small time frames that were unimaginable a few years ago. The consequent move toward lean, agile, and fast development leads to the need for the development and operations sides to work very closely together. Thus, DevOps become...
Puppet Labs on Wednesday released the DevOps Salary Report, based on salary data gathered from Puppet Labs' industry-recognized State of DevOps Report. The data confirms that market demand for DevOps skills is growing, and that DevOps engineers are among the highest paid IT practitioners today. That's because IT organizations today are grappling with how to be more agile and responsive to the business, while maintaining the stability of their infrastructure. DevOps practices, such as continuous ...
The cloud is becoming the de-facto way for enterprises to leverage common infrastructure while innovating and one of the biggest obstacles facing public cloud computing is security. In his session at 15th Cloud Expo, Jeff Aliber, a global marketing executive at Verizon, discussed how the best place for web security is in the cloud. Benefits include: Functions as the first layer of defense Easy operation –CNAME change Implement an integrated solution Best architecture for addressing network-l...
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete...
Appcore deploys cloud for service providers based on the Apache Cloud set. In this demo at 15th Cloud Expo, Nate Gordon, Director of Technology at Appcore, shows their new product that's coming out in January - Appcore Atlas, which is focused on deploying private clouds based on CloudStack in 15 minutes or less. Our upcoming June 9-11, 2015, event in New York City will present a total of 10 simultaneous tracks (the largest conference content in the world) by an all-star faculty, over three days...
In this scenarios approach Joe Thykattil, Technology Architect & Sales at TimeWarner / Navisite, presented examples that will allow business-savvy professionals to make informed decisions based on a sound business model. This model covered the technology options in detail as well as a financial analysis. The TCO (Total Cost of Ownership) and ROI (Return on Investment) demonstrated how to start, develop and formulate a business case that will allow both small and large scale projects to achieve...