@CloudExpo Authors: Yeshim Deniz, Zakia Bouachraoui, Elizabeth White, Liz McMillan, Pat Romanski

Blog Feed Post

The network probe is dead. Long live the probe!

Dynatrace blog

I came across a good discussion about complexity from one of the industry’s network probe vendors. Applying Metcalfe’s law, the blog reinforced what we’re all acutely aware of; the number of connections – not simply between servers, but between multiple processes on servers – inside today’s data centers is exploding. Here, I’ve crudely recreated a few diagrams often used to illustrate this.

https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-300x107.png 300w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-768x273.png 768w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-200x71.png 200w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-400x142.png 400w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-600x213.png 600w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-800x284.png 800w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-1000x355.png 1000w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-1200x426.png 1200w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-1400x497.png 1400w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-1-1600x568.png 1600w" sizes="(min-width: 900px) 900px, 100vw" />

This rapidly increasing complexity drove corresponding data center network architectural shifts, from hierarchical layouts (where north-south traffic dominated) to flattened and micro-segmented leaf/spine layouts that can more effectively support this server-to-server or east-west traffic.

https://dt-cdn.net/wp-content/uploads/2018/01/Image-2-300x115.png 300w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-2-768x293.png 768w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-2-200x76.png 200w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-2-400x153.png 400w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-2-600x229.png 600w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-2-800x306.png 800w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-2-1000x382.png 1000w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-2-1200x459.png 1200w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-2-1400x535.png 1400w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-2.png 1429w" sizes="(min-width: 900px) 900px, 100vw" />

It’s no coincidence that the flattened network architecture shown here resembles the most complex of the Metcalfe diagrams I drew; direct switched network connections between communicating hosts are inherently more efficient than multi-hop hierarchies.

At the same time, increases in virtualization density frequently introduced scenarios where inter-VM traffic never had to hit the (physical) network.

Existential angst

Slowly but surely, these shifts resulted in the loss of traditional traffic aggregation points that had become an enabling foundation of a network probe’s value. They presented an existential problem, and probe vendors found themselves in a battle for territorial survival inside the modern data center.

Unspoken panic can be a great motivator, and vendors entered the fray from multiple flanks; after all, if all you have is a probe, then every problem must be visible on the network. Early skirmishes saw sometimes convoluted network configurations intended to route communications on to physical links or up to artificial aggregation points. Some took the approach of installing virtual probes on each host. More recent and sophisticated battles look to deploy virtual network port mirroring and virtual taps. Even the unbiased treaty negotiators – analysts and services companies – have touted fundamental revisions to data center architectures to include “visibility planes” through distributed network packet brokers (NPBs).

Battles continue to rage on in the trenches. But even through the thick fog of war, the outcome is taking clear shape. Data center architectures are designed to provide agile application services. Providing access to network packets takes a much lower priority, to some degree in anticipation of alternate monitoring approaches.

Tough questions

How many probes are you willing to deploy in your data center? How many taps – real and virtual – would you need for full Metcalfe-like visibility? What kind of supporting network will you need to route this traffic to your probes? How will these stopgap solutions respond to the ever-increasing dynamics of data center traffic, of connections, of the services themselves?  And the most important question: to what end? To monitor application performance?

It seems reasonable, then, to conclude that network probes are not going to maintain their value for intra-data center monitoring. Take the current trajectory to its logical conclusion, where every host, every VM, every container sits on its own segment with a direct virtual path to dependent peers. Should every node have its own virtual probe? And if not – how will you measure the quality of node-to-node communication in the virtualized environment?

https://dt-cdn.net/wp-content/uploads/2018/01/Image-3-200x168.png 200w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-3-400x336.png 400w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-3-600x504.png 600w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-3.png 650w" sizes="(min-width: 900px) 900px, 100vw" />

Straight answers

Now replace the words “virtual probe” in the above paragraph with “software agent.” Suddenly, the problem of data collection isn’t so daunting; the agents do all the work, offering access not only to host network interface statistics, but also to process-level network communications – including access to network packets themselves– along with compelling host and app performance data. The challenge shifts quickly to the high-value opportunity of data analysis. And that’s where automation, full-stack visibility and AI come to play.

https://dt-cdn.net/wp-content/uploads/2018/01/Image-4-200x190.png 200w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-4-400x381.png 400w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-4-600x571.png 600w, https://dt-cdn.net/wp-content/uploads/2018/01/Image-4.png 750w" sizes="(min-width: 900px) 900px, 100vw" />

Is the network probe dead?

Not by a long shot.

If we consider the data center probe in terms of its traditional form-factor, it’s clear it doesn’t fit well in today’s dynamic and cloud-like data center architectures. But the value of network analysis endures; in fact, some argue this may gain importance as containerization, micro-segmentation, and dynamic provisioning gain firmer footing, relying more heavily on Metcalfe-like network meshes. Inside the modern data center, these insights will be derived from software agents rather than physical – or even virtual – network probes.

In fact, attempting to fulfill an outsized APM-like role may have contributed to the probe’s struggle for relevance. As Digital Performance Management (DPM) and in-depth transaction tracing become the exclusive realm of agents and APIs, there exists a natural opportunity to use these same agents to also deliver fundamental network insights. This leaves NPM – along with user visibility – for the probe, leading to this rhetorical question: Is probe-sourced wire data really the best way to understand the performance of a data center’s core network? Or can agents do the job?

Of course packet-based TCP flow analysis will also remain important, offering the most objective source of network performance insights; these insights, however, have always been more valuable on the WAN. And that is where we’ll see traditional network probes settle – rather comfortably, actually – into the role they were designed to play, shifting to the place they have always belonged: the edge of the data center, where they can continue to deliver clear value.

Monitoring at the edge of the data center

What makes the data center edge so well-suited for a network probe?

  • First and foremost, the WAN – traditional, hybrid, optimized, software-defined – is where network characteristics and TCP flow control behaviors are most likely to have an impact on application performance and availability.
  • Second, today’s WANs incorporate many devices and appliances that influence performance and availability through factors that go beyond such traditional NPM micmetrics as bandwidth, latency, loss, and routing. These appliances include WAN optimization controllers (WOCs), application delivery controllers (ADCs), load balancers, firewalls, even thin client solutions. They not only control traffic flows through TCP manipulation, they also often perform server-like functions, independently delivering some application content directly to users. Yet they’re still considered part of the network.
  • Third, the number of WAN access points to your data center – and therefore the number of probe points – is relatively small, avoiding the Metcalfe matrix problem.
  • Fourth, these access points often already have wire data access solutions in place for IDS, important to security teams; these network packet broker (NPB) solutions can easily prune and share raw packet streams with a network monitoring probe.
  • Last – but not least – end-user experience is arguably the most important metric by which to measure service delivery quality. The probe’s vantage point at the data center edge provides the best perspective to deliver this value, as it can see all user interactions with your data center apps.

Equipping the network probe with the intelligence to understand application-specific transactions and automatically analyze performance degradation makes it an invaluable triage point for BizDevOps teams responsible for application delivery and performance. User experience remains the common actionable metric important to all three groups.

It’s worth noting that, just a few short years ago, some pundits were announcing the death of the network probe; the inability to receive packets in PaaS and IaaS clouds presumably foretold a rapid demise. But the speed at which this problem has been solved by the leading NPB vendors (such as Ixia and Gigamon) proves the resilience and value of the probe at the data center’s edge. We’ve started to see this trend, working with our customers to achieve monitoring continuity as they move their complete application infrastructures – including their DC RUM network probes – from on-premises to IaaS clouds.

The post The network probe is dead. Long live the probe! appeared first on Dynatrace blog.

Read the original blog entry...

More Stories By APM Blog

APM: It’s all about application performance, scalability, and architecture: best practices, lifecycle and DevOps, mobile and web, enterprise, user experience

CloudEXPO Stories
In today's always-on world, customer expectations have changed. Competitive differentiation is delivered through rapid software innovations, the ability to respond to issues quickly and by releasing high-quality code with minimal interruptions. DevOps isn't some far off goal; it's methodologies and practices are a response to this demand. The demand to go faster. The demand for more uptime. The demand to innovate. In this keynote, we will cover the Nutanix Developer Stack. Built from the foundation of software-defined infrastructure, Nutanix has rapidly expanded into full application lifecycle management across any infrastructure or cloud .Join us as we delve into how the Nutanix Developer Stack makes it easy to build hybrid cloud applications by weaving DBaaS, micro segmentation, event driven lifecycle operations, and both financial and cloud governance together into a single unified st...
Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes. We are offering early bird savings on all ticket types where you can save significant amount of money by purchasing your conference tickets today.
Organize your corporate travel faster, at lower cost. Hotailors is a next-gen AI-powered travel platform. What is Hotailors? Hotailors is a platform for organising business travels that grants access to the best real-time offers from 2.000.000+ hotels and 700+ airlines in the whole world. Thanks to our solution you can plan, book & expense business trips in less than 5 minutes. Accordingly to your travel policy, budget limits and cashless for your employees. With our reporting, integrations and real-time analytics, you can easily control your travel policy, reduce spendings and increase the efficiency of your company. Hotailors ia an artificial intelligence powered marketplace for business travels. Friendly to use technology based on artificial intelligence to plan, book and expense business travel in 5 minutes. Within one login on one dashboard, user has access to almos...
DXWorldEXPO LLC announced today that the upcoming DXWorldEXPO | DevOpsSUMMIT | CloudEXPO New York will feature 10 companies from Poland to participate at the "Poland Digital Transformation Pavilion" on November 12-13, 2018. Polish Digital Transformation companies which will exhibit at CloudEXPO | DevOpsSUMMIT | DXWorldEXPO include All in Mobile, dhosting, Cryptomage, Perfect Gym, Polcom, Apius Technologies, Aplisens, ELZAB SA, TELDAT, and Rebug.io.
Whenever a new technology hits the high points of hype, everyone starts talking about it like it will solve all their business problems. Blockchain is one of those technologies. According to Gartner's latest report on the hype cycle of emerging technologies, blockchain has just passed the peak of their hype cycle curve. If you read the news articles about it, one would think it has taken over the technology world. No disruptive technology is without its challenges and potential impediments that frequently get lost in the hype. The panel will discuss their perspective on what they see as they key challenges and/or impediments to adoption, and how they see those issues could be resolved or mitigated.