Welcome!

@CloudExpo Authors: Zakia Bouachraoui, Elizabeth White, Liz McMillan, Pat Romanski, Roger Strukhoff

Related Topics: @CloudExpo, Agile Computing

@CloudExpo: Article

Analytics for the Cloud

How Cloud and Big Data Trends Will Impact APM Analytics

“Big Data” is everywhere. What does it mean? Just as Cloud Computing burst onto the scene a few years ago, it depends on whom you ask.

Traditionally, in the Business Intelligence (BI) world, Big Data included analyzing historical business data from large data warehouse with the purpose of identifying long-term trends that could be leveraged in consumer business strategies. In recent years, Big Data has been a term talked about in the IT industry as an application of technology to attack extremely large, unstructured data sets that can reside both within and outside of an organization. If you look at a recent definition of Big Data, it is a term applied to data sets whose size has grown beyond the capability of commonly used software tools to capture, manage and analyze within a tolerable period of times for different use cases.

Application Performance Management (APM) is an extremely relevant use case and has a developing “Big Data” problem. Several factors are contributing to the explosive growth and type of data that must be analyzed and/or correlated in application performance monitoring and business service management (BSM).

First, the number of components that make up today’s mission critical applications has exploded. Instead of hundreds of servers for an application, nowadays, because of virtualization, you can easily be talking about thousands of virtual servers and objects for web applications.

Secondly, the diversity of data that people want to analyze to provide a holistic perspective has increased drastically. It is no longer good enough to simply understand traditional IT infrastructure performance based on server operating system, network traffic, and storage capacity. Application Performance analysis is now based on the relationships of IT infrastructure components, application performance metrics from applications and application servers, business activity monitors (BAM) data, customer experience monitors (CEM) and Real-User Monitoring (RUM). In addition to the aggregated transactional data, there are new systems that capture transactions’ actual path encompassing the entire application stack.

Finally, the requirements for analysis speed and data granularity have also increased significantly. Mission critical application performance now requires real-time or near real-time data analysis. When we were doing server availability and performance monitoring 10 years ago, it was the norm to collect and analyze data every 15 minutes. Today, this has evolved to data analysis every 5 minutes or less with sub-minute data collection where all transaction paths are collected for data analysis. When mapped out, it’s easy to see the enormous growth particularly when you look at APM related storage requirements that are quickly growing from gigabytes to terabytes and tomorrow petabytes.

All this data requires extremely complex analysis and correlation in order to truly understand performance of critical applications. One of Netuitive’s large enterprise customers reported that it monitors and correlates a billion infrastructure and application data points and business metrics daily as part of its global service delivery. This is what I am referring to as APM-generated Big Data. In addition to the shear number of data points, IT operators are expected to provide real-time analysis to the business and long-term storage for post-mortem analysis, capacity planning and compliance.

So where does this leads us? This is where APM and Big Data meet The Cloud. The cloud can deliver cheaper and more flexible storage and computing power crucial to analytics for Big Data. It also has the capability to be much more elastic for your APM data storage and analytics needs. Organizations can actually think about storing years of collected and aggregated APM data for compliance and analysis purposes without the cost being prohibitive.

But what does this mean to vendors in the APM space?

First of all, the analytics platform for APM data has to evolve to be able to process the growing number of different data sources across business, customer experience, applications and IT domains. Netuitive’s “Open” analytics platform is engineered to address virtually any data source in real-time.

Secondly, data storage and access time will be critical even as APM data volumes continue to explode, so not only does the technology need to be able to run in the cloud, but the traditional pull-based data collection architecture has to evolve into a push based model with an horizontally scalable computing and storage architecture in order to become virtually limitless in terms of scalability. This is critical for larger organizations as “real” time no longer means analysis every 5 to 15 minutes, but sub-minute analytics.

Lastly, because storage and computing costs should not significantly exceed the cost of analytics software for a solution to be viable, Netuitive is advancing its product architecture to leverage NoSQL columnar data store as a replacement to traditional database. Netuitive is also experimenting with a SaaS model for long-term time series data store and running its analytics software in the cloud.

While our R&D challenges are complex, the goal is simple: provide APM Analytics that matters by enabling our enterprise customers to process billions of infrastructure, application, and business metrics from hundreds of thousands of managed elements at 10x less cost than existing infrastructures.

I look forward to reporting on our progress. Check the blog for updates.

More Stories By Jean-François (JF) Huard

Jean-François (JF) Huard is Chief Technical Officer and Vice President of Research and Development at Netuitive, Inc. In this role he is responsible for leading the company’s vision and technology innovation effort. Previously, he worked in network control and management focusing on optimal flow control, decision-theoretic troubleshooting and game-theoretic bandwidth trading.

His current interest focus on IT analytics, APM analytics, cloud management and big data technologies.

Jean-François received his PhD (EE) from Columbia University, New York.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


CloudEXPO Stories
The precious oil is extracted from the seeds of prickly pear cactus plant. After taking out the seeds from the fruits, they are adequately dried and then cold pressed to obtain the oil. Indeed, the prickly seed oil is quite expensive. Well, that is understandable when you consider the fact that the seeds are really tiny and each seed contain only about 5% of oil in it at most, plus the seeds are usually handpicked from the fruits. This means it will take tons of these seeds to produce just one bottle of the oil for commercial purpose. But from its medical properties to its culinary importance, skin lightening, moisturizing, and protection abilities, down to its extraordinary hair care properties, prickly seed oil has got lots of excellent rewards for anyone who pays the price.
The platform combines the strengths of Singtel's extensive, intelligent network capabilities with Microsoft's cloud expertise to create a unique solution that sets new standards for IoT applications," said Mr Diomedes Kastanis, Head of IoT at Singtel. "Our solution provides speed, transparency and flexibility, paving the way for a more pervasive use of IoT to accelerate enterprises' digitalisation efforts. AI-powered intelligent connectivity over Microsoft Azure will be the fastest connected path for IoT innovators to scale globally, and the smartest path to cross-device synergy in an instrumented, connected world.
There are many examples of disruption in consumer space – Uber disrupting the cab industry, Airbnb disrupting the hospitality industry and so on; but have you wondered who is disrupting support and operations? AISERA helps make businesses and customers successful by offering consumer-like user experience for support and operations. We have built the world’s first AI-driven IT / HR / Cloud / Customer Support and Operations solution.
ScaleMP is presenting at CloudEXPO 2019, held June 24-26 in Santa Clara, and we’d love to see you there. At the conference, we’ll demonstrate how ScaleMP is solving one of the most vexing challenges for cloud — memory cost and limit of scale — and how our innovative vSMP MemoryONE solution provides affordable larger server memory for the private and public cloud. Please visit us at Booth No. 519 to connect with our experts and learn more about vSMP MemoryONE and how it is already serving some of the world’s largest data centers. Click here to schedule a meeting with our experts and executives.
Darktrace is the world's leading AI company for cyber security. Created by mathematicians from the University of Cambridge, Darktrace's Enterprise Immune System is the first non-consumer application of machine learning to work at scale, across all network types, from physical, virtualized, and cloud, through to IoT and industrial control systems. Installed as a self-configuring cyber defense platform, Darktrace continuously learns what is ‘normal' for all devices and users, updating its understanding as the environment changes.