|By AppDynamics Blog||
|April 13, 2012 05:00 AM EDT||
As data has grown exponentially at many sites, companies have been forced to horizontally scale their data. Some have turned to sharding of databases like Postrres or MySQL , while others have switched to newer NoSQL data systems. There have been many debates in the last few years about SQL vs. NoSQL data management systems and which is better. What many have failed to grasp, though, is how similar these systems are and how complex they both are to run in production in high scale.
Both of these systems represent what I call a Data Cloud. This Data Cloud is logical data set spread across many nodes. While developers have heated debates about which system is better and how to design code around it, those in DevOps usually struggle with very similar issues because the two systems are mostly the same. Both systems
- Run across many nodes with large amounts of data flowing between them and from/to the application
- Strain both the hardware of all nodes, and the network connecting them
- Maintain duplicate data across nodes for fault tolerance, and must have failover ability
- Must be tuned on a per node and cluster-wide bases
- Must allow for growth by adding additional nodes.
Running this Data Cloud in production presents a new set of challenges for DevOps, many of which are not well understood or addressed. One of the main challenges is the management and monitoring of these systems, for which few (if any) tools or products exist at this time.
When systems were smaller and you ran a single Database in production, you probably had all the necessary systems in place. With a plethora of products for Management, monitoring, visualizing data, and backups, it was not hard to be successful and meet your SLAs.
But now all this is much more complex once you move into the world of the Data Cloud. Now you have a large number of nodes, all representing the same system and still needing to meet the same SLAs as the old simple DB from before. Let us look at the challenges for running a production Data Cloud successfully.
Do you know how many nodes you need? How many nodes do you put in each replica set? How much latency and throughput do you need in your network for the nodes to communicate fast enough? What is the ideal hardware to use for each node to balance performance with costs?
How do you monitor dozens, hundreds or even thousands of nodes all at once? How do you get a unified view of your data cloud, and then drill down to the problem nodes? Are there even any off-the-shelf monitoring tools that can help? Your old monitoring tool won’t be very useful anymore unless you are willing to look at every node one by one to see what is going on there.
How do you set up a common set of alerts across all nodes? And how do you keep your alert thresholds in sync as you add nodes and remove them? More importantly, even assuming you have alerting in place, once staff receives critical alerts, how will they know where to find the troubled node in the massive cloud, or whether it’s a node level issue or more global in nature? This must be done quickly during critical outages.
How does your staff view the data when it is distributed? In case of data inaccuracy, how can they quickly identify the faulty nodes and fix up the data?
As performance degrades, how do you troubleshoot and identify the bottlenecks? How do you find which nodes by be the cause of the problem? How do you improve performance across all the nodes.
Data Cloud Management
How do you back up all the data while consistently tracking which nodes were backed up successfully and when? How do you make schema changes across all the nodes in one consistent step without breaking your app? And how do you make configuration changes on various nodes or across all nodes? And how do you track the configurations of each node and keep them consistent across your system?
By now you should see that there is a lot to think about before endeavoring to launch a production Data Cloud. Too many companies focus all their energies on deciding which DB or NoSQL system to use and developing their apps for it. But that might turn out to be the lesser of your challenges once you struggle to put the system into production. Be sure you can answer all the questions I have listed above before your launch.
“DevOps is really about the business. The business is under pressure today, competitively in the marketplace to respond to the expectations of the customer. The business is driving IT and the problem is that IT isn't responding fast enough," explained Mark Levy, Senior Product Marketing Manager at Serena Software, in this SYS-CON.tv interview at DevOps Summit, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Feb. 1, 2015 07:00 PM EST Reads: 2,740
Entuity®, a provider of enterprise-class network management solutions, today announced that it solidifies its position as a market leader through global enterprise customer acquisitions and a refined channel strategy. In 2014, Entuity increased new license revenues in EMEA by over 75 percent, and LATAM by over 125 percent as customers embraced Entuity for its highly automated solution and unified architecture. Entuity’s refined channel strategy focuses on even deeper strategic alignment with ke...
Feb. 1, 2015 05:00 PM EST Reads: 873
The 3rd International Internet of @ThingsExpo, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that its Call for Papers is now open. The Internet of Things (IoT) is the biggest idea since the creation of the Worldwide Web more than 20 years ago.
Feb. 1, 2015 04:15 PM EST Reads: 3,302
We are all here because we are sold on the transformative promise of The Cloud. But what good is all of this ephemeral, on-demand infrastructure if your usage doesn't actually improve the agility and speed of your business? How must Operations adapt in order to avoid stifling your Cloud initiative? In his session at DevOps Summit, Damon Edwards, co-founder and managing partner of the DTO Solutions, will highlight the successful organizational, process, and tooling patterns of high-performing c...
Feb. 1, 2015 04:15 PM EST Reads: 3,083
Technology is enabling a new approach to collecting and using data. This approach, commonly referred to as the "Internet of Things" (IoT), enables businesses to use real-time data from all sorts of things including machines, devices and sensors to make better decisions, improve customer service, and lower the risk in the creation of new revenue opportunities. In his General Session at Internet of @ThingsExpo, Dave Wagstaff, Vice President and Chief Architect at BSQUARE Corporation, discuss the ...
Feb. 1, 2015 03:45 PM EST Reads: 3,206
“We help people build clusters, in the classical sense of the cluster. We help people put a full stack on top of every single one of those machines. We do the full bare metal install," explained Greg Bruno, Vice President of Engineering and co-founder of StackIQ, in this SYS-CON.tv interview at 15th Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Feb. 1, 2015 02:45 PM EST Reads: 2,657
At 15th Cloud Expo, Shrikant Pattathil, Executive Vice President at Harbinger Systems, demos a video delivery platform that helps you do interactive videos. He discusses how Harbinger is accomplishing it in the cloud world, the problems they faced and the choices they made to get around these problems.
Feb. 1, 2015 02:45 PM EST Reads: 1,786
"People are a lot more knowledgeable about APIs now. There are two types of people who work with APIs - IT people who want to use APIs for something internal and the product managers who want to do something outside APIs for people to connect to them," explained Roberto Medrano, Executive Vice President at SOA Software, in this SYS-CON.tv interview at Cloud Expo, held Nov 4–6, 2014, at the Santa Clara Convention Center in Santa Clara, CA.
Feb. 1, 2015 02:30 PM EST Reads: 2,811
CloudBees, Inc., has announced a $23.5 million financing round, led by longtime CloudBees investor Lightspeed Venture Partners. Existing investors Matrix Partners, Verizon Ventures and Blue Cloud Ventures also participated in the round. The latest funding announcement follows earlier rounds of $4 million, $10.5 million and $10.8 million, bringing the total investment in CloudBees to just under $50 million since the company’s inception in 2010. Previous venture investment rounds were led by Ma...
Feb. 1, 2015 02:00 PM EST Reads: 1,099
In this Women in Technology Power Panel at 15th Cloud Expo, moderated by Anne Plese, Senior Consultant, Cloud Product Marketing at Verizon Enterprise, Esmeralda Swartz, CMO at MetraTech; Evelyn de Souza, Data Privacy and Compliance Strategy Leader at Cisco Systems; Seema Jethani, Director of Product Management at Basho Technologies; Victoria Livschitz, CEO of Qubell Inc.; Anne Hungate, Senior Director of Software Quality at DIRECTV, discussed what path they took to find their spot within the tec...
Feb. 1, 2015 01:45 PM EST Reads: 2,492
The cloud is becoming the de-facto way for enterprises to leverage common infrastructure while innovating and one of the biggest obstacles facing public cloud computing is security. In his session at 15th Cloud Expo, Jeff Aliber, a global marketing executive at Verizon, discussed how the best place for web security is in the cloud. Benefits include: Functions as the first layer of defense Easy operation –CNAME change Implement an integrated solution Best architecture for addressing network-l...
Feb. 1, 2015 01:30 PM EST Reads: 2,396
DevOps Summit 2015 New York, co-located with the 16th International Cloud Expo - to be held June 9-11, 2015, at the Javits Center in New York City, NY - announces that it is now accepting Keynote Proposals. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to wait for long development cycles that produce software that is obsolete...
Feb. 1, 2015 01:15 PM EST Reads: 2,814
Cloud computing started a technology revolution; now DevOps is driving that revolution forward. By enabling new approaches to service delivery, cloud and DevOps together are delivering even greater speed, agility, and efficiency. No wonder leading innovators are adopting DevOps and cloud together! In his session at DevOps Summit, Andi Mann, Vice President of Strategic Solutions at CA Technologies, explored the synergies in these two approaches, with practical tips, techniques, research data, wa...
Feb. 1, 2015 01:00 PM EST Reads: 3,321
Software AG and Wipro Ltd. have announced a joint solution platform for streaming analytics that provides real-time actionable intelligence for the Internet of Things (IoT) market. “The key to successfully addressing the IoT market is the ability to rapidly build and evolve apps that tap into, analyze and make smart decisions on fast, big data”, said John Bates, Global Head of Industry Solutions and CMO, Software AG. To address the huge market potential created by streaming analytics in conj...
Feb. 1, 2015 01:00 PM EST Reads: 1,233
Appcore deploys cloud for service providers based on the Apache Cloud set. In this demo at 15th Cloud Expo, Nate Gordon, Director of Technology at Appcore, shows their new product that's coming out in January - Appcore Atlas, which is focused on deploying private clouds based on CloudStack in 15 minutes or less. Our upcoming June 9-11, 2015, event in New York City will present a total of 10 simultaneous tracks (the largest conference content in the world) by an all-star faculty, over three days...
Feb. 1, 2015 01:00 PM EST Reads: 1,649
Amazon, Google and Facebook are household names in part because of their mastery of Big Data. But what about organizations without billions of dollars to spend on Big Data tools - how can they extract value from their data? In his session at 6th Big Data Expo®, Ali Ghodsi, Co-Founder and Head of Engineering at Databricks, discussed how the zero management cost and scalability of the cloud is addressing the challenges and pain points that data engineers face when working with Big Data. He also s...
Feb. 1, 2015 01:00 PM EST Reads: 2,601
The Industrial Internet revolution is now underway, enabled by connected machines and billions of devices that communicate and collaborate. The massive amounts of Big Data requiring real-time analysis is flooding legacy IT systems and giving way to cloud environments that can handle the unpredictable workloads. Yet many barriers remain until we can fully realize the opportunities and benefits from the convergence of machines and devices with Big Data and the cloud, including interoperability, ...
Feb. 1, 2015 01:00 PM EST Reads: 3,055
IBM has announced a new strategic technology services agreement with Anthem, Inc., a health benefits company in the U.S. IBM has been selected to provide operational services for Anthem's mainframe and data center server and storage infrastructure for the next five years. Among the benefits of the relationship, Anthem has the ability to leverage IBM Cloud solutions that will help increase the ease, availability and speed of adding infrastructure to support new business requirements.
Feb. 1, 2015 01:00 PM EST Reads: 1,630
Today, IT is not just a cost center. IT is an enabler and driver of business. With the emergence of the hybrid cloud paradigm, IT now has increasingly more capabilities to create new strategic opportunities for a business. Hybrid cloud allows an organization to utilize multi-tenant public clouds, dedicated private clouds, bare metal hosting, and the associated support and services for the right use cases through an on-demand, XaaS model. This model of IT creates tremendous opportunities for busi...
Feb. 1, 2015 01:00 PM EST Reads: 1,718
The term culture has had a polarizing effect among DevOps supporters. Some propose that culture change is critical for success with DevOps, but are remiss to define culture. Some talk about a DevOps culture but then reference activities that could lead to culture change and there are those that talk about culture change as a set of behaviors that need to be adopted by those in IT. There is no question that businesses successful in adopting a DevOps mindset have seen departmental culture change, ...
Feb. 1, 2015 12:45 PM EST Reads: 2,707