|By AppDynamics Blog||
|April 13, 2012 05:00 AM EDT||
As data has grown exponentially at many sites, companies have been forced to horizontally scale their data. Some have turned to sharding of databases like Postrres or MySQL , while others have switched to newer NoSQL data systems. There have been many debates in the last few years about SQL vs. NoSQL data management systems and which is better. What many have failed to grasp, though, is how similar these systems are and how complex they both are to run in production in high scale.
Both of these systems represent what I call a Data Cloud. This Data Cloud is logical data set spread across many nodes. While developers have heated debates about which system is better and how to design code around it, those in DevOps usually struggle with very similar issues because the two systems are mostly the same. Both systems
- Run across many nodes with large amounts of data flowing between them and from/to the application
- Strain both the hardware of all nodes, and the network connecting them
- Maintain duplicate data across nodes for fault tolerance, and must have failover ability
- Must be tuned on a per node and cluster-wide bases
- Must allow for growth by adding additional nodes.
Running this Data Cloud in production presents a new set of challenges for DevOps, many of which are not well understood or addressed. One of the main challenges is the management and monitoring of these systems, for which few (if any) tools or products exist at this time.
When systems were smaller and you ran a single Database in production, you probably had all the necessary systems in place. With a plethora of products for Management, monitoring, visualizing data, and backups, it was not hard to be successful and meet your SLAs.
But now all this is much more complex once you move into the world of the Data Cloud. Now you have a large number of nodes, all representing the same system and still needing to meet the same SLAs as the old simple DB from before. Let us look at the challenges for running a production Data Cloud successfully.
Do you know how many nodes you need? How many nodes do you put in each replica set? How much latency and throughput do you need in your network for the nodes to communicate fast enough? What is the ideal hardware to use for each node to balance performance with costs?
How do you monitor dozens, hundreds or even thousands of nodes all at once? How do you get a unified view of your data cloud, and then drill down to the problem nodes? Are there even any off-the-shelf monitoring tools that can help? Your old monitoring tool won’t be very useful anymore unless you are willing to look at every node one by one to see what is going on there.
How do you set up a common set of alerts across all nodes? And how do you keep your alert thresholds in sync as you add nodes and remove them? More importantly, even assuming you have alerting in place, once staff receives critical alerts, how will they know where to find the troubled node in the massive cloud, or whether it’s a node level issue or more global in nature? This must be done quickly during critical outages.
How does your staff view the data when it is distributed? In case of data inaccuracy, how can they quickly identify the faulty nodes and fix up the data?
As performance degrades, how do you troubleshoot and identify the bottlenecks? How do you find which nodes by be the cause of the problem? How do you improve performance across all the nodes.
Data Cloud Management
How do you back up all the data while consistently tracking which nodes were backed up successfully and when? How do you make schema changes across all the nodes in one consistent step without breaking your app? And how do you make configuration changes on various nodes or across all nodes? And how do you track the configurations of each node and keep them consistent across your system?
By now you should see that there is a lot to think about before endeavoring to launch a production Data Cloud. Too many companies focus all their energies on deciding which DB or NoSQL system to use and developing their apps for it. But that might turn out to be the lesser of your challenges once you struggle to put the system into production. Be sure you can answer all the questions I have listed above before your launch.
Cognitive Computing is becoming the foundation for a new generation of solutions that have the potential to transform business. Unlike traditional approaches to building solutions, a cognitive computing approach allows the data to help determine the way applications are designed. This contrasts with conventional software development that begins with defining logic based on the current way a business operates. In her session at 18th Cloud Expo, Judith S. Hurwitz, President and CEO of Hurwitz & ...
Feb. 9, 2016 10:15 PM EST Reads: 225
Father business cycles and digital consumers are forcing enterprises to respond faster to customer needs and competitive demands. Successful integration of DevOps and Agile development will be key for business success in today’s digital economy. In his session at DevOps Summit, Pradeep Prabhu, Co-Founder & CEO of Cloudmunch, covered the critical practices that enterprises should consider to seamlessly integrate Agile and DevOps processes, barriers to implementing this in the enterprise, and pr...
Feb. 9, 2016 06:00 PM EST Reads: 366
It's easy to assume that your app will run on a fast and reliable network. The reality for your app's users, though, is often a slow, unreliable network with spotty coverage. What happens when the network doesn't work, or when the device is in airplane mode? You get unhappy, frustrated users. An offline-first app is an app that works, without error, when there is no network connection.
Feb. 9, 2016 06:00 PM EST Reads: 187
With an estimated 50 billion devices connected to the Internet by 2020, several industries will begin to expand their capabilities for retaining end point data at the edge to better utilize the range of data types and sheer volume of M2M data generated by the Internet of Things. In his session at @ThingsExpo, Don DeLoach, CEO and President of Infobright, will discuss the infrastructures businesses will need to implement to handle this explosion of data by providing specific use cases for filte...
Feb. 9, 2016 05:00 PM EST Reads: 153
SYS-CON Events announced today that VAI, a leading ERP software provider, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. VAI (Vormittag Associates, Inc.) is a leading independent mid-market ERP software developer renowned for its flexible solutions and ability to automate critical business functions for the distribution, manufacturing, specialty retail and service sectors. An IBM Premier Business Part...
Feb. 9, 2016 04:00 PM EST Reads: 592
More and more companies are looking to microservices as an architectural pattern for breaking apart applications into more manageable pieces so that agile teams can deliver new features quicker and more effectively. What this pattern has done more than anything to date is spark organizational transformations, setting the foundation for future application development. In practice, however, there are a number of considerations to make that go beyond simply “build, ship, and run,” which changes ho...
Feb. 9, 2016 04:00 PM EST Reads: 197
With the proliferation of both SQL and NoSQL databases, organizations can now target specific fit-for-purpose database tools for their different application needs regarding scalability, ease of use, ACID support, etc. Platform as a Service offerings make this even easier now, enabling developers to roll out their own database infrastructure in minutes with minimal management overhead. However, this same amount of flexibility also comes with the challenges of picking the right tool, on the right ...
Feb. 9, 2016 03:45 PM EST Reads: 133
Fortunately, meaningful and tangible business cases for IoT are plentiful in a broad array of industries and vertical markets. These range from simple warranty cost reduction for capital intensive assets, to minimizing downtime for vital business tools, to creating feedback loops improving product design, to improving and enhancing enterprise customer experiences. All of these business cases, which will be briefly explored in this session, hinge on cost effectively extracting relevant data from ...
Feb. 9, 2016 03:15 PM EST
SYS-CON Events announced today that Catchpoint Systems, Inc., a provider of innovative web and infrastructure monitoring solutions, has been named “Silver Sponsor” of SYS-CON's DevOps Summit at 18th Cloud Expo New York, which will take place June 7-9, 2016, at the Javits Center in New York City, NY. Catchpoint is a leading Digital Performance Analytics company that provides unparalleled insight into customer-critical services to help consistently deliver an amazing customer experience. Designed...
Feb. 9, 2016 03:00 PM EST Reads: 353
As enterprises work to take advantage of Big Data technologies, they frequently become distracted by product-level decisions. In most new Big Data builds this approach is completely counter-productive: it presupposes tools that may not be a fit for development teams, forces IT to take on the burden of evaluating and maintaining unfamiliar technology, and represents a major up-front expense. In his session at @BigDataExpo at @ThingsExpo, Andrew Warfield, CTO and Co-Founder of Coho Data, will dis...
Feb. 9, 2016 03:00 PM EST Reads: 168
In most cases, it is convenient to have some human interaction with a web (micro-)service, no matter how small it is. A traditional approach would be to create an HTTP interface, where user requests will be dispatched and HTML/CSS pages must be served. This approach is indeed very traditional for a web site, but not really convenient for a web service, which is not intended to be good looking, 24x7 up and running and UX-optimized. Instead, talking to a web service in a chat-bot mode would be muc...
Feb. 9, 2016 03:00 PM EST Reads: 235
SYS-CON Events announced today that Men & Mice, the leading global provider of DNS, DHCP and IP address management overlay solutions, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. The Men & Mice Suite overlay solution is already known for its powerful application in heterogeneous operating environments, enabling enterprises to scale without fuss. Building on a solid range of diverse platform support,...
Feb. 9, 2016 03:00 PM EST Reads: 186
SYS-CON Events announced today that iDevices®, the preeminent brand in the connected home industry, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. iDevices, the preeminent brand in the connected home industry, has a growing line of HomeKit-enabled products available at the largest retailers worldwide. Through the “Designed with iDevices” co-development program and its custom-built IoT Cloud Infrastruc...
Feb. 9, 2016 02:45 PM EST
SYS-CON Events announced today that Pythian, a global IT services company specializing in helping companies adopt disruptive technologies to optimize revenue-generating systems, has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Founded in 1997, Pythian is a global IT services company that helps companies compete by adopting disruptive technologies such as cloud, Big Data, advanced analytics, and DevO...
Feb. 9, 2016 02:45 PM EST Reads: 193
SYS-CON Events announced today that Alert Logic, Inc., the leading provider of Security-as-a-Service solutions for the cloud, will exhibit at SYS-CON's 18th International Cloud Expo®, which will take place on June 7-9, 2016, at the Javits Center in New York City, NY. Alert Logic, Inc., provides Security-as-a-Service for on-premises, cloud, and hybrid infrastructures, delivering deep security insight and continuous protection for customers at a lower cost than traditional security solutions. Ful...
Feb. 9, 2016 02:15 PM EST Reads: 393
Advances in technology and ubiquitous connectivity have made the utilization of a dispersed workforce more common. Whether that remote team is located across the street or country, management styles/ approaches will have to be adjusted to accommodate this new dynamic. In his session at 17th Cloud Expo, Sagi Brody, Chief Technology Officer at Webair Internet Development Inc., focused on the challenges of managing remote teams, providing real-world examples that demonstrate what works and what do...
Feb. 9, 2016 02:00 PM EST Reads: 262
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2015 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 ad...
Feb. 9, 2016 01:15 PM EST Reads: 372
How Best to Integrate Cloud Foundry into Your Existing Ecosystem By @Gidrontxt | @DevOpsSummit #DevOps
As someone who has been dedicated to automation and Application Release Automation (ARA) technology for almost six years now, one of the most common questions I get asked regards Platform-as-a-Service (PaaS). Specifically, people want to know whether release automation is still needed when a PaaS is in place, and why. Isn't that what a PaaS provides? A solution to the deployment and runtime challenges of an application? Why would anyone using a PaaS then need an automation engine with workflow ...
Feb. 9, 2016 12:15 PM EST Reads: 172
Predictive analytics tools monitor, report, and troubleshoot in order to make proactive decisions about the health, performance, and utilization of storage. Most enterprises combine cloud and on-premise storage, resulting in blended environments of physical, virtual, cloud, and other platforms, which justifies more sophisticated storage analytics. In his session at 18th Cloud Expo, Peter McCallum, Vice President of Datacenter Solutions at FalconStor, will discuss using predictive analytics to ...
Feb. 9, 2016 11:45 AM EST Reads: 384
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 18th International Cloud Expo, which will take place on June 7–9, 2016, at the Javits Center in New York City, NY, and the 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management...
Feb. 9, 2016 11:30 AM EST Reads: 404