Welcome!

@CloudExpo Authors: Elizabeth White, Yeshim Deniz, Kevin Sparenberg, Karyn Jeffery, Liz McMillan

Related Topics: @CloudExpo, Microservices Expo, Containers Expo Blog

@CloudExpo: Article

The Growing Relevance of In-Memory Data Grids

Cloud is bringing IMDG technology back to the spotlight

The growing consumer affinity to cloud is spurring on various new technological trends. It's not all new technology mind you, but there seems to be a growing appetite for anything that can remotely be put into the context of cloud computing. In some ways, cloud has been good for bringing previously existing technologies back to the forefront and resulting in needed innovation in the area. Besides virtualization technologies, I believe that one of the best examples of this is in-memory data grid technology.

Despite what some may try to purport, in-memory data grids are not a by-product of the cloud computing revolution. The truth is they existed for a while before cloud, but to be fair, IMDGs probably owe a tip of the hat to the cloud computing craze for bringing them back into the spotlight. Increasingly, we are seeing workloads that are highly scalable, temporal, and elastic making their way into the cloud. These application characteristics often align nicely with the use of IMDGs, so we see renewed interest and quite a bit of innovative activity around these solutions.

I have been spending a lot of time lately working with users that are in the process of defining an evolution to their current enterprise middleware environments. In this work, cloud and IMDG technology has been front and center. My last few posts have been dedicated to talking about some of the cloud trends I have seen during the course of this work, but today I want to focus on IMDG solutions. Specifically, I want to share with you what I'm seeing in terms of how users are currently looking to use this technology, and provide my own thoughts about possible usage scenarios going forward. Let's start with the common, currently targeted usage scenarios:

1) IMDG as a database buffer: This is perhaps the single most common use case. Here users look to front traditional data stores with a distributed IMDG. This can serve to increase the performance of the application and thereby improve end-user experience by offering faster data access. It can also help to reduce the pressure on the backend database by making the IMDG instance the system of record and periodically batching changes to send off to the database. There are many different techniques for buffering an existing database with an IMDG, but the motivations (increase performance, decrease database reliance, decrease costs) are usually much the same.

2) IMDG as a simple cache: You may hear this referred to as the side cache scenario as well. This usage pattern is a little less intrusive to existing application architecture than #1 above. Here, applications receive an incoming request that requires data, and they first check in the simple cache instance to see if the data exists there. If not, the application proceeds to retrieve the data normally, typically from a relational database system, and then inserts this data into the simple cache. Obviously, when the application finds data in the simple cache, you reduce the path of the application and thus decrease overall response time. This is an especially prevalent pattern for storing conversational state (think HTTP sessions) for applications.

3) IMDG as a service request cache: This is really a variant of #2 above, but I call it out separately because users typically implement it at a different tier in the application architecture. Instead of updating the application to be aware of some IMDG for a simple cache, users insert this awareness further up the stream, often at the ESB tier. Requests come into the application environment, but before they hit the application, some component dissects the request to determine if it can fulfill it from the cache. If so, the entire path becomes significantly shorter, and if not the mediation component inserts the response in the simple cache on the way back out. Again, it is not very different than #2, but in my opinion it is worth distinguishing as it occurs at a completely different tier in the overall architecture.

Those are three common patterns that I run into quite frequently today with IMDG technology. There are a couple of more usage patterns that I have seen once or twice, but are not yet prevalent. ‘Yet' is the key word here as I tend to think we will be seeing more of these use cases in the near future:

1) IMDG as an event filter: To be clear here, I am not suggesting that IMDG instances would morph into event processors. There are completely separate solutions that do that very well. What I am talking about is using the distributed logic processing capabilities shipped with most IMDG solutions to quickly determine if a given event needs further processing. In this way, events that occur in the enterprise can flow through an easily scalable IMDG filter, and then only if necessary sent to an event processor for more expensive computation. In the increasingly event-driven architectures that are emerging today, I feel this could become a popular pattern.

2) IMDG as a map/reduce engine: Many IMDG solutions deliver the capability to distribute logic, as mentioned in #1 above, and many offer a map/reduce model as a means to do this. As skills on map/reduce programming start to permeate enterprise IT shops, I think users will see compelling use cases for IMDG built primarily on this methodology. The ability to quickly distribute logic, calculate results, and further refine those results all out in the grid is a powerful tool to have in your arsenal. It can in fact be quite liberating to leverage the processing power of a distributed grid to solve important yet complex business problems.

Now, I have no idea if the two use cases above will in fact pan out as mainstream in the near future. I see massive potential there because of their alignment with emerging architectures and their ability to deliver real business value.  However, I can also imagine IMDG technology taking off in entirely different directions. Whatever the near future holds though, I think one thing is certain: We are just beginning to explore the art of the possible when it comes to IMDG technology.

More Stories By Dustin Amrhein

Dustin Amrhein joined IBM as a member of the development team for WebSphere Application Server. While in that position, he worked on the development of Web services infrastructure and Web services programming models. In his current role, Dustin is a technical specialist for cloud, mobile, and data grid technology in IBM's WebSphere portfolio. He blogs at http://dustinamrhein.ulitzer.com. You can follow him on Twitter at http://twitter.com/damrhein.

@CloudExpo Stories
"I think DevOps is now a rambunctious teenager – it’s starting to get a mind of its own, wanting to get its own things but it still needs some adult supervision," explained Thomas Hooker, VP of marketing at CollabNet, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We are still a relatively small software house and we are focusing on certain industries like FinTech, med tech, energy and utilities. We help our customers with their digital transformation," noted Piotr Stawinski, Founder and CEO of EARP Integration, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"DX encompasses the continuing technology revolution, and is addressing society's most important issues throughout the entire $78 trillion 21st-century global economy," said Roger Strukhoff, Conference Chair. "DX World Expo has organized these issues along 10 tracks with more than 150 of the world's top speakers coming to Istanbul to help change the world."
"We've been engaging with a lot of customers including Panasonic, we've been involved with Cisco and now we're working with the U.S. government - the Department of Homeland Security," explained Peter Jung, Chief Product Officer at Pulzze Systems, in this SYS-CON.tv interview at @ThingsExpo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We're here to tell the world about our cloud-scale infrastructure that we have at Juniper combined with the world-class security that we put into the cloud," explained Lisa Guess, VP of Systems Engineering at Juniper Networks, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"I will be talking about ChatOps and ChatOps as a way to solve some problems in the DevOps space," explained Himanshu Chhetri, CTO of Addteq, in this SYS-CON.tv interview at @DevOpsSummit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"We are focused on SAP running in the clouds, to make this super easy because we believe in the tremendous value of those powerful worlds - SAP and the cloud," explained Frank Stienhans, CTO of Ocean9, Inc., in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, provided a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services with...
The financial services market is one of the most data-driven industries in the world, yet it’s bogged down by legacy CPU technologies that simply can’t keep up with the task of querying and visualizing billions of records. In his session at 20th Cloud Expo, Karthik Lalithraj, a Principal Solutions Architect at Kinetica, discussed how the advent of advanced in-database analytics on the GPU makes it possible to run sophisticated data science workloads on the same database that is housing the rich...
SYS-CON Events announced today that Massive Networks will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Massive Networks mission is simple. To help your business operate seamlessly with fast, reliable, and secure internet and network solutions. Improve your customer's experience with outstanding connections to your cloud.
"We are an IT services solution provider and we sell software to support those solutions. Our focus and key areas are around security, enterprise monitoring, and continuous delivery optimization," noted John Balsavage, President of A&I Solutions, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution and join Akvelon expert and IoT industry leader, Sergey Grebnov, in his session at @ThingsExpo, for an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
"We want to show that our solution is far less expensive with a much better total cost of ownership so we announced several key features. One is called geo-distributed erasure coding, another is support for KVM and we introduced a new capability called Multi-Part," explained Tim Desai, Senior Product Marketing Manager at Hitachi Data Systems, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol,...
DX World EXPO, LLC., a Lighthouse Point, Florida-based startup trade show producer and the creator of "DXWorldEXPO® - Digital Transformation Conference & Expo" has announced its executive management team. The team is headed by Levent Selamoglu, who has been named CEO. "Now is the time for a truly global DX event, to bring together the leading minds from the technology world in a conversation about Digital Transformation," he said in making the announcement.
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
"The Striim platform is a full end-to-end streaming integration and analytics platform that is middleware that covers a lot of different use cases," explained Steve Wilkes, Founder and CTO at Striim, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that Calligo, an innovative cloud service provider offering mid-sized companies the highest levels of data privacy and security, has been named "Bronze Sponsor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalised support service from its globally located cloud plat...
21st International Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Me...