Welcome!

@CloudExpo Authors: Pat Romanski, Zakia Bouachraoui, Elizabeth White, Yeshim Deniz, Liz McMillan

Related Topics: @CloudExpo, Microservices Expo, Containers Expo Blog

@CloudExpo: Article

The Growing Relevance of In-Memory Data Grids

Cloud is bringing IMDG technology back to the spotlight

The growing consumer affinity to cloud is spurring on various new technological trends. It's not all new technology mind you, but there seems to be a growing appetite for anything that can remotely be put into the context of cloud computing. In some ways, cloud has been good for bringing previously existing technologies back to the forefront and resulting in needed innovation in the area. Besides virtualization technologies, I believe that one of the best examples of this is in-memory data grid technology.

Despite what some may try to purport, in-memory data grids are not a by-product of the cloud computing revolution. The truth is they existed for a while before cloud, but to be fair, IMDGs probably owe a tip of the hat to the cloud computing craze for bringing them back into the spotlight. Increasingly, we are seeing workloads that are highly scalable, temporal, and elastic making their way into the cloud. These application characteristics often align nicely with the use of IMDGs, so we see renewed interest and quite a bit of innovative activity around these solutions.

I have been spending a lot of time lately working with users that are in the process of defining an evolution to their current enterprise middleware environments. In this work, cloud and IMDG technology has been front and center. My last few posts have been dedicated to talking about some of the cloud trends I have seen during the course of this work, but today I want to focus on IMDG solutions. Specifically, I want to share with you what I'm seeing in terms of how users are currently looking to use this technology, and provide my own thoughts about possible usage scenarios going forward. Let's start with the common, currently targeted usage scenarios:

1) IMDG as a database buffer: This is perhaps the single most common use case. Here users look to front traditional data stores with a distributed IMDG. This can serve to increase the performance of the application and thereby improve end-user experience by offering faster data access. It can also help to reduce the pressure on the backend database by making the IMDG instance the system of record and periodically batching changes to send off to the database. There are many different techniques for buffering an existing database with an IMDG, but the motivations (increase performance, decrease database reliance, decrease costs) are usually much the same.

2) IMDG as a simple cache: You may hear this referred to as the side cache scenario as well. This usage pattern is a little less intrusive to existing application architecture than #1 above. Here, applications receive an incoming request that requires data, and they first check in the simple cache instance to see if the data exists there. If not, the application proceeds to retrieve the data normally, typically from a relational database system, and then inserts this data into the simple cache. Obviously, when the application finds data in the simple cache, you reduce the path of the application and thus decrease overall response time. This is an especially prevalent pattern for storing conversational state (think HTTP sessions) for applications.

3) IMDG as a service request cache: This is really a variant of #2 above, but I call it out separately because users typically implement it at a different tier in the application architecture. Instead of updating the application to be aware of some IMDG for a simple cache, users insert this awareness further up the stream, often at the ESB tier. Requests come into the application environment, but before they hit the application, some component dissects the request to determine if it can fulfill it from the cache. If so, the entire path becomes significantly shorter, and if not the mediation component inserts the response in the simple cache on the way back out. Again, it is not very different than #2, but in my opinion it is worth distinguishing as it occurs at a completely different tier in the overall architecture.

Those are three common patterns that I run into quite frequently today with IMDG technology. There are a couple of more usage patterns that I have seen once or twice, but are not yet prevalent. ‘Yet' is the key word here as I tend to think we will be seeing more of these use cases in the near future:

1) IMDG as an event filter: To be clear here, I am not suggesting that IMDG instances would morph into event processors. There are completely separate solutions that do that very well. What I am talking about is using the distributed logic processing capabilities shipped with most IMDG solutions to quickly determine if a given event needs further processing. In this way, events that occur in the enterprise can flow through an easily scalable IMDG filter, and then only if necessary sent to an event processor for more expensive computation. In the increasingly event-driven architectures that are emerging today, I feel this could become a popular pattern.

2) IMDG as a map/reduce engine: Many IMDG solutions deliver the capability to distribute logic, as mentioned in #1 above, and many offer a map/reduce model as a means to do this. As skills on map/reduce programming start to permeate enterprise IT shops, I think users will see compelling use cases for IMDG built primarily on this methodology. The ability to quickly distribute logic, calculate results, and further refine those results all out in the grid is a powerful tool to have in your arsenal. It can in fact be quite liberating to leverage the processing power of a distributed grid to solve important yet complex business problems.

Now, I have no idea if the two use cases above will in fact pan out as mainstream in the near future. I see massive potential there because of their alignment with emerging architectures and their ability to deliver real business value.  However, I can also imagine IMDG technology taking off in entirely different directions. Whatever the near future holds though, I think one thing is certain: We are just beginning to explore the art of the possible when it comes to IMDG technology.

More Stories By Dustin Amrhein

Dustin Amrhein joined IBM as a member of the development team for WebSphere Application Server. While in that position, he worked on the development of Web services infrastructure and Web services programming models. In his current role, Dustin is a technical specialist for cloud, mobile, and data grid technology in IBM's WebSphere portfolio. He blogs at http://dustinamrhein.ulitzer.com. You can follow him on Twitter at http://twitter.com/damrhein.

CloudEXPO Stories
Lori MacVittie is a subject matter expert on emerging technology responsible for outbound evangelism across F5's entire product suite. MacVittie has extensive development and technical architecture experience in both high-tech and enterprise organizations, in addition to network and systems administration expertise. Prior to joining F5, MacVittie was an award-winning technology editor at Network Computing Magazine where she evaluated and tested application-focused technologies including app security and encryption-related solutions. She holds a B.S. in Information and Computing Science from the University of Wisconsin at Green Bay, and an M.S. in Computer Science from Nova Southeastern University, and is an O'Reilly author.
CloudEXPO New York 2018, colocated with DevOpsSUMMIT and DXWorldEXPO New York 2018 will be held November 12-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI and Machine Learning to one location.
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
ICC is a computer systems integrator and server manufacturing company focused on developing products and product appliances to meet a wide range of computational needs for many industries. Their solutions provide benefits across many environments, such as datacenter deployment, HPC, workstations, storage networks and standalone server installations. ICC has been in business for over 23 years and their phenomenal range of clients include multinational corporations, universities, and small businesses.
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or personal computing needs.