@CloudExpo Authors: Elizabeth White, Pat Romanski, Liz McMillan, Yeshim Deniz, Ram Sonagara

Related Topics: @CloudExpo, Microservices Expo, Containers Expo Blog, Agile Computing, @DXWorldExpo, SDN Journal

@CloudExpo: Article

Can We Finally Find the Database Holy Grail?

What are the alternative patterns for designing a truly distributed transactional database system?

The world runs on transactional database systems. Every business depends on them, and we each interact with them many times each day. Furthermore the world needs to build thousands more applications of transactional database systems to support the next-generation web. Nothing controversial there, but there is a problem: transactional database systems have stubbornly refused to join the 21st century.

The rest of the world is moving toward data center architectures predicated on thousands of commodity machines, commodity networks, and "scale-out" designs. These data centers will offer on-demand computing services that can instantly increase or decrease capacity to applications as needed. It's easy to just add more web servers, application servers, or storage servers as required. Unfortunately while that works at every other level in the computing stack it does not work at the database layer. Database systems are scale-up systems, not scale-out systems.

Traditionally if you want a transactional database to go faster you need a bigger machine. Period. Solving that problem is the Holy Grail of the database world. And as with the other Holy Grail, many people have given up the quest. But there is a new set of ideas that may just change the game. There are three common design patterns for building distributed transactional databases ("Shared Disk", "Shared Nothing" and "Synchronous Commit"), and there is now a new idea called Durable Distributed Cache (DDC). It is this fourth way that we at NuoDB are really excited about. But it makes sense to provide some background before talking about that.

Database transactions are pretty powerful things. People have often asked me to name applications that really need transactions, and my response is usually to ask them to name an application that really needs a database. It's the wrong question in both cases, as you plainly don't really need transactions, databases, compilers, operating systems, etc., any more than you really need an umbrella. The absence of some of these things can be worked around, and other tools and subsystems can be substituted by a rudimentary home-baked thing. The reason for using database transactions is not that you can't find some kludgy way of working around them in your application, but that it vastly simplifies the semantics of your system, especially as relates to exceptional conditions. The simplification serves to reduce cost, reduce implementation time and increase quality and maintainability of the end product. Transactional guarantees are highly desirable in almost any application but we sometimes consider the computational (and dollar) costs too high. The best answer would not be in the direction of using transactional systems more sparingly, but in a technological innovation that allows us to use them more pervasively.

In order to illustrate the situation, let's take an example in which you and I both want to take out the last $1,000 from the same account at the same time. The account balance is stored in an ACID compliant transactional database, and therefore the database system must ensure that at most one of us succeeds. There are multiple ways to do this if both transactions have to run on the same machine. Whatever algorithm is used it effectively orders the transactions so that whichever of us is judged to have arrived first will get the $1,000. That works. And one cool thing is that it will work even faster if I get a bigger machine.

But what if I don't want to buy a bigger machine? What if I want to speed it up by using a second small machine? Even in our very simple example you can quickly see the problem. Each of us has a transaction running on our behalf that is trying to decrement the $1,000 account balance by $1,000. If your transactions run on different machines, then we would each have more machine resources allocated to us and might expect to go faster, but our machines have to coordinate. There is no choice because it is not allowable for both transactions to succeed. In principle each machine needs to obtain permission to update the value of the account balance. This involves network communications, and is orders-of-magnitude more expensive than coordinating the two transactions on a single machine. In many cases a second machine can actually slow down your applications, hence the traditional answer of just getting a bigger machine.

The Ever Bigger Machine strategy is far from ideal. Obviously there are limits to how big your big machine can be. You have to hope that your new web site does not attract too many active users for your maxed-out database server to handle. In addition to on-demand capacity, scale-out systems make other very tempting promises. Aside from much more attractive base economics relating to commodity hardware and/or cloud deployment, they also allow you to provision capacity to your load requirements, and potentially offer much better redundancy and availability models. So we are directed to the topic of this blog series, namely what are the alternative patterns for designing a truly distributed transactional database system?

In the rest of this series I'll lay out my views of the four alternative models: Shared-Disk, Shared-Nothing, Synchronous Commit and Durable Distributed Cache (DDC).

More Stories By Barry Morris

Barry Morris is CEO & Co-Founder of NuoDB, Inc. An accomplished software CEO with over 25 years of industry experience in the USA and Europe, running private and public companies ranging in scale from early startup phase to 1,000+ employees, he loves to build companies around industry-changing paradigm-shifts in technology. Morris was previously CEO of StreamBase and Iona Technologies.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.

@CloudExpo Stories
DXWorldEXPO LLC announced today that Kevin Jackson joined the faculty of CloudEXPO's "10-Year Anniversary Event" which will take place on November 11-13, 2018 in New York City. Kevin L. Jackson is a globally recognized cloud computing expert and Founder/Author of the award winning "Cloud Musings" blog. Mr. Jackson has also been recognized as a "Top 100 Cybersecurity Influencer and Brand" by Onalytica (2015), a Huffington Post "Top 100 Cloud Computing Experts on Twitter" (2013) and a "Top 50 C...
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol,...
Enterprises have taken advantage of IoT to achieve important revenue and cost advantages. What is less apparent is how incumbent enterprises operating at scale have, following success with IoT, built analytic, operations management and software development capabilities - ranging from autonomous vehicles to manageable robotics installations. They have embraced these capabilities as if they were Silicon Valley startups.
Daniel Jones is CTO of EngineerBetter, helping enterprises deliver value faster. Previously he was an IT consultant, indie video games developer, head of web development in the finance sector, and an award-winning martial artist. Continuous Delivery makes it possible to exploit findings of cognitive psychology and neuroscience to increase the productivity and happiness of our teams.
The standardization of container runtimes and images has sparked the creation of an almost overwhelming number of new open source projects that build on and otherwise work with these specifications. Of course, there's Kubernetes, which orchestrates and manages collections of containers. It was one of the first and best-known examples of projects that make containers truly useful for production use. However, more recently, the container ecosystem has truly exploded. A service mesh like Istio addr...
Poor data quality and analytics drive down business value. In fact, Gartner estimated that the average financial impact of poor data quality on organizations is $9.7 million per year. But bad data is much more than a cost center. By eroding trust in information, analytics and the business decisions based on these, it is a serious impediment to digital transformation.
Predicting the future has never been more challenging - not because of the lack of data but because of the flood of ungoverned and risk laden information. Microsoft states that 2.5 exabytes of data are created every day. Expectations and reliance on data are being pushed to the limits, as demands around hybrid options continue to grow.
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
Digital Transformation: Preparing Cloud & IoT Security for the Age of Artificial Intelligence. As automation and artificial intelligence (AI) power solution development and delivery, many businesses need to build backend cloud capabilities. Well-poised organizations, marketing smart devices with AI and BlockChain capabilities prepare to refine compliance and regulatory capabilities in 2018. Volumes of health, financial, technical and privacy data, along with tightening compliance requirements by...
The IoT Will Grow: In what might be the most obvious prediction of the decade, the IoT will continue to expand next year, with more and more devices coming online every single day. What isn’t so obvious about this prediction: where that growth will occur. The retail, healthcare, and industrial/supply chain industries will likely see the greatest growth. Forrester Research has predicted the IoT will become “the backbone” of customer value as it continues to grow. It is no surprise that retail is ...
Evan Kirstel is an internationally recognized thought leader and social media influencer in IoT (#1 in 2017), Cloud, Data Security (2016), Health Tech (#9 in 2017), Digital Health (#6 in 2016), B2B Marketing (#5 in 2015), AI, Smart Home, Digital (2017), IIoT (#1 in 2017) and Telecom/Wireless/5G. His connections are a "Who's Who" in these technologies, He is in the top 10 most mentioned/re-tweeted by CMOs and CIOs (2016) and have been recently named 5th most influential B2B marketeer in the US. H...
Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settlement products to hedge funds and investment banks. After, he co-founded a revenue cycle management company where he learned about Bitcoin and eventually Ethereal. Andrew's role at ConsenSys Enterprise is a mul...
The best way to leverage your Cloud Expo presence as a sponsor and exhibitor is to plan your news announcements around our events. The press covering Cloud Expo and @ThingsExpo will have access to these releases and will amplify your news announcements. More than two dozen Cloud companies either set deals at our shows or have announced their mergers and acquisitions at Cloud Expo. Product announcements during our show provide your company with the most reach through our targeted audiences.
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of bus...
With 10 simultaneous tracks, keynotes, general sessions and targeted breakout classes, @CloudEXPO and DXWorldEXPO are two of the most important technology events of the year. Since its launch over eight years ago, @CloudEXPO and DXWorldEXPO have presented a rock star faculty as well as showcased hundreds of sponsors and exhibitors! In this blog post, we provide 7 tips on how, as part of our world-class faculty, you can deliver one of the most popular sessions at our events. But before reading...
DXWorldEXPO LLC announced today that "Miami Blockchain Event by FinTechEXPO" has announced that its Call for Papers is now open. The two-day event will present 20 top Blockchain experts. All speaking inquiries which covers the following information can be submitted by email to [email protected] Financial enterprises in New York City, London, Singapore, and other world financial capitals are embracing a new generation of smart, automated FinTech that eliminates many cumbersome, slow, and expe...
Cloud Expo | DXWorld Expo have announced the conference tracks for Cloud Expo 2018. Cloud Expo will be held June 5-7, 2018, at the Javits Center in New York City, and November 6-8, 2018, at the Santa Clara Convention Center, Santa Clara, CA. Digital Transformation (DX) is a major focus with the introduction of DX Expo within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive ov...
DXWordEXPO New York 2018, colocated with CloudEXPO New York 2018 will be held November 11-13, 2018, in New York City and will bring together Cloud Computing, FinTech and Blockchain, Digital Transformation, Big Data, Internet of Things, DevOps, AI, Machine Learning and WebRTC to one location.
DXWorldEXPO | CloudEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
@DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises - and delivering real results.