Welcome!

@CloudExpo Authors: William Schmarzo, Liz McMillan, Elizabeth White, Scott Allen, Sematext Blog

Related Topics: @CloudExpo

@CloudExpo: Article

Cloud Database Design, Scale Out Using Shared Nothing Pattern

Virtual storage and database partitions

Abstraction Vs Flexbility :
It is evident that storage plays a major part in the data center and for cloud services. The storage virtualization plays a key part in the dynamic infrastructure attribute of Cloud Computing. Which means the storage is provisioned and de-allocated on demand and usage needs. The good part is that this complex stuff is hidden from the cloud consumer.

However while the storage allocation is abstracted it also brings in performance concerns in a multi tenant cloud environment where by most of the cloud consumers are geographically dispersed and a good amount of data retrieval and manipulation at stake.

Currently most cloud storage virtualization platforms rely on the hardware options like fiber channel, Ethernet switch, network attached storage (NAS) and iSCSI as the implementation medium.

Currently Cloud platforms have very little support for database design related virtualization enhancements. But in future designing databases specific for Cloud especially for private clouds in large enterprises is a sure possibility. In this context the following design principles are important when you design database applications which needs to be delivered using Cloud platform.

Shared Nothing Vs Shared Every Thing
Shared resource and shared nothing represent forms of data access architectures. In a shared (every thing) resource model various processes in the DBMS have access to all system resources, including the data. In the shared-nothing environment, separate DBMS resources divide up the workload, rach responsible for its own data, memory locations, and other resources.

Most popular databases support a shared-nothing model using different strategies. For example,

  • Oracle : Utilizes Range & Hash portioning of tables towards shared nothing . Here it is a table and table space level segregation of data based either on range of key values or a hashing algorithm. (note : Oracle Real Application clusters at the instance level adopt a Shared Every Thing Model)
  • DB2 UDB follows a shared nothing model. Data is partitioned according to a partition key. Rows are assigned to a partition, and each partition has total control of that row. If another partition wants to read or update a row, it must send the request to the owning partition. The owning partition then executes the command on behalf of the requestor.
  • Sql Server : Utilizes Distributed Partitioned views to implement Shared nothing architecture.

Shared nothing model greatly simplifies things like resource contention, including memory, locks and processors . Implemented properly , it offers unlimited scalability. As new rows or data sources are added, more partitions can be added. The workload on any individual partition remains the same.

Storage Virtualization & Shared Nothing Meet Each Other

While the dynamic infrastructure tenant of cloud computing is satisfied by the current classic model as seen in the picture , but the load balancing and scalability needs are fully satisfied by hardware at this time. However implementing the shared nothing data access architecture with a proper access key will highly improve the scalability and performance of cloud databases which in turn increases the QoS of the Cloud platform and make the XaaS a real success providing the best of cost efficiency and improved performance.

View Of a Shared Nothing Cloud Database

As evident implementing a Shared Nothing Cloud Database Model does not replace Storage Virtualization, but compliments the scalability further by splitting the data within multiple virtual storage drives. As seen in the diagram,

Load balancing algorithms implemented at the shared nothing cloud databases further augment the Load balancing options provided by the Cloud Platform.

Shared Nothing Cloud Database Design Considerations

  • Implement Hashing or Ranging Keys as part of the tables which logically separate the data based on the Cloud platform needs. For example if in a SaaS platform multi tenancy is the need then having the CUSTOMER ID or CUSTOMER KEY as a Hashing key for the shared nothing cloud databases
  • Increase the scalability further by implementing a Geography or Time based partition which scales up well depending on the situations
  • Ensure that the queries utilize the logical partitions established as part of the shared nothing architecture. For example if you can query a customer by SSN in a cloud database which has implemented a shared nothing architecture using Geography, then utilize the geography also part of the query filter conditions
  • Have an option to rebalance the data within the virtualized storage if they started becoming skewed due to the way the hashing algorithm is built in

What is expected from Cloud Platform Providers & Database Vendors

  • Expecting the database vendors provide shared nothing algorithms tailored for virtual storage, especially on the rebalancing based on the disk usage
  • New cloud aware partitioning techniques
  • Complimenting load balancing algorithm to augment what is currently available at the hardware
  • Ability to PIN SQLs to a virtual machine based on dynamically queried resource usage

Over all Private Clouds at Enterprise need to satisfy still more SLA or QoS as part of their service and full support from database engines is key to the success. Share Nothing Architecture is one important thing to achieve these non functional needs of the Cloud Platform.

More Stories By Srinivasan Sundara Rajan

Highly passionate about utilizing Digital Technologies to enable next generation enterprise. Believes in enterprise transformation through the Natives (Cloud Native & Mobile Native).

@CloudExpo Stories
All clouds are not equal. To succeed in a DevOps context, organizations should plan to develop/deploy apps across a choice of on-premise and public clouds simultaneously depending on the business needs. This is where the concept of the Lean Cloud comes in - resting on the idea that you often need to relocate your app modules over their life cycles for both innovation and operational efficiency in the cloud. In his session at @DevOpsSummit at19th Cloud Expo, Valentin (Val) Bercovici, CTO of So...
SYS-CON Events announced today that Commvault, a global leader in enterprise data protection and information management, has been named “Bronze Sponsor” of SYS-CON's 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. Commvault is a leading provider of data protection and information management solutions, helping companies worldwide activate their data to drive more value and business insight and to transform moder...
SYS-CON Events announced today that eCube Systems, a leading provider of middleware modernization, integration, and management solutions, will exhibit at @DevOpsSummit at 19th International Cloud Expo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. eCube Systems offers a family of middleware evolution products and services that maximize return on technology investment by leveraging existing technical equity to meet evolving business needs. ...
Personalization has long been the holy grail of marketing. Simply stated, communicate the most relevant offer to the right person and you will increase sales. To achieve this, you must understand the individual. Consequently, digital marketers developed many ways to gather and leverage customer information to deliver targeted experiences. In his session at @ThingsExpo, Lou Casal, Founder and Principal Consultant at Practicala, discussed how the Internet of Things (IoT) has accelerated our abil...
SYS-CON Events has announced today that Roger Strukhoff has been named conference chair of Cloud Expo and @ThingsExpo 2016 Silicon Valley. The 19th Cloud Expo and 6th @ThingsExpo will take place on November 1-3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. "The Internet of Things brings trillions of dollars of opportunity to developers and enterprise IT, no matter how you measure it," stated Roger Strukhoff. "More importantly, it leverages the power of devices and the Interne...
Digital innovation is the next big wave of business transformation based on digital technologies of which IoT and Big Data are key components, For example: Business boundary innovation is a challenge to excavate third-party business value using IoT and BigData, like Nest Business structure innovation may propose re-building business structure from scratch, as Uber does in the taxicab industry The social model innovation is also a big challenge to the new social architecture with the design fr...
Whether they’re located in a public, private, or hybrid cloud environment, cloud technologies are constantly evolving. While the innovation is exciting, the end mission of delivering business value and rapidly producing incremental product features is paramount. In his session at @DevOpsSummit at 19th Cloud Expo, Kiran Chitturi, CTO Architect at Sungard AS, will discuss DevOps culture, its evolution of frameworks and technologies, and how it is achieving maturity. He will also cover various st...
If you had a chance to enter on the ground level of the largest e-commerce market in the world – would you? China is the world’s most populated country with the second largest economy and the world’s fastest growing market. It is estimated that by 2018 the Chinese market will be reaching over $30 billion in gaming revenue alone. Admittedly for a foreign company, doing business in China can be challenging. Often changing laws, administrative regulations and the often inscrutable Chinese Interne...
So, you bought into the current machine learning craze and went on to collect millions/billions of records from this promising new data source. Now, what do you do with them? Too often, the abundance of data quickly turns into an abundance of problems. How do you extract that "magic essence" from your data without falling into the common pitfalls? In her session at @ThingsExpo, Natalia Ponomareva, Software Engineer at Google, provided tips on how to be successful in large scale machine learning...
Why do your mobile transformations need to happen today? Mobile is the strategy that enterprise transformation centers on to drive customer engagement. In his general session at @ThingsExpo, Roger Woods, Director, Mobile Product & Strategy – Adobe Marketing Cloud, covered key IoT and mobile trends that are forcing mobile transformation, key components of a solid mobile strategy and explored how brands are effectively driving mobile change throughout the enterprise.
In his session at @ThingsExpo, Kausik Sridharabalan, founder and CTO of Pulzze Systems, Inc., will focus on key challenges in building an Internet of Things solution infrastructure. He will shed light on efficient ways of defining interactions within IoT solutions, leading to cost and time reduction. He will also introduce ways to handle data and how one can develop IoT solutions that are lean, flexible and configurable, thus making IoT infrastructure agile and scalable.
Creating replica copies to tolerate a certain number of failures is easy, but very expensive at cloud-scale. Conventional RAID has lower overhead, but it is limited in the number of failures it can tolerate. And the management is like herding cats (overseeing capacity, rebuilds, migrations, and degraded performance). Download Slide Deck: ▸ Here In his general session at 18th Cloud Expo, Scott Cleland, Senior Director of Product Marketing for the HGST Cloud Infrastructure Business Unit, discusse...
Data is an unusual currency; it is not restricted by the same transactional limitations as money or people. In fact, the more that you leverage your data across multiple business use cases, the more valuable it becomes to the organization. And the same can be said about the organization’s analytics. In his session at 19th Cloud Expo, Bill Schmarzo, CTO for the Big Data Practice at EMC, will introduce a methodology for capturing, enriching and sharing data (and analytics) across the organizati...
Internet of @ThingsExpo has announced today that Chris Matthieu has been named tech chair of Internet of @ThingsExpo 2016 Silicon Valley. The 6thInternet of @ThingsExpo will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA.
SYS-CON Events announced today that Bsquare has been named “Silver Sponsor” of SYS-CON's @ThingsExpo, which will take place on November 1–3, 2016, at the Santa Clara Convention Center in Santa Clara, CA. For more than two decades, Bsquare has helped its customers extract business value from a broad array of physical assets by making them intelligent, connecting them, and using the data they generate to optimize business processes.
A strange thing is happening along the way to the Internet of Things, namely far too many devices to work with and manage. It has become clear that we'll need much higher efficiency user experiences that can allow us to more easily and scalably work with the thousands of devices that will soon be in each of our lives. Enter the conversational interface revolution, combining bots we can literally talk with, gesture to, and even direct with our thoughts, with embedded artificial intelligence, wh...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life sett...
Cloud computing is being adopted in one form or another by 94% of enterprises today. Tens of billions of new devices are being connected to The Internet of Things. And Big Data is driving this bus. An exponential increase is expected in the amount of information being processed, managed, analyzed, and acted upon by enterprise IT. This amazing is not part of some distant future - it is happening today. One report shows a 650% increase in enterprise data by 2020. Other estimates are even higher....
Most of us already know that adopting new cloud applications can boost a business’s productivity by enabling organizations to be more agile and ready to change course in our fast-moving and connected digital world. But the rapid adoption of cloud apps and services also brings with it profound security threats, including visibility and control challenges that aren’t present in traditional on-premises environments. At the same time, the cloud – because of its interconnected, flexible and adaptable...
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.