@CloudExpo Authors: Yeshim Deniz, Jnan Dash, Liz McMillan, Zakia Bouachraoui, Janakiram MSV

Related Topics: @CloudExpo, Recurring Revenue

@CloudExpo: Article

Oracle RDBMS and Very Large Data Set Processing

An overview of very large data set processing technologies with an emphasis on Oracle RDBMS

Oracle database is a relational database management system that mostly complies with ACID transaction requirements ( atomicity, consistency, isolation, durability ). It means that each database transaction will be executed in a reliable, safe and integral manner. In order to comply with ACID Oracle database software implements fairly complex and expensive (in terms of computing resources, i.e., CPU, disk, memory) set of processes like redo and undo logging, memory latching, meta data maintenance etc. that make concurrent work possible, while maintaining data integrity. Any database transaction or even SELECT statement makes relational database systems perform tremendous amounts of work behind the scene, thus making it inherently slow and resource intensive.

Oracle is trying to address scalability and performance problem in a variety of ways:

- by introducing constant performance enhancements to the query optimizer

- Oracle RAC - Real Application Clusters based scale out ( much increased complexity with little practical value in terms of performance and functionality )

- appliances ( Exa* line of products - complex, unbalanced architecture, suboptimally utilized hardware,  patched up and repackaged  software )

All these attempt still feature performance and scalability bottleneck in shape of Oracle RDBMS and its shared-nothing, or assymmetric MPP ( in case of Exadata ) architecture.

Companies dealing with millions of users, huge volumes of data and needing great performance like Google could not use proprietary RDBMSs.  They developed their own solutions, relying on utilizing commodity hardware and open source software. They developed software that can make thousands of Intel boxes behave like a single system that can process your query searching through petabytes of data  in sub-second time. This could never be accomplished if they used standard RDBMS like Oracle. RDBMSs were not designed to deal with problems that Google is facing.

Data processing technologies that originated at Google or other places ( including direct Google research descendant Hadoop,  NoSQL group of products, etc. )  parallelize work and distribute data over thousands of servers, relax ACID requirement so it now maybe becomes BASE (Basically Available, Soft state, Eventual consistency) i.e., they provide weaker consistency guarantees ( CAP theorem ), they loose relational data structure i.e. basically go back to flat files, or distributed, scalable hash tables. By relaxing, modifying or loosing some properties of RDBMs and optimizing to run on commodity hardware they were able to get  results that are good enough in terms of data quality and consistency, while achieving great performance and sufficient accuracy for very basic transactions. Ironically many of these technologies are step backwards i.e. they will end up reinventing many RDBMS features as they mature ( Hadapt - Hadoop based relational database; Hive, Hbase ).

Oracle RDBMS has completely different purpose, i.e., it is targeted for corporate/business use where complex transactions must be accurately executed. Eventually consistent paradigm does not suffice here. While Oracle database can contain huge volumes of multi-media data, its main purpose is to store and concurrently process structured data sets relatively limited in size, for a limited number of concurrent corporate/business users.

More Stories By Ranko Mosic

Ranko Mosic, BScEng, is specializing in Big Data/Data Architecture consulting services ( database/data architecture, machine learning ). His clients are in finance, retail, telecommunications industries. Ranko is welcoming inquiries about his availability for consulting engagements and can be reached at 408-757-0053 or [email protected]

CloudEXPO Stories
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, will discuss how to use Kubernetes to setup a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace. His expertise is in automating deployment, management, and problem resolution in these environments, allowing his teams to run large transactional applications with high availability and the speed the consumer demands.
At CloudEXPO Silicon Valley, June 24-26, 2019, Digital Transformation (DX) is a major focus with expanded DevOpsSUMMIT and FinTechEXPO programs within the DXWorldEXPO agenda. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of business. Only 12% still survive. Similar percentages are found throughout enterprises of all sizes.
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the benefits of the cloud without losing performance as containers become the new paradigm.
As you know, enterprise IT conversation over the past year have often centered upon the open-source Kubernetes container orchestration system. In fact, Kubernetes has emerged as the key technology -- and even primary platform -- of cloud migrations for a wide variety of organizations. Kubernetes is critical to forward-looking enterprises that continue to push their IT infrastructures toward maximum functionality, scalability, and flexibility.
Because Linkerd is a transparent proxy that runs alongside your application, there are no code changes required. It even comes with Prometheus to store the metrics for you and pre-built Grafana dashboards to show exactly what is important for your services - success rate, latency, and throughput. In this session, we'll explain what Linkerd provides for you, demo the installation of Linkerd on Kubernetes and debug a real world problem. We will also dig into what functionality you can build on top of the tools provided by Linkerd such as alerting and autoscaling.