Welcome!

@CloudExpo Authors: Pat Romanski, Liz McMillan, Kong Yang, Jason Bloomberg, Derek Weeks

Related Topics: @CloudExpo

@CloudExpo: Article

Creating a Common Cloud Computing Reference API - Part One

A lot of what I'm outlining here may be better suited to a reference implementation than a standard

Reuven Cohen's "ElasticVapor" Blog

Over the last few weeks I've been engaged in several conversations about the need for a common, interoperable and open set of cloud computing standards. During these conversations a recurring theme has started to emerge. A need for cloud interoperability or the ability for diverse cloud systems and organizations to work together in a common way.

In my discussion yesterday with Rich Wolski of the Eucalyptus project he described the need for a "CloudVirt" API similar to that of the Libvirt project for virtualization. For those of you that don't know about libvirt, it's an open source toolkit which enables a common API interaction with the virtualization capabilities of recent versions of Linux (and other OSes).

I would like to take this opportunity to share my ideas as well as get some feedback on some of the key points I see for the creation of common cloud computing reference API or standard.

* Cloud Resource Description
The ability to describe resources is (in my opinion) the most important aspect of any standardization effort. One potential avenue might be to use the Resource Description Framework proposed by the W3C. The Resource Description Framework (RDF) is a family of specifications, originally designed as a metadata data model, which has come to be used as a general method of modeling information through a variety of syntax formats. The RDF metadata model is based upon the idea of making statements about Web resources (or Cloud Resources) in the form of subject-predicate-object expressions, called triples in RDF lingo. This standardized approach could be modified as a primary mechanism for describing cloud resources both locally and remotely.

* Cloud Federation (Cloud 2 Cloud)
The holy grail of cloud computing may very well be the ability to seamlessly bridge both private clouds (datacenters) and remote cloud resources such as EC2 in a secure and efficient manor. To accomplish this a federation standard must be enabled. One of the biggest hurdles to over come in federation is the lack of clear definition to what federation is.

So let me take a stab at defining it.

Cloud federation manages consistency and access controls when two or more independent geographically distinct clouds share either authentication, files, computing resources, command and control or access to storage resources. Cloud federations can be classified into three categories: peer-to-peer, replication, and hierarchical. Peer 2 peer seems to be the most logical first step in creating a federation spec. Protocols like XMPP, P4P and Virtual Distributed Ethernet may make for good starting points.

* Distributed Network Management
The need for a distributed and optimized virtual network is an important aspect in any multi-cloud deployment. One potential direction could be to explore the use of VPN or VDE technologies. My preference would be to use VDE, (Virtual Distributed Ethernet). A quick refresher, a VPN is a way to connect one or more remote computers to a protected network, generally tunnelling the traffic through another network. VDE implements a virtual ethernet in all its aspects, virtual switches, virtual cables. A VDE can also be used to create a VPN.

VDE interconnects real computers running (through a tap interface), virtual machines as well as the other networking interfaces through a common open framework. VDE supports heterogeneous virtual machines running on different hosting computers and could be the ideal starting point. Network shaping and optimization may also play an important role in the ability to bridge two or cloud resources.

Some network optimization aspects may include;

  • Compression - Relies on data patterns that can be represented more efficiently.
  • Caching/Proxy - Relies on human behavior , accessing the same data over and over.
  • Protocol Spoofing - Bundles multiple requests from chatty applications into one.
  • Application Shaping - Controls data usage based on spotting specific patterns in the data and allowing or disallowing specific traffic.
  • Equalizing - Makes assumptions on what needs immediate priority based on the data usage.
  • Connection Limits - Prevents access gridlock in routers and access points due to denial of service or peer to peer.
  • Simple Rate Limits - Prevents one user from getting more than a fixed amount of data.

* Memory Management
When looking at the creation of compute cloud memory tends to be a major factor in the performance of a given virtual environment, whether a virtual machine or some other application component. Cloud memory management will need to involve ways to allocate portions of virtual memory to programs at their request, and freeing it for reuse when no longer needed. This is particularly important in "platform as a service" cloud deployments.

Several key memory management aspects may include;

  • Provide memory space to enable several processes to be executed at the same time
  • Provide a satisfactory level of performance for the system users
  • Protect each program's resources
  • Share (if desired) memory space between processes
  • Make the addressing of memory space as transparent as possible for the programmer.

* Distributed Storage
I've been working on creating a cloud abstraction layer called "cloud raid" as part of our ElasticDrive platform and have been looking at different approaches for our implementation. My initial idea is to connect multiple remote cloud storage services (S3, Nirvanix, CloudFS) for a variety of purposes. During my research the XAM specification began to look like the most suitable candidate. XAM addresses storage interoperability, information assurance (security), storage transparency, long-term records retention and automation for Information Lifecycle Management (ILM)-based practices.

XAM looks to solve key cloud storage problem spots including;

  • Interoperability: Applications can work with any XAM conformant storage system; information can be migrated and shared
  • Compliance: Integrated record retention and disposition metadata
  • ILM Practices: Framework for classification, policy, and implementation
  • Migration: Ability to automate migration process to maintain long-term readability
  • Discovery: Application-independent structured discovery avoids application obsolescence

Potential Future Additions to the API

* I/o
The virtualization of I/O resources is a critical part of enabling a set of emerging cloud deployment models. In large scale cloud deployments a recurring issue has the ability to effectively management I/o resources whether on a machine level or network. One of the problems a lot of users are encountering is that of the "nasty neighbor" or a user who has taken all available system I/o resources.

A common I/o API for sharing, security, performance, and scalability will need to be addressed to help resolve these issues. I've been speaking with several hardware vendors on how we might be able to address this problem. This will most like have to be done at a later point after a first draft has been released.

* Monitoring and System Metrics
One of the best aspects of using cloud technology is the ability to scale applications in tandem to the underlying infrastructure and the demands placed on it. Rather then just scaling on system load, users should have the ability to selectively scale on other metrics such as response time, network throughput or other metrics made available. Having a uniform way to interact with system metrics will enable cloud providers and consumers a common way to scale applications.

Security & Auditability.
In my conversations with several wall street CIO's the questions of both security and cloud transparency with regards to external audits has come up frequently.

My list of requirements is by no means a complete list. Cloud computing encompasses a wide variety of technologies, architectures and deployment models. What I am attempting to do is address the initial pain points whether you are deploying a cloud or just using it. A lot of what I've outlined may be better suited to a reference implementation than a standard, but none the less I thought I'd put these out ideas out for discussion.

-- Update --

Looks like I've forgotten an obvious yet important aspect to my cloud standards. Authentication. Maybe something like OAuth or OpenID could form the basis for this as well. I'll need to do some more thinking on this one.

More Stories By Reuven Cohen

An instigator, part time provocateur, bootstrapper, amateur cloud lexicographer, and purveyor of random thoughts, 140 characters at a time.

Reuven is an early innovator in the cloud computing space as the founder of Enomaly in 2004 (Acquired by Virtustream in February 2012). Enomaly was among the first to develop a self service infrastructure as a service (IaaS) platform (ECP) circa 2005. As well as SpotCloud (2011) the first commodity style cloud computing Spot Market.

Reuven is also the co-creator of CloudCamp (100+ Cities around the Globe) CloudCamp is an unconference where early adopters of Cloud Computing technologies exchange ideas and is the largest of the ‘barcamp’ style of events.

Comments (1) View Comments

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


Most Recent Comments
MiamiWebDesigner 08/25/08 06:25:14 AM EDT

Kudos to the Cloud Crowd for Re-Inventing the Wheel!

One thing 30 years in the IT industry has taught me is that the more things change, the more they stay the same. Another is that the only memory we seem to access is short-term. Yet another is that techno-marketeers rely on that, so they can put labels like "revolutionary" and "innovative" on platforms, products and services that are mere re-inventions of the wheel ... and often poor copies at that.

A good example is all the buzz about "Cloud Computing" in general and "SaaS" (software as a service) in particular:

http://tinyurl.com/6let8x

Both terms are bogus. The only true cloud computing takes place in aircraft. What they're actually referring to by "the cloud" is a large-scale and often remotely located and managed computing platform. We have had those since the dawn of electronic IT. IBM calls them "mainframes":

http://tinyurl.com/5kdhcb

The only innovation offered by today's cloud crowd is actually more of a speculation, i.e. that server farms can deliver the same solid performance as Big Iron. And even that's not original. Anyone remember Datapoint's ARCnet, or DEC's VAXclusters? Whatever happened to those guys, anyway...?

And as for SaaS, selling the sizzle while keeping the steak is a marketing ploy most rightfully accredited to society's oldest profession. Its first application in IT was (and for many still is) known as the "service bureau". And I don't mean the contemporary service bureau (mis)conception labelled "Service 2.0" by a Wikipedia contributor whose historical perspective is apparently constrained to four years:

http://tinyurl.com/5fpb8e

Instead, I mean the computer service bureau industry that spawned ADAPSO (the Association of Data Processing Service Organizations) in 1960, and whose chronology comprises a notable portion of the IEEE's "Annals of the History of Computing":

http://tinyurl.com/5lvjdl

So ... for any of you slide rule-toting, pocket-protected keypunch-card cowboys who may be just coming out of a 40-year coma, let me give you a quick IT update:

1. "Mainframe" is now "Cloud" (with concomitant ethereal substance).

2. "Terminal" is now "Web Browser" (with much cooler games, and infinitely more distractions).

3. "Service Bureau" is now "SaaS" (but app upgrades are just as painful, and custom mods equally elusive).

4. Most IT buzzwords boil down to techno-hyped BS (just as they always have).

Bruce Arnold, Web Design Miami Florida
http://www.PervasivePersuasion.com

@CloudExpo Stories
SYS-CON Events announced today that Conference Guru has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. A valuable conference experience generates new contacts, sales leads, potential strategic partners and potential investors; helps gather competitive intelligence and even provides inspiration for new products and services. Conference Guru works with conference organi...
"MobiDev is a Ukraine-based software development company. We do mobile development, and we're specialists in that. But we do full stack software development for entrepreneurs, for emerging companies, and for enterprise ventures," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
It is ironic, but perhaps not unexpected, that many organizations who want the benefits of using an Agile approach to deliver software use a waterfall approach to adopting Agile practices: they form plans, they set milestones, and they measure progress by how many teams they have engaged. Old habits die hard, but like most waterfall software projects, most waterfall-style Agile adoption efforts fail to produce the results desired. The problem is that to get the results they want, they have to ch...
"We focus on composable infrastructure. Composable infrastructure has been named by companies like Gartner as the evolution of the IT infrastructure where everything is now driven by software," explained Bruno Andrade, CEO and Founder of HTBase, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Wooed by the promise of faster innovation, lower TCO, and greater agility, businesses of every shape and size have embraced the cloud at every layer of the IT stack – from apps to file sharing to infrastructure. The typical organization currently uses more than a dozen sanctioned cloud apps and will shift more than half of all workloads to the cloud by 2018. Such cloud investments have delivered measurable benefits. But they’ve also resulted in some unintended side-effects: complexity and risk. ...
Cloud applications are seeing a deluge of requests to support the exploding advanced analytics market. “Open analytics” is the emerging strategy to deliver that data through an open data access layer, in the cloud, to be directly consumed by external analytics tools and popular programming languages. An increasing number of data engineers and data scientists use a variety of platforms and advanced analytics languages such as SAS, R, Python and Java, as well as frameworks such as Hadoop and Spark...
In 2014, Amazon announced a new form of compute called Lambda. We didn't know it at the time, but this represented a fundamental shift in what we expect from cloud computing. Now, all of the major cloud computing vendors want to take part in this disruptive technology. In his session at 20th Cloud Expo, Doug Vanderweide, an instructor at Linux Academy, discussed why major players like AWS, Microsoft Azure, IBM Bluemix, and Google Cloud Platform are all trying to sidestep VMs and containers wit...
No hype cycles or predictions of zillions of things here. IoT is big. You get it. You know your business and have great ideas for a business transformation strategy. What comes next? Time to make it happen. In his session at @ThingsExpo, Jay Mason, Associate Partner at M&S Consulting, presented a step-by-step plan to develop your technology implementation strategy. He discussed the evaluation of communication standards and IoT messaging protocols, data analytics considerations, edge-to-cloud tec...
New competitors, disruptive technologies, and growing expectations are pushing every business to both adopt and deliver new digital services. This ‘Digital Transformation’ demands rapid delivery and continuous iteration of new competitive services via multiple channels, which in turn demands new service delivery techniques – including DevOps. In this power panel at @DevOpsSummit 20th Cloud Expo, moderated by DevOps Conference Co-Chair Andi Mann, panelists examined how DevOps helps to meet the de...
When growing capacity and power in the data center, the architectural trade-offs between server scale-up vs. scale-out continue to be debated. Both approaches are valid: scale-out adds multiple, smaller servers running in a distributed computing model, while scale-up adds fewer, more powerful servers that are capable of running larger workloads. It’s worth noting that there are additional, unique advantages that scale-up architectures offer. One big advantage is large memory and compute capacity...
"When we talk about cloud without compromise what we're talking about is that when people think about 'I need the flexibility of the cloud' - it's the ability to create applications and run them in a cloud environment that's far more flexible,” explained Matthew Finnie, CTO of Interoute, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities. In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, posited that disruption is inevitable for comp...
SYS-CON Events announced today that Datanami has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datanami is a communication channel dedicated to providing insight, analysis and up-to-the-minute information about emerging trends and solutions in Big Data. The publication sheds light on all cutting-edge technologies including networking, storage and applications, and thei...
"We are a monitoring company. We work with Salesforce, BBC, and quite a few other big logos. We basically provide monitoring for them, structure for their cloud services and we fit into the DevOps world" explained David Gildeh, Co-founder and CEO of Outlyer, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
SYS-CON Events announced today that Silicon India has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Published in Silicon Valley, Silicon India magazine is the premiere platform for CIOs to discuss their innovative enterprise solutions and allows IT vendors to learn about new solutions that can help grow their business.
Join us at Cloud Expo June 6-8 to find out how to securely connect your cloud app to any cloud or on-premises data source – without complex firewall changes. More users are demanding access to on-premises data from their cloud applications. It’s no longer a “nice-to-have” but an important differentiator that drives competitive advantages. It’s the new “must have” in the hybrid era. Users want capabilities that give them a unified view of the data to get closer to customers and grow business. The...
The Internet giants are fully embracing AI. All the services they offer to their customers are aimed at drawing a map of the world with the data they get. The AIs from these companies are used to build disruptive approaches that cannot be used by established enterprises, which are threatened by these disruptions. However, most leaders underestimate the effect this will have on their businesses. In his session at 21st Cloud Expo, Rene Buest, Director Market Research & Technology Evangelism at Ara...
"We do one of the best file systems in the world. We learned how to deal with Big Data many years ago and we implemented this knowledge into our software," explained Jakub Ratajczak, Business Development Manager at MooseFS, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), provided an overview of various initiatives to certify the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldwide re...