Welcome!

@CloudExpo Authors: Liz McMillan, Pat Romanski, Elizabeth White, Nate Vickery, Gopala Krishna Behara

Related Topics: @CloudExpo, Cloud Security, @DevOpsSummit

@CloudExpo: Article

Surviving the Zombie Apocalypse | @CloudExpo #Cloud #Security

Security is one of the most controversial topics in the software industry

It must be, I thought, one of the races most persistent and comforting hallucinations to trust that "it can't happen here" - that one's own time and place is beyond cataclysm.

—John Wyndham

Security is one of the most controversial topics in the software industry. How do you measure security? Is your favorite software fundamentally insecure? Are Docker containers secure?

Dan Walsh, SELinux architect, wrote: "Some people make the mistake of thinking of containers as a better and faster way of running virtual machines. From a security point of view, containers are much weaker." Meanwhile, James Bottomley, Linux Maintainer and former Parallels CTO, wrote: "There's contentions all over the place that containers are not actually as secure as hypervisors. This is not really true. Parallels and Virtuozo, we've been running secure containers for at least 10 years." To add to the mix, Theo de Raadt, OpenBSD project lead, wrote back in 2007: "You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes."

Who is right? If the experts disagree, how can customers evaluate their claims? If all software applications have bugs, aren't they all equally unsafe? To help guide you through, this article introduces basic software security concepts to provide you with a perspective to form an opinion on the subject.

Defining Software Vulnerability
A vulnerability is essentially a bug or an error in software. A vulnerability affects a software interface in a way that allows an attacker to take control over the system.

When you lock your doors and windows in your house, you don't want anybody to get inside unless they have a key. Typically, strangers need to ring the doorbell to request entry; then you decided whether they should be let in. This - in effect - is your interface. But what if a couple of screws in one of the hinges of your back door were loose? With enough strength and persistence, somebody might be able to break it, push the door ajar, and allow themselves in without a key. The loose screws are the vulnerability.

Let's assume that you had noticed the loose screws a while back but couldn't be bothered to fix them. In your physical house, it's likely that no one will exploit that vulnerability, but this is not the case in the hostile internet environment. To keep up with the analogy, just imagine the house is in a rough neighborhood where the house is one of the few last standing structures in a world overrun by zombies. Zombies are crawling around everywhere looking for food. Fortunately, you are safe inside your house, living off of your provisions. You can hear them right now, just outside the walls, scratching and scraping, looking for a way in. All of the sudden, those two loose screws look a bit more scary, don't they?

Software systems have many interfaces. The most important ones separate different privilege domains. Userland applications, such as Firefox, are less privileged than Linux, the operating system kernel. On hypervisor deployments, anything inside a virtual machine is less privileged than the hypervisor. For example, Linux is less privileged than Xen. These interfaces are called surface of attack, because they separate potentially malicious code from the software that is in charge of running the system, such as Linux and Xen. These are the interfaces that malevolent attackers try to exploit. In the house analogy, inside is a higher privilege domain than outside; doors and windows are the surface of attack.

When running containers, the most important attack surface is the interface that separates the Linux kernel from Docker containers. It's usually called syscall interface because syscalls are what programs use to talk to the kernel. Linux offers a set of namespaces to create the illusion that each container is the only one running on the system. Namespaces reduce the scope of the syscalls available to a container, as such they are key to security. Syscalls are exposed to applications as nice little C functions by a core system library named libc. libc makes syscalls to implement its functionalities and provide POSIX compatibility (POSIX stands for "Portable Operating System Interface for Computer Environments", a standard which goes back to the golden era of the large UNIX systems of the '80s). This compatibility allows the same application to run somewhat unmodified on Linux, FreeBSD, OSx and other POSIX systems.

When running virtual machines, the surface of attack is the interface exposed by the hypervisor, which varies depending on the implementation. In the case of KVM, it consists of a set of virtual hardware devices available within the VM. In the case of Xen PV guests, it consists of a set of hypercalls.

Back at the house, you managed to leave unnoticed just in time before zombies stormed the place. You really should have fixed those screws the first time you saw them. Darting from shadow to shadow, you are desperately looking for another secure location while remaining undetected. Up ahead you spot a couple of buildings: a large condo on the left and a small house on the right. What luck! But which one do you choose to take shelter in? The condo has more than a dozen doors and windows, while the house just one door and five windows. As a zombie survivor, you know that the smaller house is easier to defend. It has a smaller... surface of attack.

A large surface of attack is harder to secure than a smaller one. More doors mean more hinges. In software, it is hard to measure the precise size of an interface. But it is undeniable that the syscall interface is large, even with namespaces. It is an order of magnitude larger than any hypervisor interface. This translates into more security vulnerabilities. All software has security vulnerabilities but some programs have more than others. The POSIX interface has more vulnerabilities per year than hypervisors. The last time that somebody made a comparison, Xen PV guests had no privilege escalation vulnerabilities in the previous 12 months, while Linux containers had 9. Unsurprisingly, the difference tends to be proportional to the size difference of the attack surfaces.

You walk up to the porch of the smaller house. You grasp the handle when you notice a noise coming from the inside. Playing it safe, you peek through the window. The living room is crowded with zombies. You step away immediately, heading toward the larger building. This time you have no choice. After a careful sweep, it proves to be empty. It even has a gym. But what now? How can you secure a place with so many openings? You could try to bar and bolt them all. You certainly don't need so many ways in and out. But what if you make a mistake barring a door or a window? One would be enough. What if a slab is not as sturdy as you thought?

Blocking entrances is a difficult job, one that needs to be performed flawlessly. It is the equivalent of using SELinux and Seccomp to reduce or close access to syscalls. Fewer syscalls mean a smaller surface of attack exposed to potentially malicious applications and containers. Some hypervisors support similar technologies, for example, Xen comes with XSM, which can be used to control access to hypercalls. The issue with these techniques is that they are hard to use. A small mistake in their configuration can be paid with a very high price. In the case of Docker and Seccomp, it's easy to block all syscalls which are so uncommon they are effectively unused but specific knowledge of what is running within the container is required. What about an application update that uses a new system call. Can you afford a security configuration mistake to break a running production application?

After a few hard days of work, you managed to block all entrances to the building but one, which has a functioning lockable door. You make regular rounds to inspect the security perimeter. Learning from your past mistake, when you see a loose screw you fix it immediately.

A key aspect of software security is updating vulnerable systems as fast as possible. It is key to shorten the period during which software defenses are at their weakest. Different projects have different disclosure policies. Some of them use "full disclosure": they release all information they have about their vulnerabilities as quickly as possible to the public. The idea is that this way attackers and system administrators stand on equal footing in the fight for security. Other projects use "responsible disclosure": they evaluate and fix vulnerabilities privately. They pre-disclose information about them to a limited number of trusted users. They prepare software updates so that when a vulnerability gets publicly announced, the fix is immediately ready. Even with the best security patching processes, there is still some time when systems are vulnerable.

You look around your new building knowing that you have done everything you could, but you still feel unsafe. The place is just too big. You don't need so many rooms all to yourself. All empty spaces unsettle you in any case. You decide to retreat to a smaller, more defensible area within the building. The gym you saw earlier has everything you need and only one door. It is perfect. You could feel safe there. You constantly check the entrance to the gym, but you also look after the other doors and windows facing outside. It's starting to feel like home.

Setting up two security perimeters is an example of defense is depth. Many of the techniques mentioned before, such as SELinux and Seccomp, can be stacked on top of each other. Another example of defense in depth is running containers inside virtual machines. In this case, POSIX and Linux namespaces are only the first surface of attack exposed to malicious workloads. If an attacker manages to penetrate it, she could damage all the other containers running inside the same virtual machine but would not be able to access anything outside of it. To take over the whole system, an attacker would also have to break through the Xen hypercall interface or the KVM virtual hardware interface, depending on the hypervisor.

Going back to the questions at the beginning of this blog, it should be clearer that vulnerabilities are inevitable but that not all software is equally insecure. In fact, security is not an on or off switch; rather it's a spectrum. It's more productive to talk about security risk. Some software solutions have a greater risk of being broken into compared to others because, on average, they present more vulnerabilities over the same period of time. In addition the process used by software projects to deal with their vulnerabilities and public announcement policies have a direct impact on the vulnerabilities' duration. The differences can be dramatic.

The risk of vulnerabilities can be reduced but can never be eliminated. Decreasing security risk is a difficult business, especially when the surface of attack is large to begin with. Many software interfaces used today were written at a time when performance and convenience were the foremost priorities. Security was only retrofitted into them.

Users should not have to deal with multiple layers of complex techniques to reduce the surface of attack of ill-suited interfaces to acceptable levels. Because there is a probability that your system will have a cataclysmic security event, it is rational to adopt simplicity as a vital software design principle to minimize the risks. After all, with zombies as in software, doors and windows are a threat to your life.

More Stories By Stefano Stabellini

Stefano Stabellini serves as virtualization expert in a new dynamic team at Aporeto. Previously, he was senior principal software engineer at Citrix, leading a group of open source engineers working on the Xen Project. He has been involved in Xen development since 2007, focusing on several different projects, spanning from Qemu to Xen and the Linux kernel. He created libxenlight in November 2009 and started the Xen port to ARM with virtualization extensions with Ian Campbell in 2011. Currently, he maintains Xen support in Qemu and Xen on ARM in Linux and Xen.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
Digital transformation is about embracing digital technologies into a company's culture to better connect with its customers, automate processes, create better tools, enter new markets, etc. Such a transformation requires continuous orchestration across teams and an environment based on open collaboration and daily experiments. In his session at 21st Cloud Expo, Alex Casalboni, Technical (Cloud) Evangelist at Cloud Academy, explored and discussed the most urgent unsolved challenges to achieve f...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
The past few years have brought a sea change in the way applications are architected, developed, and consumed—increasing both the complexity of testing and the business impact of software failures. How can software testing professionals keep pace with modern application delivery, given the trends that impact both architectures (cloud, microservices, and APIs) and processes (DevOps, agile, and continuous delivery)? This is where continuous testing comes in. D
Mobile device usage has increased exponentially during the past several years, as consumers rely on handhelds for everything from news and weather to banking and purchases. What can we expect in the next few years? The way in which we interact with our devices will fundamentally change, as businesses leverage Artificial Intelligence. We already see this taking shape as businesses leverage AI for cost savings and customer responsiveness. This trend will continue, as AI is used for more sophistica...
The “Digital Era” is forcing us to engage with new methods to build, operate and maintain applications. This transformation also implies an evolution to more and more intelligent applications to better engage with the customers, while creating significant market differentiators. In both cases, the cloud has become a key enabler to embrace this digital revolution. So, moving to the cloud is no longer the question; the new questions are HOW and WHEN. To make this equation even more complex, most ...
Blockchain is a shared, secure record of exchange that establishes trust, accountability and transparency across business networks. Supported by the Linux Foundation's open source, open-standards based Hyperledger Project, Blockchain has the potential to improve regulatory compliance, reduce cost as well as advance trade. Are you curious about how Blockchain is built for business? In her session at 21st Cloud Expo, René Bostic, Technical VP of the IBM Cloud Unit in North America, discussed the b...
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone in...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
As you move to the cloud, your network should be efficient, secure, and easy to manage. An enterprise adopting a hybrid or public cloud needs systems and tools that provide: Agility: ability to deliver applications and services faster, even in complex hybrid environments Easier manageability: enable reliable connectivity with complete oversight as the data center network evolves Greater efficiency: eliminate wasted effort while reducing errors and optimize asset utilization Security: imple...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
Companies are harnessing data in ways we once associated with science fiction. Analysts have access to a plethora of visualization and reporting tools, but considering the vast amount of data businesses collect and limitations of CPUs, end users are forced to design their structures and systems with limitations. Until now. As the cloud toolkit to analyze data has evolved, GPUs have stepped in to massively parallel SQL, visualization and machine learning.
The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
Kubernetes is an open source system for automating deployment, scaling, and management of containerized applications. Kubernetes was originally built by Google, leveraging years of experience with managing container workloads, and is now a Cloud Native Compute Foundation (CNCF) project. Kubernetes has been widely adopted by the community, supported on all major public and private cloud providers, and is gaining rapid adoption in enterprises. However, Kubernetes may seem intimidating and complex ...
In his session at 21st Cloud Expo, Michael Burley, a Senior Business Development Executive in IT Services at NetApp, described how NetApp designed a three-year program of work to migrate 25PB of a major telco's enterprise data to a new STaaS platform, and then secured a long-term contract to manage and operate the platform. This significant program blended the best of NetApp’s solutions and services capabilities to enable this telco’s successful adoption of private cloud storage and launching ...
In his general session at 21st Cloud Expo, Greg Dumas, Calligo’s Vice President and G.M. of US operations, discussed the new Global Data Protection Regulation and how Calligo can help business stay compliant in digitally globalized world. Greg Dumas is Calligo's Vice President and G.M. of US operations. Calligo is an established service provider that provides an innovative platform for trusted cloud solutions. Calligo’s customers are typically most concerned about GDPR compliance, application p...
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It’s clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. Tha...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...