Welcome!

@CloudExpo Authors: Kevin Benedict, Kevin Jackson, Liz McMillan, Elizabeth White, Yeshim Deniz

Related Topics: @CloudExpo, Cloud Security, @DevOpsSummit

@CloudExpo: Article

Surviving the Zombie Apocalypse | @CloudExpo #Cloud #Security

Security is one of the most controversial topics in the software industry

It must be, I thought, one of the races most persistent and comforting hallucinations to trust that "it can't happen here" - that one's own time and place is beyond cataclysm.

—John Wyndham

Security is one of the most controversial topics in the software industry. How do you measure security? Is your favorite software fundamentally insecure? Are Docker containers secure?

Dan Walsh, SELinux architect, wrote: "Some people make the mistake of thinking of containers as a better and faster way of running virtual machines. From a security point of view, containers are much weaker." Meanwhile, James Bottomley, Linux Maintainer and former Parallels CTO, wrote: "There's contentions all over the place that containers are not actually as secure as hypervisors. This is not really true. Parallels and Virtuozo, we've been running secure containers for at least 10 years." To add to the mix, Theo de Raadt, OpenBSD project lead, wrote back in 2007: "You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes."

Who is right? If the experts disagree, how can customers evaluate their claims? If all software applications have bugs, aren't they all equally unsafe? To help guide you through, this article introduces basic software security concepts to provide you with a perspective to form an opinion on the subject.

Defining Software Vulnerability
A vulnerability is essentially a bug or an error in software. A vulnerability affects a software interface in a way that allows an attacker to take control over the system.

When you lock your doors and windows in your house, you don't want anybody to get inside unless they have a key. Typically, strangers need to ring the doorbell to request entry; then you decided whether they should be let in. This - in effect - is your interface. But what if a couple of screws in one of the hinges of your back door were loose? With enough strength and persistence, somebody might be able to break it, push the door ajar, and allow themselves in without a key. The loose screws are the vulnerability.

Let's assume that you had noticed the loose screws a while back but couldn't be bothered to fix them. In your physical house, it's likely that no one will exploit that vulnerability, but this is not the case in the hostile internet environment. To keep up with the analogy, just imagine the house is in a rough neighborhood where the house is one of the few last standing structures in a world overrun by zombies. Zombies are crawling around everywhere looking for food. Fortunately, you are safe inside your house, living off of your provisions. You can hear them right now, just outside the walls, scratching and scraping, looking for a way in. All of the sudden, those two loose screws look a bit more scary, don't they?

Software systems have many interfaces. The most important ones separate different privilege domains. Userland applications, such as Firefox, are less privileged than Linux, the operating system kernel. On hypervisor deployments, anything inside a virtual machine is less privileged than the hypervisor. For example, Linux is less privileged than Xen. These interfaces are called surface of attack, because they separate potentially malicious code from the software that is in charge of running the system, such as Linux and Xen. These are the interfaces that malevolent attackers try to exploit. In the house analogy, inside is a higher privilege domain than outside; doors and windows are the surface of attack.

When running containers, the most important attack surface is the interface that separates the Linux kernel from Docker containers. It's usually called syscall interface because syscalls are what programs use to talk to the kernel. Linux offers a set of namespaces to create the illusion that each container is the only one running on the system. Namespaces reduce the scope of the syscalls available to a container, as such they are key to security. Syscalls are exposed to applications as nice little C functions by a core system library named libc. libc makes syscalls to implement its functionalities and provide POSIX compatibility (POSIX stands for "Portable Operating System Interface for Computer Environments", a standard which goes back to the golden era of the large UNIX systems of the '80s). This compatibility allows the same application to run somewhat unmodified on Linux, FreeBSD, OSx and other POSIX systems.

When running virtual machines, the surface of attack is the interface exposed by the hypervisor, which varies depending on the implementation. In the case of KVM, it consists of a set of virtual hardware devices available within the VM. In the case of Xen PV guests, it consists of a set of hypercalls.

Back at the house, you managed to leave unnoticed just in time before zombies stormed the place. You really should have fixed those screws the first time you saw them. Darting from shadow to shadow, you are desperately looking for another secure location while remaining undetected. Up ahead you spot a couple of buildings: a large condo on the left and a small house on the right. What luck! But which one do you choose to take shelter in? The condo has more than a dozen doors and windows, while the house just one door and five windows. As a zombie survivor, you know that the smaller house is easier to defend. It has a smaller... surface of attack.

A large surface of attack is harder to secure than a smaller one. More doors mean more hinges. In software, it is hard to measure the precise size of an interface. But it is undeniable that the syscall interface is large, even with namespaces. It is an order of magnitude larger than any hypervisor interface. This translates into more security vulnerabilities. All software has security vulnerabilities but some programs have more than others. The POSIX interface has more vulnerabilities per year than hypervisors. The last time that somebody made a comparison, Xen PV guests had no privilege escalation vulnerabilities in the previous 12 months, while Linux containers had 9. Unsurprisingly, the difference tends to be proportional to the size difference of the attack surfaces.

You walk up to the porch of the smaller house. You grasp the handle when you notice a noise coming from the inside. Playing it safe, you peek through the window. The living room is crowded with zombies. You step away immediately, heading toward the larger building. This time you have no choice. After a careful sweep, it proves to be empty. It even has a gym. But what now? How can you secure a place with so many openings? You could try to bar and bolt them all. You certainly don't need so many ways in and out. But what if you make a mistake barring a door or a window? One would be enough. What if a slab is not as sturdy as you thought?

Blocking entrances is a difficult job, one that needs to be performed flawlessly. It is the equivalent of using SELinux and Seccomp to reduce or close access to syscalls. Fewer syscalls mean a smaller surface of attack exposed to potentially malicious applications and containers. Some hypervisors support similar technologies, for example, Xen comes with XSM, which can be used to control access to hypercalls. The issue with these techniques is that they are hard to use. A small mistake in their configuration can be paid with a very high price. In the case of Docker and Seccomp, it's easy to block all syscalls which are so uncommon they are effectively unused but specific knowledge of what is running within the container is required. What about an application update that uses a new system call. Can you afford a security configuration mistake to break a running production application?

After a few hard days of work, you managed to block all entrances to the building but one, which has a functioning lockable door. You make regular rounds to inspect the security perimeter. Learning from your past mistake, when you see a loose screw you fix it immediately.

A key aspect of software security is updating vulnerable systems as fast as possible. It is key to shorten the period during which software defenses are at their weakest. Different projects have different disclosure policies. Some of them use "full disclosure": they release all information they have about their vulnerabilities as quickly as possible to the public. The idea is that this way attackers and system administrators stand on equal footing in the fight for security. Other projects use "responsible disclosure": they evaluate and fix vulnerabilities privately. They pre-disclose information about them to a limited number of trusted users. They prepare software updates so that when a vulnerability gets publicly announced, the fix is immediately ready. Even with the best security patching processes, there is still some time when systems are vulnerable.

You look around your new building knowing that you have done everything you could, but you still feel unsafe. The place is just too big. You don't need so many rooms all to yourself. All empty spaces unsettle you in any case. You decide to retreat to a smaller, more defensible area within the building. The gym you saw earlier has everything you need and only one door. It is perfect. You could feel safe there. You constantly check the entrance to the gym, but you also look after the other doors and windows facing outside. It's starting to feel like home.

Setting up two security perimeters is an example of defense is depth. Many of the techniques mentioned before, such as SELinux and Seccomp, can be stacked on top of each other. Another example of defense in depth is running containers inside virtual machines. In this case, POSIX and Linux namespaces are only the first surface of attack exposed to malicious workloads. If an attacker manages to penetrate it, she could damage all the other containers running inside the same virtual machine but would not be able to access anything outside of it. To take over the whole system, an attacker would also have to break through the Xen hypercall interface or the KVM virtual hardware interface, depending on the hypervisor.

Going back to the questions at the beginning of this blog, it should be clearer that vulnerabilities are inevitable but that not all software is equally insecure. In fact, security is not an on or off switch; rather it's a spectrum. It's more productive to talk about security risk. Some software solutions have a greater risk of being broken into compared to others because, on average, they present more vulnerabilities over the same period of time. In addition the process used by software projects to deal with their vulnerabilities and public announcement policies have a direct impact on the vulnerabilities' duration. The differences can be dramatic.

The risk of vulnerabilities can be reduced but can never be eliminated. Decreasing security risk is a difficult business, especially when the surface of attack is large to begin with. Many software interfaces used today were written at a time when performance and convenience were the foremost priorities. Security was only retrofitted into them.

Users should not have to deal with multiple layers of complex techniques to reduce the surface of attack of ill-suited interfaces to acceptable levels. Because there is a probability that your system will have a cataclysmic security event, it is rational to adopt simplicity as a vital software design principle to minimize the risks. After all, with zombies as in software, doors and windows are a threat to your life.

More Stories By Stefano Stabellini

Stefano Stabellini serves as virtualization expert in a new dynamic team at Aporeto. Previously, he was senior principal software engineer at Citrix, leading a group of open source engineers working on the Xen Project. He has been involved in Xen development since 2007, focusing on several different projects, spanning from Qemu to Xen and the Linux kernel. He created libxenlight in November 2009 and started the Xen port to ARM with virtualization extensions with Ian Campbell in 2011. Currently, he maintains Xen support in Qemu and Xen on ARM in Linux and Xen.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
DevOps at Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The widespread success of cloud computing is driving the DevOps revolution in enterprise IT. Now as never before, development teams must communicate and collaborate in a dynamic, 24/7/365 environment. There is no time to w...
SYS-CON Events announced today that Massive Networks will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Massive Networks mission is simple. To help your business operate seamlessly with fast, reliable, and secure internet and network solutions. Improve your customer's experience with outstanding connections to your cloud.
"We want to show that our solution is far less expensive with a much better total cost of ownership so we announced several key features. One is called geo-distributed erasure coding, another is support for KVM and we introduced a new capability called Multi-Part," explained Tim Desai, Senior Product Marketing Manager at Hitachi Data Systems, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability and development velocity challenges. In his session at 21st Cloud Expo, Ryland Degnan, a Senior Software Engineer on the Netflix Edge Platform team, will discuss how by leveraging a reactive stream-based protocol,...
FinTechs use the cloud to operate at the speed and scale of digital financial activity, but are often hindered by the complexity of managing security and compliance in the cloud. In his session at 20th Cloud Expo, Sesh Murthy, co-founder and CTO of Cloud Raxak, showed how proactive and automated cloud security enables FinTechs to leverage the cloud to achieve their business goals. Through business-driven cloud security, FinTechs can speed time-to-market, diminish risk and costs, maintain continu...
DX World EXPO, LLC., a Lighthouse Point, Florida-based startup trade show producer and the creator of "DXWorldEXPO® - Digital Transformation Conference & Expo" has announced its executive management team. The team is headed by Levent Selamoglu, who has been named CEO. "Now is the time for a truly global DX event, to bring together the leading minds from the technology world in a conversation about Digital Transformation," he said in making the announcement.
In his session at 20th Cloud Expo, Mike Johnston, an infrastructure engineer at Supergiant.io, discussed how to use Kubernetes to set up a SaaS infrastructure for your business. Mike Johnston is an infrastructure engineer at Supergiant.io with over 12 years of experience designing, deploying, and maintaining server and workstation infrastructure at all scales. He has experience with brick and mortar data centers as well as cloud providers like Digital Ocean, Amazon Web Services, and Rackspace. H...
Internet of @ThingsExpo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, is co-located with 21st Cloud Expo and will feature technical sessions from a rock star conference faculty and the leading industry players in the world. The Internet of Things (IoT) is the most profound change in personal and enterprise IT since the creation of the Worldwide Web more than 20 years ago. All major researchers estimate there will be tens of billions devic...
"The Striim platform is a full end-to-end streaming integration and analytics platform that is middleware that covers a lot of different use cases," explained Steve Wilkes, Founder and CTO at Striim, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Everything run by electricity will eventually be connected to the Internet. Get ahead of the Internet of Things revolution and join Akvelon expert and IoT industry leader, Sergey Grebnov, in his session at @ThingsExpo, for an educational dive into the world of managing your home, workplace and all the devices they contain with the power of machine-based AI and intelligent Bot services for a completely streamlined experience.
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, will examine the regulations and provide insight on how it affects technology, challenges the established rules and will usher in new levels of diligence...
SYS-CON Events announced today that Calligo, an innovative cloud service provider offering mid-sized companies the highest levels of data privacy and security, has been named "Bronze Sponsor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalised support service from its globally located cloud plat...
SYS-CON Events announced today that Calligo has been named “Bronze Sponsor” of SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Calligo is an innovative cloud service provider offering mid-sized companies the highest levels of data privacy. Calligo offers unparalleled application performance guarantees, commercial flexibility and a personalized support service from its globally located cloud platfor...
What sort of WebRTC based applications can we expect to see over the next year and beyond? One way to predict development trends is to see what sorts of applications startups are building. In his session at @ThingsExpo, Arin Sime, founder of WebRTC.ventures, discussed the current and likely future trends in WebRTC application development based on real requests for custom applications from real customers, as well as other public sources of information.
SYS-CON Events announced today that SkyScale will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SkyScale is a world-class provider of cloud-based, ultra-fast multi-GPU hardware platforms for lease to customers desiring the fastest performance available as a service anywhere in the world. SkyScale builds, configures, and manages dedicated systems strategically located in maximum-securit...
21st International Cloud Expo, taking place October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA, will feature technical sessions from a rock star conference faculty and the leading industry players in the world. Cloud computing is now being embraced by a majority of enterprises of all sizes. Yesterday's debate about public vs. private has transformed into the reality of hybrid cloud: a recent survey shows that 74% of enterprises have a hybrid cloud strategy. Me...
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
SYS-CON Events announced today that Datera, that offers a radically new data management architecture, has been named "Exhibitor" of SYS-CON's 21st International Cloud Expo ®, which will take place on Oct 31 - Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Datera is transforming the traditional datacenter model through modern cloud simplicity. The technology industry is at another major inflection point. The rise of mobile, the Internet of Things, data storage and Big...
"With Digital Experience Monitoring what used to be a simple visit to a web page has exploded into app on phones, data from social media feeds, competitive benchmarking - these are all components that are only available because of some type of digital asset," explained Leo Vasiliou, Director of Web Performance Engineering at Catchpoint Systems, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
"Outscale was founded in 2010, is based in France, is a strategic partner to Dassault Systémes and has done quite a bit of work with divisions of Dassault," explained Jackie Funk, Digital Marketing exec at Outscale, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.