Welcome!

@CloudExpo Authors: Yeshim Deniz, Liz McMillan, Elizabeth White, Pat Romanski, Shelly Palmer

Related Topics: @CloudExpo, Cloud Security, @DevOpsSummit

@CloudExpo: Article

Surviving the Zombie Apocalypse | @CloudExpo #Cloud #Security

Security is one of the most controversial topics in the software industry

It must be, I thought, one of the races most persistent and comforting hallucinations to trust that "it can't happen here" - that one's own time and place is beyond cataclysm.

—John Wyndham

Security is one of the most controversial topics in the software industry. How do you measure security? Is your favorite software fundamentally insecure? Are Docker containers secure?

Dan Walsh, SELinux architect, wrote: "Some people make the mistake of thinking of containers as a better and faster way of running virtual machines. From a security point of view, containers are much weaker." Meanwhile, James Bottomley, Linux Maintainer and former Parallels CTO, wrote: "There's contentions all over the place that containers are not actually as secure as hypervisors. This is not really true. Parallels and Virtuozo, we've been running secure containers for at least 10 years." To add to the mix, Theo de Raadt, OpenBSD project lead, wrote back in 2007: "You are absolutely deluded, if not stupid, if you think that a worldwide collection of software engineers who can't write operating systems or applications without security holes, can then turn around and suddenly write virtualization layers without security holes."

Who is right? If the experts disagree, how can customers evaluate their claims? If all software applications have bugs, aren't they all equally unsafe? To help guide you through, this article introduces basic software security concepts to provide you with a perspective to form an opinion on the subject.

Defining Software Vulnerability
A vulnerability is essentially a bug or an error in software. A vulnerability affects a software interface in a way that allows an attacker to take control over the system.

When you lock your doors and windows in your house, you don't want anybody to get inside unless they have a key. Typically, strangers need to ring the doorbell to request entry; then you decided whether they should be let in. This - in effect - is your interface. But what if a couple of screws in one of the hinges of your back door were loose? With enough strength and persistence, somebody might be able to break it, push the door ajar, and allow themselves in without a key. The loose screws are the vulnerability.

Let's assume that you had noticed the loose screws a while back but couldn't be bothered to fix them. In your physical house, it's likely that no one will exploit that vulnerability, but this is not the case in the hostile internet environment. To keep up with the analogy, just imagine the house is in a rough neighborhood where the house is one of the few last standing structures in a world overrun by zombies. Zombies are crawling around everywhere looking for food. Fortunately, you are safe inside your house, living off of your provisions. You can hear them right now, just outside the walls, scratching and scraping, looking for a way in. All of the sudden, those two loose screws look a bit more scary, don't they?

Software systems have many interfaces. The most important ones separate different privilege domains. Userland applications, such as Firefox, are less privileged than Linux, the operating system kernel. On hypervisor deployments, anything inside a virtual machine is less privileged than the hypervisor. For example, Linux is less privileged than Xen. These interfaces are called surface of attack, because they separate potentially malicious code from the software that is in charge of running the system, such as Linux and Xen. These are the interfaces that malevolent attackers try to exploit. In the house analogy, inside is a higher privilege domain than outside; doors and windows are the surface of attack.

When running containers, the most important attack surface is the interface that separates the Linux kernel from Docker containers. It's usually called syscall interface because syscalls are what programs use to talk to the kernel. Linux offers a set of namespaces to create the illusion that each container is the only one running on the system. Namespaces reduce the scope of the syscalls available to a container, as such they are key to security. Syscalls are exposed to applications as nice little C functions by a core system library named libc. libc makes syscalls to implement its functionalities and provide POSIX compatibility (POSIX stands for "Portable Operating System Interface for Computer Environments", a standard which goes back to the golden era of the large UNIX systems of the '80s). This compatibility allows the same application to run somewhat unmodified on Linux, FreeBSD, OSx and other POSIX systems.

When running virtual machines, the surface of attack is the interface exposed by the hypervisor, which varies depending on the implementation. In the case of KVM, it consists of a set of virtual hardware devices available within the VM. In the case of Xen PV guests, it consists of a set of hypercalls.

Back at the house, you managed to leave unnoticed just in time before zombies stormed the place. You really should have fixed those screws the first time you saw them. Darting from shadow to shadow, you are desperately looking for another secure location while remaining undetected. Up ahead you spot a couple of buildings: a large condo on the left and a small house on the right. What luck! But which one do you choose to take shelter in? The condo has more than a dozen doors and windows, while the house just one door and five windows. As a zombie survivor, you know that the smaller house is easier to defend. It has a smaller... surface of attack.

A large surface of attack is harder to secure than a smaller one. More doors mean more hinges. In software, it is hard to measure the precise size of an interface. But it is undeniable that the syscall interface is large, even with namespaces. It is an order of magnitude larger than any hypervisor interface. This translates into more security vulnerabilities. All software has security vulnerabilities but some programs have more than others. The POSIX interface has more vulnerabilities per year than hypervisors. The last time that somebody made a comparison, Xen PV guests had no privilege escalation vulnerabilities in the previous 12 months, while Linux containers had 9. Unsurprisingly, the difference tends to be proportional to the size difference of the attack surfaces.

You walk up to the porch of the smaller house. You grasp the handle when you notice a noise coming from the inside. Playing it safe, you peek through the window. The living room is crowded with zombies. You step away immediately, heading toward the larger building. This time you have no choice. After a careful sweep, it proves to be empty. It even has a gym. But what now? How can you secure a place with so many openings? You could try to bar and bolt them all. You certainly don't need so many ways in and out. But what if you make a mistake barring a door or a window? One would be enough. What if a slab is not as sturdy as you thought?

Blocking entrances is a difficult job, one that needs to be performed flawlessly. It is the equivalent of using SELinux and Seccomp to reduce or close access to syscalls. Fewer syscalls mean a smaller surface of attack exposed to potentially malicious applications and containers. Some hypervisors support similar technologies, for example, Xen comes with XSM, which can be used to control access to hypercalls. The issue with these techniques is that they are hard to use. A small mistake in their configuration can be paid with a very high price. In the case of Docker and Seccomp, it's easy to block all syscalls which are so uncommon they are effectively unused but specific knowledge of what is running within the container is required. What about an application update that uses a new system call. Can you afford a security configuration mistake to break a running production application?

After a few hard days of work, you managed to block all entrances to the building but one, which has a functioning lockable door. You make regular rounds to inspect the security perimeter. Learning from your past mistake, when you see a loose screw you fix it immediately.

A key aspect of software security is updating vulnerable systems as fast as possible. It is key to shorten the period during which software defenses are at their weakest. Different projects have different disclosure policies. Some of them use "full disclosure": they release all information they have about their vulnerabilities as quickly as possible to the public. The idea is that this way attackers and system administrators stand on equal footing in the fight for security. Other projects use "responsible disclosure": they evaluate and fix vulnerabilities privately. They pre-disclose information about them to a limited number of trusted users. They prepare software updates so that when a vulnerability gets publicly announced, the fix is immediately ready. Even with the best security patching processes, there is still some time when systems are vulnerable.

You look around your new building knowing that you have done everything you could, but you still feel unsafe. The place is just too big. You don't need so many rooms all to yourself. All empty spaces unsettle you in any case. You decide to retreat to a smaller, more defensible area within the building. The gym you saw earlier has everything you need and only one door. It is perfect. You could feel safe there. You constantly check the entrance to the gym, but you also look after the other doors and windows facing outside. It's starting to feel like home.

Setting up two security perimeters is an example of defense is depth. Many of the techniques mentioned before, such as SELinux and Seccomp, can be stacked on top of each other. Another example of defense in depth is running containers inside virtual machines. In this case, POSIX and Linux namespaces are only the first surface of attack exposed to malicious workloads. If an attacker manages to penetrate it, she could damage all the other containers running inside the same virtual machine but would not be able to access anything outside of it. To take over the whole system, an attacker would also have to break through the Xen hypercall interface or the KVM virtual hardware interface, depending on the hypervisor.

Going back to the questions at the beginning of this blog, it should be clearer that vulnerabilities are inevitable but that not all software is equally insecure. In fact, security is not an on or off switch; rather it's a spectrum. It's more productive to talk about security risk. Some software solutions have a greater risk of being broken into compared to others because, on average, they present more vulnerabilities over the same period of time. In addition the process used by software projects to deal with their vulnerabilities and public announcement policies have a direct impact on the vulnerabilities' duration. The differences can be dramatic.

The risk of vulnerabilities can be reduced but can never be eliminated. Decreasing security risk is a difficult business, especially when the surface of attack is large to begin with. Many software interfaces used today were written at a time when performance and convenience were the foremost priorities. Security was only retrofitted into them.

Users should not have to deal with multiple layers of complex techniques to reduce the surface of attack of ill-suited interfaces to acceptable levels. Because there is a probability that your system will have a cataclysmic security event, it is rational to adopt simplicity as a vital software design principle to minimize the risks. After all, with zombies as in software, doors and windows are a threat to your life.

More Stories By Stefano Stabellini

Stefano Stabellini serves as virtualization expert in a new dynamic team at Aporeto. Previously, he was senior principal software engineer at Citrix, leading a group of open source engineers working on the Xen Project. He has been involved in Xen development since 2007, focusing on several different projects, spanning from Qemu to Xen and the Linux kernel. He created libxenlight in November 2009 and started the Xen port to ARM with virtualization extensions with Ian Campbell in 2011. Currently, he maintains Xen support in Qemu and Xen on ARM in Linux and Xen.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
SYS-CON Events announced today that SourceForge has been named “Media Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. SourceForge is the largest, most trusted destination for Open Source Software development, collaboration, discovery and download on the web serving over 32 million viewers, 150 million downloads and over 460,000 active development projects each and every month.
"NetApp's vision is how we help organizations manage data - delivering the right data in the right place, in the right time, to the people who need it, and doing it agnostic to what the platform is," explained Josh Atwell, Developer Advocate for NetApp, in this SYS-CON.tv interview at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
What You Need to Know You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technolog...
SYS-CON Events announced today that DXWorldExpo has been named “Global Sponsor” of SYS-CON's 21st International Cloud Expo, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Digital Transformation is the key issue driving the global enterprise IT business. Digital Transformation is most prominent among Global 2000 enterprises and government institutions.
One of the biggest challenges with adopting a DevOps mentality is: new applications are easily adapted to cloud-native, microservice-based, or containerized architectures - they can be built for them - but old applications need complex refactoring. On the other hand, these new technologies can require relearning or adapting new, oftentimes more complex, methodologies and tools to be ready for production. In his general session at @DevOpsSummit at 20th Cloud Expo, Chris Brown, Solutions Marketi...
SYS-CON Events announced today that Nihon Micron will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Nihon Micron Co., Ltd. strives for technological innovation to establish high-density, high-precision processing technology for providing printed circuit board and metal mount RFID tags used for communication devices. For more inf...
SYS-CON Events announced today that Ryobi Systems will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Ryobi Systems Co., Ltd., as an information service company, specialized in business support for local governments and medical industry. We are challenging to achive the precision farming with AI. For more information, visit http:...
SYS-CON Events announced today that mruby Forum will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. mruby is the lightweight implementation of the Ruby language. We introduce mruby and the mruby IoT framework that enhances development productivity. For more information, visit http://forum.mruby.org/.
SYS-CON Events announced today that Mobile Create USA will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Mobile Create USA Inc. is an MVNO-based business model that uses portable communication devices and cellular-based infrastructure in the development, sales, operation and mobile communications systems incorporating GPS capabi...
SYS-CON Events announced today that Daiya Industry will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Daiya Industry specializes in orthotic support systems and assistive devices with pneumatic artificial muscles in order to contribute to an extended healthy life expectancy. For more information, please visit https://www.daiyak...
SYS-CON Events announced today that Fusic will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Fusic Co. provides mocks as virtual IoT devices. You can customize mocks, and get any amount of data at any time in your test. For more information, visit https://fusic.co.jp/english/.
SYS-CON Events announced today that Keisoku Research Consultant Co. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Keisoku Research Consultant, Co. offers research and consulting in a wide range of civil engineering-related fields from information construction to preservation of cultural properties. For more information, vi...
SYS-CON Events announced today that MIRAI Inc. will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. MIRAI Inc. are IT consultants from the public sector whose mission is to solve social issues by technology and innovation and to create a meaningful future for people.
SYS-CON Events announced today that Interface Corporation will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Interface Corporation is a company developing, manufacturing and marketing high quality and wide variety of industrial computers and interface modules such as PCIs and PCI express. For more information, visit http://www.i...
Today companies are looking to achieve cloud-first digital agility to reduce time-to-market, optimize utilization of resources, and rapidly deliver disruptive business solutions. However, leveraging the benefits of cloud deployments can be complicated for companies with extensive legacy computing environments. In his session at 21st Cloud Expo, Craig Sproule, founder and CEO of Metavine, will outline the challenges enterprises face in migrating legacy solutions to the cloud. He will also prese...
SYS-CON Events announced today that Enroute Lab will exhibit at the Japan External Trade Organization (JETRO) Pavilion at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Enroute Lab is an industrial design, research and development company of unmanned robotic vehicle system. For more information, please visit http://elab.co.jp/.
Elon Musk is among the notable industry figures who worries about the power of AI to destroy rather than help society. Mark Zuckerberg, on the other hand, embraces all that is going on. AI is most powerful when deployed across the vast networks being built for Internets of Things in the manufacturing, transportation and logistics, retail, healthcare, government and other sectors. Is AI transforming IoT for the good or the bad? Do we need to worry about its potential destructive power? Or will we...
In his session at @ThingsExpo, Greg Gorman is the Director, IoT Developer Ecosystem, Watson IoT, will provide a short tutorial on Node-RED, a Node.js-based programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using a wide range of nodes in the palette that can be deployed to its runtime in a single-click. There is a large library of contributed nodes that help so...
With major technology companies and startups seriously embracing Cloud strategies, now is the perfect time to attend 21st Cloud Expo October 31 - November 2, 2017, at the Santa Clara Convention Center, CA, and June 12-14, 2018, at the Javits Center in New York City, NY, and learn what is going on, contribute to the discussions, and ensure that your enterprise is on the right path to Digital Transformation.
DevOps at Cloud Expo – being held October 31 - November 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA – announces that its Call for Papers is open. Born out of proven success in agile development, cloud computing, and process automation, DevOps is a macro trend you cannot afford to miss. From showcase success stories from early adopters and web-scale businesses, DevOps is expanding to organizations of all sizes, including the world's largest enterprises – and delivering real r...