Welcome!

@CloudExpo Authors: Liz McMillan, Yeshim Deniz, PagerDuty Blog, XebiaLabs Blog, Pat Romanski

Related Topics: @CloudExpo, Cloud Security, @BigDataExpo

@CloudExpo: Article

The Blind Spot of the Security Industry | @CloudExpo #Cloud #Security

The SSH protocol is not well understood by the majority of people, much like the subprime loan market

The Blind Spot of the Security Industry: SSH User Keys

More than 95 percent of the world's enterprises rely on SSH user keys to provide administrators and developers an effective means of gaining encrypted access to critical infrastructure: operating systems, applications, payment processing systems, databases, human resource and financial systems, routers, switches, firewalls and other network devices. It is a lifeline of traffic flow within our data centers, our cloud environments and how our third-party vendors and supply chain access our environments. It has done its job quietly and efficiently over the last two decades. Unfortunately, the access that SSH has been providing, in particular the access SSH user keys provide, has gone largely unmanaged - to an epic degree.

There's an unfortunate parallel here to a recent Oscar-winning film based on the events that took down Wall Street in 2008: The Big Short. Iconoclastic investor Michael Burry discovered that the subprime home loans market was headed for default. He bet more than $1 billion against the housing industry - an unprecedented move. Sadly, for everyone but him and his investors, Burry was right.

Just as Wall Street had the blind spot when it came to subprime loans, the security industry has a blind spot regarding SSH protocol and SSH user keys. In fact, there are three parallels between the movie's plot and the current state of SSH management:

  • Lack of understanding about the problem and its severity
  • How challenging oversight of the problem is
  • How significant the impact or consequences are of not addressing the problem

A Real and Far-Reaching Problem
The SSH protocol is not well understood by the majority of people, much like the subprime loan market. In fact, very few understand in sufficient detail the underlying critical access it is providing to our most important infrastructure. In this case, however, ignorance is not bliss.

For instance, in a typical financial enterprise with 20,000 Unix/Linux servers, one can expect to find up to 4 million SSH user keys providing interactive and machine-to-machine-based access. In many cases, 10 to 20 percent of these keys provide root-level access and cannot be associated to an owner within the enterprise. Root-level access is the highest level of privilege at an operating system level. It is not just a compliance and risk issue. It is an issue of resilience that has the opportunity to impact the potential downtime of critical services within operations.

Oversight and Governance Are Difficult or Non-Existent
Chiefly because SSH has long been seen as an encryption protocol rather than a means of access, this problem has gone unnoticed for a long time. As a result, it has not been considered as a part of the access governance processes and frameworks. In fact, up until October 2015, there were no NIST guidelines related to the best practices associated with SSH user key-based access.

HIPAA, SOX, PCI and other regulatory guidelines do mention access controls, such as least privilege and segregation of duties, but none of them specifically address SSH user keys as a form of access that needs to be controlled. Is this due to a lack of understanding of SSH or an unwillingness to open Pandora's Box?

In terms of the lack oversight and governance of SSH user key-based access, there are three technical aspects that must be considered. First, SSH user keys are the only form of access a user can provision themselves without oversight or control. Unlike passwords, an administrator, developer or third-party vendor can provision themselves with access to your critical infrastructure or automate a process or file transfer. This process of provisioning, de-provisioning and recertification of SSH user key-based access is infrequently if never addressed in the IAM frameworks of enterprises.

The second consideration is that SSH user keys don't have expiration dates, in contrast to certificates. This means that SSH user key-based access continues to exist, even after an application is decommissioned or a user leaves the company.

And the third consideration: an SSH user key does not need to be associated to a user account. This means a key will not necessarily establish the identity of the user associated with it. When an SSH user key is generated, the identity that associated is not connected.

Taken together, here's what these considerations mean. A user can provision SSH user key-based access themselves without oversight to my most critical infrastructure. This key-based access does not expire. It's not clear which identity it's associated with, and there are millions of these across an enterprise, which no one has an inventory of, and they are providing access to the most critical systems.

The Ramifications Are Significant
There is an additional cause for concern in addition to the lack of understanding of the technical aspects of the SSH protocol and the lack of regulatory oversight and governance of SSH user key-based access. SSH is the malicious actor's (either a rogue insider or hacker) tool of choice, and it is the primary means by which they move laterally within an enterprise to gain access to new assets and further elevate privilege, as well as how they exfiltrate data and assets from an organization. In fact, LightCyber's Cyberweapons 2016 Report indicates that in over 50 percent of the cases, the SSH protocol is the tool of choice to gain this lateral movement across the assets of an enterprise.

It's also important to factor in the reality that bad actors are aware that SSH user keys are not being managed. As a result, they go after private keys to gain access to assets. From there, they will often create new key pairs, which will generate them access to the outside directly or move those assets automatically to servers in the cloud. This is all achieved using something called "port forwarding" via "SSH tunneling," which would allow the malicious actor to extract this data via the encryption itself, rendering firewalls ineffective.

This kind of activity takes place right within our defensive structures. Why? Because we don't know who owns the keys to attain that encrypted access and because we are not able to look inside the encryption of those sessions. The potential impact is clear. We lose our data - or worse, our customer's data. We lose our intellectual property. We lose our reputation, we lose our brand and, in turn, we lose revenue and shareholder value.

The financial implications are clear here, but there's another implication that's at least as serious: operational resilience. The potential downtime in critical systems, should SSH user key-based access to those systems be compromised, is a concern we need to seriously consider. Our disaster recovery systems, which are failovers to our production systems, often share identical SSH user keys that ensure the processes of those systems fail over in an identical manner. This means that any compromised keys in production environments can also equally affect the failover to disaster recovery systems.

Clear Parallels Pointing to Danger
Let's circle back to the similarities between what's going on with SSH user key management and the events leading up to the housing crisis. The parallels are uncanny. There's a lack of market understanding around the power of the SSH protocol and the criticality of the access it provides to our infrastructures. There's a gap in regulatory and governing oversight of how this SSH user key-based access should be monitored, provisioned, de-provisioned and recertified. There's an encrypted protocol, which malicious actors use as their tool of choice to extend their access and reach within our enterprises, create backdoors and exfiltrate data. As a result, there's a potential financial, operational and brand impact that can conservatively be described as significant to our businesses.

Poor SSH user key management may not send all organizations using SSH into a rapid downward spiral that obliterates most of them, but it can still result in devastating damage to any organization that's ignored this issue.

More Stories By Matthew McKenna

Matthew McKenna is Chief Strategy Officer and vice president of Key Accounts at SSH Communications Security. He brings over 15 years of high technology sales, marketing and management experience to SSH Communications Security and drives strategy, key account sales and evangelism. His expertise in strategically delivering technology solutions that anticipate the marketplace has helped the company become a market leader.

Prior to joining the company, Matthew served as a member of the executive management team of ADP Dealer Services Nordic and Automaster Oy, where he was responsible for international channel operations and manufacturer relations. In addition, he was responsible for key accounts including Mercedes Benz, General Motors, and Scania CV. Before this, he played professional soccer in Germany and Finland.

Matthew holds a Bachelor of Arts degree in German from the University of South Carolina and an MBA from the Helsinki School of Economics and Business Administration.

Comments (0)

Share your thoughts on this story.

Add your comment
You must be signed in to add a comment. Sign-in | Register

In accordance with our Comment Policy, we encourage comments that are on topic, relevant and to-the-point. We will remove comments that include profanity, personal attacks, racial slurs, threats of violence, or other inappropriate material that violates our Terms and Conditions, and will block users who make repeated violations. We ask all readers to expect diversity of opinion and to treat one another with dignity and respect.


@CloudExpo Stories
Interoute has announced the integration of its Global Cloud Infrastructure platform with Rancher Labs’ container management platform, Rancher. This approach enables enterprises to accelerate their digital transformation and infrastructure investments. Matthew Finnie, Interoute CTO commented “Enterprises developing and building apps in the cloud and those on a path to Digital Transformation need Digital ICT Infrastructure that allows them to build, test and deploy faster than ever before. The int...
Without a clear strategy for cost control and an architecture designed with cloud services in mind, costs and operational performance can quickly get out of control. To avoid multiple architectural redesigns requires extensive thought and planning. Boundary (now part of BMC) launched a new public-facing multi-tenant high resolution monitoring service on Amazon AWS two years ago, facing challenges and learning best practices in the early days of the new service.
Niagara Networks exhibited at the 19th International Cloud Expo, which took place at the Santa Clara Convention Center in Santa Clara, CA, in November 2016. Niagara Networks offers the highest port-density systems, and the most complete Next-Generation Network Visibility systems including Network Packet Brokers, Bypass Switches, and Network TAPs.
SYS-CON Events announced today that Outlyer, a monitoring service for DevOps and operations teams, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Outlyer is a monitoring service for DevOps and Operations teams running Cloud, SaaS, Microservices and IoT deployments. Designed for today's dynamic environments that need beyond cloud-scale monitoring, we make monitoring effortless so you ...
My team embarked on building a data lake for our sales and marketing data to better understand customer journeys. This required building a hybrid data pipeline to connect our cloud CRM with the new Hadoop Data Lake. One challenge is that IT was not in a position to provide support until we proved value and marketing did not have the experience, so we embarked on the journey ourselves within the product marketing team for our line of business within Progress. In his session at @BigDataExpo, Sum...
Keeping pace with advancements in software delivery processes and tooling is taxing even for the most proficient organizations. Point tools, platforms, open source and the increasing adoption of private and public cloud services requires strong engineering rigor - all in the face of developer demands to use the tools of choice. As Agile has settled in as a mainstream practice, now DevOps has emerged as the next wave to improve software delivery speed and output. To make DevOps work, organization...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
China Unicom exhibit at the 19th International Cloud Expo, which took place at the Santa Clara Convention Center in Santa Clara, CA, in November 2016. China United Network Communications Group Co. Ltd ("China Unicom") was officially established in 2009 on the basis of the merger of former China Netcom and former China Unicom. China Unicom mainly operates a full range of telecommunications services including mobile broadband (GSM, WCDMA, LTE FDD, TD-LTE), fixed-line broadband, ICT, data communica...
As software becomes more and more complex, we, as software developers, have been splitting up our code into smaller and smaller components. This is also true for the environment in which we run our code: going from bare metal, to VMs to the modern-day Cloud Native world of containers, schedulers and micro services. While we have figured out how to run containerized applications in the cloud using schedulers, we've yet to come up with a good solution to bridge the gap between getting your contain...
Apache Hadoop is emerging as a distributed platform for handling large and fast incoming streams of data. Predictive maintenance, supply chain optimization, and Internet-of-Things analysis are examples where Hadoop provides the scalable storage, processing, and analytics platform to gain meaningful insights from granular data that is typically only valuable from a large-scale, aggregate view. One architecture useful for capturing and analyzing streaming data is the Lambda Architecture, represent...
As organizations realize the scope of the Internet of Things, gaining key insights from Big Data, through the use of advanced analytics, becomes crucial. However, IoT also creates the need for petabyte scale storage of data from millions of devices. A new type of Storage is required which seamlessly integrates robust data analytics with massive scale. These storage systems will act as “smart systems” provide in-place analytics that speed discovery and enable businesses to quickly derive meaningf...
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, will provide a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services ...
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
Building a cross-cloud operational model can be a daunting task. Per-cloud silos are not the answer, but neither is a fully generic abstraction plane that strips out capabilities unique to a particular provider. In his session at 20th Cloud Expo, Chris Wolf, VP & Chief Technology Officer, Global Field & Industry at VMware, will discuss how successful organizations approach cloud operations and management, with insights into where operations should be centralized and when it’s best to decentraliz...
The taxi industry never saw Uber coming. Startups are a threat to incumbents like never before, and a major enabler for startups is that they are instantly “cloud ready.” If innovation moves at the pace of IT, then your company is in trouble. Why? Because your data center will not keep up with frenetic pace AWS, Microsoft and Google are rolling out new capabilities In his session at 20th Cloud Expo, Don Browning, VP of Cloud Architecture at Turner, will posit that disruption is inevitable for c...
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm. In his Day 3 Keynote at 20th Cloud Expo, Chris Brown, a Solutions Marketing Manager at Nutanix, will explore t...
DevOps has often been described in terms of CAMS: Culture, Automation, Measuring, Sharing. While we’ve seen a lot of focus on the “A” and even on the “M”, there are very few examples of why the “C" is equally important in the DevOps equation. In her session at @DevOps Summit, Lori MacVittie, of F5 Networks, explored HTTP/1 and HTTP/2 along with Microservices to illustrate why a collaborative culture between Dev, Ops, and the Network is critical to ensuring success.
SYS-CON Events announced today that SoftLayer, an IBM Company, has been named “Gold Sponsor” of SYS-CON's 18th Cloud Expo, which will take place on June 7-9, 2016, at the Javits Center in New York, New York. SoftLayer, an IBM Company, provides cloud infrastructure as a service from a growing number of data centers and network points of presence around the world. SoftLayer’s customers range from Web startups to global enterprises.
SYS-CON Events announced today that Conference Guru has been named “Media Sponsor” of SYS-CON's 20th International Cloud Expo, which will take place on June 6–8, 2017, at the Javits Center in New York City, NY. A valuable conference experience generates new contacts, sales leads, potential strategic partners and potential investors; helps gather competitive intelligence and even provides inspiration for new products and services. Conference Guru works with conference organizers to pass great dea...
SYS-CON Events announced today that Technologic Systems Inc., an embedded systems solutions company, will exhibit at SYS-CON's @ThingsExpo, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Technologic Systems is an embedded systems company with headquarters in Fountain Hills, Arizona. They have been in business for 32 years, helping more than 8,000 OEM customers and building over a hundred COTS products that have never been discontinued. Technologic Systems’ pr...