@CloudExpo Authors: Automic Blog, Liz McMillan, Elizabeth White, Pat Romanski, Nate Vickery

Related Topics: @CloudExpo, @DevOpsSummit

@CloudExpo: Blog Feed Post

Gene Kim on Automated Dev/Test Environments | @DevOpsSummit [#DevOps]

Gene Kim on the Need for On-demand Access to Automated Dev/Test Environments

Gene Kim on the Need for On-demand Access to Automated Dev/Test Environments

​Noel: Hello, this is Noel Wurst with Skytap, and I am speaking with Gene Kim today. At Skytap, a lot of us are big fans of Gene and his work, and we just finished reading The Phoenix Project, which we will talk about, as well as The DevOps Cookbook. I'm curious to know how that's coming along, and when we can expect to read it. Gene, how are you today.

Gene: I'm doing great, and it's great being here with you.

Noel: So, to get started, being the DevOps aficionado that you are, we recently had a contest on our blog where we asked people to submit their best definition of DevOps, and we knew we would get a bunch of different definitions.

What I thought was the best definition, had a line about how “communication was the responsibility of both Dev and Ops.” I know sounds really simple, but if you extend that definition of “communication” I thought that did a good job of showing that DevOps is absolutely both departments’ responsibility. It’s not just relying on one team to change culturally. I’d love to get your short definition of how you define DevOps today.

Gene: This is my personal definition: I would define DevOps by the outcomes. In my mind, DevOps is those set of cultural norms and technology practices that enable the fast flow of planned work from, among others, development, through tests into operations while preserving world class reliability, operation and security.

DevOps is not about what you do, but what your outcomes are. So many things that we associate with DevOps fits underneath this very broad umbrella of beliefs and practices—which of course, communication and culture are part of them. I love this notion that it takes more than just a bunch of nice people communicating well to achieve DevOps outcomes.

For organizations to be able to do tens, hundreds of thousands of deploys per day, while preserving great reliability, security, availability, it takes amazing technical practices, as well.

Otherwise, we're just a bunch of nice people getting suboptimal outcomes.

Noel: Right. That reminds me of the way we talk about Agile sometimes. There's so much focus on culture. Obviously culture is very important, but you can have the greatest culture in the world, but without the technology to enable you to put all that communication and culture into play—it's not going to get you more than having a bunch of nice people at work.

Gene: Exactly. In fact, one of my heroes is Elizabeth Hendrickson. She was one of the speakers at the DevOps Enterprise a couple months ago. We were having this exact same conversation at Agile 2014 is that some of our best friends are Agile coaches and Agile practitioners, but a talk about a success story that doesn't involve the actual technical practices, is in some ways unsatisfactory.

Exactly what you said, it takes more than just people communicating, and it takes more than just trust. Those are necessary, but those set the stage for the really hard things, which is how do you get a level of productivity and reliability that we didn't even think possible five years ago.



Noel: Exactly. That really leads to a great quote from The Phoenix Project. It's a quote from Erik, who has so many great lines, but there's a line that says:

Allspaw taught us that Dev and Ops working together, along with QA and the business are a super tribe that can achieve amazing things.

He did this by ensuring environments were always available when they were needed. He automated the build and deployment process, recognizing that infrastructure could be treated as code, and says that enabled him to create a “one step environment creation and deploy procedure.”

Here at Skytap, we’ve seen firsthand that concept of a one-step environment creation and deployment procedure is not something that everyone is even aware exists. What do you think will make that technology common sense for everyone? Is there something that will come along where it won’t be so surprising for people to find out they have access to these things?

Gene: This is one of my favorite lines in the book, because it's actually one of my own personal “ah-ha” moments. This whole notion, what we see in high performers, are several things that substantiate how important this capability is. One, this is in the DevOps Survey of Practice that I did with Jez Humble and the great folks at Puppet Labs. We found that certain behaviors were not only a high predictor of performance, but in some cases, they were, I would elevate them to be prerequisites of performance.

One of them was that there is an automated environment creation process that can build the dev, test, and the production environments all at the same time. This is so amazing because it eliminates a whole category of errors that most of us grew up with, which was that those environments that actually match. You only find out during the deploy.

A second piece is that they have an automated deployment process, as opposed to a six-week, thirteen hundred step deploy process, which is, all the steps are manually error prone.

The third is, where do these environments come from? There are amazing technologies like Skytap, but I think what's important is that the same build mechanism that creates these environments are all under some sort of version control, so we can repeatedly, reliably, build these environments and I think the constituency, who would benefit from this most is Developers.

That means that Developers can now work, write the code in, and run their code in production-like environments, even at the earliest stage of the process.

That is such a game changer. Because it eliminates the whole phenomenon around, “hey, it worked on my laptop, why doesn't it work in production?” I think so much of our experience has been, “hey, I would like a test environment” and the answer is, “I can get one for you in thirty weeks.”  The whole notion that you can get one on-demand in minutes changes everything.

Noel: There's a great survey that was recently done by voke, Inc., and voke's founder Teresa Lanowitz uses the phrase “lifecycle virtualization” to describe the benefits of extending virtualization into dev and test, and shifting everything left. I've seen her speak on this a couple times and she talks about the fact that it's 2014. It shouldn't be so mind blowing that you have the ability to do these things.

We're not sitting around waiting on environments and provisioning and hardware anymore. There's just no need.

Gene: John Willis said something I found startling. He said these capabilities have been around for nearly a decade around virtualization. I think the magic was virtualization where you could spin up environments in seconds or minutes. The problem was that the provisioning and code deployment process was still manual, so the whole notion of “infrastructure as code” has elevated our sights on not only what's possible, but what is actually required to achieve world class performance.

The notion of virtualization, and being able to spin up environments, it's not just good for Ops. The fact you can shift everything left dramatically improves productivity for developers. Jez Humble has this great quote. He said the best proxy measure, for him, of developer productivity, is lead time.

Lead time would be measured by, “how quickly can we go from code committed through the test cycle were everything is running in a production-like environment and being deployed into production?”

This is such a provocative statement. Development productivity is not about how many features can you complete, it's about how quickly can you go from, it runs on your laptop, to where you have confidence that it's going to run in production, and then it's actually running in production. I love that quote.

Noel: That leads to another quote from The Phoenix Project. “Until code is in production, there's no value actually being generated.”

Gene: Right. In fact, one of the sayings from the lean manufacturing community that I think about so often. There's this deeply held belief that lead time is the most accurate predictor of customer satisfaction, quality, and employee happiness. This is one of the most cherished expressions.

The fact that as we benchmark over 14,000 organizations in the last two years, we found that was absolutely true. Lead time correlates with change success rate, mean time to repair, deployment frequency, employee happiness and even organizational performance, meaning how likely is the organization to exceed productivity, profitability, and market share goals.

They're twice as likely to exceed profitability, market share and productivity goals.

Noel: That’s awesome.

Gene: It’s true of manufacturing, and apparently true in DevOps as well.

Noel: That's very cool. So, talk about The DevOps Cookbook for a little while. I noticed on your website it says that your intent behind it was, "To catalog what the high performing DevOps organizations all have in common. Attempting to describe all the necessary and sufficient steps to create the culture of values, processes, procedures, and daily work behind their transformations.”

That sounds like the most massive undertaking. It's so much to cover. How long have you been compiling that research and then how many organizations did you collect that information from?

Gene: The DevOps Cookbook was actually supposed to come out before The Phoenix Project. Arguably, it's about two years late. I've promised everybody and myself that a first draft manuscript will be complete this year.

I would say in some ways I'm very glad that we were doing this, coming out now as opposed to two years before.

Noel: I think it works well.

Gene: Yeah, we've been able up to now to benchmark 14,000 organizations and really prove to a level of confidence that we wouldn't have been able to do, had you not done that research.

It's been now at least, almost over two years in the making.

Noel: That sounds like a pretty short time for the number of things you're attempting to show in there. Did you find that many of those who were “high performing” as you call them, were “doing DevOps” in a similar way, or kind of like Agile, in all its ambiguity and all the definitions of DevOps—did you find people doing DevOps in their own unique way, yet still getting that same goal.

Gene: No, I would say our experience has been the opposite. There's a certain uniformity about “the emergent behavior.” It’s as if all these different organizations have simultaneously and independently derived the same sort of systems of work of how exactly does dev, test, InfoSec, and Ops work together to get these amazing outcomes.

We now know through the research that there are seven predictors of performance.

First off is the use of version control by operations. In other words, not only do you have a build mechanism, that can build the Dev, test, and prod environments all at the same time, but this is under version control. There's a continuous integration and continuous tests in integration practices.

There's, develop the checking code into a chunk every day, and automated test suites to validate that it’s actually working as designed. There's automated deployment processes, there's proactive monitoring of the production environment, and there's a high trust culture. The presence of a high level of trust seems to be a prerequisite for their performance, as opposed to a culture fear, a cultured bureaucracy.

Those seem to be the emergent behaviors and beliefs that drive all organizations whether it's Google, Amazon, Etsy or whether it's Disney, Atheon, or the U.S. Department of Homeland Security.

It's always been very cool to be able to show that it's not an amorphous bunch of nice people doing random things. Instead, it's a common philosophy, a common set of practices that lead to high performance.

Noel: It's almost a relief to find out that, one, there are 14,000 that are doing something similar to this. You look at the track record of some of those bigger names you were talking about and to know it's a set of things that are hard to argue against.

It almost takes a little bit of the weight off of, “can we do this, will this work here, are we too big, are we too small?.” Those are all things that would be great for everyone to be doing.

Gene: I think one of the reasons for the separate research we do, or you benchmark 14,000 organizations, it's called “a cross-population study.” That's type of research they used to show how they linked smoking with early morbidity.

You cast a very wide net, and then you try to figure out what is the link between early death, behaviors, and environmental factors and so forth.

One of the reasons that one does cross-population studies is to negate objections. People say, “Is it really true that large organizations can't be high performers?” No, it turns out that that’s not true.

How about, “Does it really matter if you have a DevOps team or not?” Jez thought yes, I thought no. We tested it, and it turned out it didn’t matter. Does it matter who does your deploys? We measured change success rate,  Jez thought it would be better if Dev does deploys, I though it was better if Ops does deploys, it turns out it didn't matter. The two populations have a statistically identical performance.

These are the kind of awesome things you can do when you can do benchmark studies like this.

Noel: That’s awesome. To move to another bit of writing that you've done, you have a whitepaper on your site that I really enjoyed and would suggest other people check out. It’s, “The top 11 things you need to know about DevOps.”

One of your favorite DevOps patterns resonated with me a lot. You talk about the need for an automated environment creation process, to go back to that again.

You said that it makes environments not available just early, but you said perhaps before a project even begins. Like these things you just listed, your seven principles, this automated creation process has become almost essential at this point.

There's not a lot of room for saying, “Our wait time isn't that bad, or we're not being that hampered.” They should be available before you've even started.

Gene: These are certainly, out of the set of set of about seven things we found, that were the strongest predictors of performance. I think one can argue that, it's hard to argue that you can get high performance if Developers are building their own environments. Environments that are different from the production environments, and even if they used production environments, or it takes a long time, or they're lagging behind what their real production environment is. It's one of these sort of logical, satisfying conclusions. Absolutely, yes.

I think it matches most people's experiences. I love this phrase that someone's said to me. He said, “The goal of science is to explain the most amount of observable phenomenon with the fewest number of principles, to confirm deeply held intuitions, and to reveal surprising insights.

I think this is one of those. It makes a lot of sense.

Noel: Absolutely. I've got one last question for you. In that same list of your eleven things to know about DevOps, number eight asks a question: “How do InfoSec and QA integrate into the DevOps work team?” I think this is a great question and whenever you've been talking about it in this conversation, you've always mentioned reliability and security as well.

But, you mentioned something I hadn't heard before, and it was:


When software is delivered as a service, and defects can be fixed very quickly, then QA can reduce its reliance on testing and instead rely more on production monitoring to detect defects in production, as long as they can be quickly fixed.

I just wanted to get clear on that. I'd always thought about testers wanting to test more often and the whole move to test earlier, but I wanted to find out  the risks or the rewards of being able to detect defects in production and not sooner.

Gene: There's an adage in the DevOps community of, “Architect not for meantime between failures, but instead architect for meantime to repair.”

In other words, instead of spending our time trying to create the perfect system that never fails, let's instead create systems that are designed so that we can detect problems early and fix them fast.

I guess what I might have imprecisely stated in that sentences was, if you adopt this philosophy, it means that we don't have to find and prevent all the issues before we release. If we can have confidence that we'll find issues fast enough, and that it's safe to make changes, and also that it then restores quickly—then let’s consider the opposite of that. If we have this fragile, big monolithic application that takes two weeks to deploy, and whenever something goes wrong, three hundred people have to drop what they're doing to nurse a quick, hot fix in production.

It means that the desire is that we want to do more testing to try to prevent future failures. This whole notion of the safe environment where it's safe to make changes and can rely upon production monitoring to find issues quickly, it alleviates a huge burden for developers, and quality engineers and information security.

I think that's another game changer in technology.

Noel: That reminds me that video of the Jeep.

Gene: Yeah!

Noel: They completely, they didn't just break it, the broke it into every single part imaginable, but like you just said, they rebuilt it on the road.

It didn't have to go back to the body shop or back to the testing team. They rebuilt it on the road in just as quick a time as it took to break. It was back up and running in no time.

Gene: Exactly right. Imagine if they had to take it back to shop and they had to go through a two-month certification process in order to be back on the road. What we're talking about is the exact opposite.

Noel: Exactly. Awesome. That’s about all I had for you today, thank you so much for your time. And if anyone wants to learn more about Gene, to learn more about The Phoenix Project, or The DevOps Cookbook, I recommend doing all three.

Gene: Hey, no problem. And if anyone wants to subscribe to updates about the DevOps Cookbook release and receive an advance draft of the manuscript, you can sign up here.Gene is a multiple award winning CTO, researcher and author.  He was founder and CTO of Tripwire for 13 years. He has written three books, including “The Visible Ops Handbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win."  Gene is a huge fan of IT operations, and how it can enable developers to maximize throughput of features from “code complete” to “in production,” without causing chaos and disruption to the IT environment.

Read the original blog entry...

More Stories By Skytap Blog

Author: Noel Wurst is the managing content editor at Skytap. Skytap provides SaaS-based dev/test environments to the enterprise. Skytap solution removes the inefficiencies and constraints that companies have within their software development lifecycle. As a result, customers release better software faster. In this blog, we publish engaging, thought provoking stories that revolve around agile enterprise applications and cloud-based development and testing.

@CloudExpo Stories
SYS-CON Events announced today that Evatronix will exhibit at SYS-CON's 21st International Cloud Expo®, which will take place on Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA. Evatronix SA offers comprehensive solutions in the design and implementation of electronic systems, in CAD / CAM deployment, and also is a designer and manufacturer of advanced 3D scanners for professional applications.
"I focus on what we are calling CAST Highlight, which is our SaaS application portfolio analysis tool. It is an extremely lightweight tool that can integrate with pretty much any build process right now," explained Andrew Siegmund, Application Migration Specialist for CAST, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
To get the most out of their data, successful companies are not focusing on queries and data lakes, they are actively integrating analytics into their operations with a data-first application development approach. Real-time adjustments to improve revenues, reduce costs, or mitigate risk rely on applications that minimize latency on a variety of data sources. In his session at @BigDataExpo, Jack Norris, Senior Vice President, Data and Applications at MapR Technologies, reviewed best practices to ...
As many know, the first generation of Cloud Management Platform (CMP) solutions were designed for managing virtual infrastructure (IaaS) and traditional applications. But that's no longer enough to satisfy evolving and complex business requirements. In his session at 21st Cloud Expo, Scott Davis, Embotics CTO, explored how next-generation CMPs ensure organizations can manage cloud-native and microservice-based application architectures, while also facilitating agile DevOps methodology. He expla...
SYS-CON Events announced today that Synametrics Technologies will exhibit at SYS-CON's 22nd International Cloud Expo®, which will take place on June 5-7, 2018, at the Javits Center in New York, NY. Synametrics Technologies is a privately held company based in Plainsboro, New Jersey that has been providing solutions for the developer community since 1997. Based on the success of its initial product offerings such as WinSQL, Xeams, SynaMan and Syncrify, Synametrics continues to create and hone inn...
"Evatronix provides design services to companies that need to integrate the IoT technology in their products but they don't necessarily have the expertise, knowledge and design team to do so," explained Adam Morawiec, VP of Business Development at Evatronix, in this SYS-CON.tv interview at @ThingsExpo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
The dynamic nature of the cloud means that change is a constant when it comes to modern cloud-based infrastructure. Delivering modern applications to end users, therefore, is a constantly shifting challenge. Delivery automation helps IT Ops teams ensure that apps are providing an optimal end user experience over hybrid-cloud and multi-cloud environments, no matter what the current state of the infrastructure is. To employ a delivery automation strategy that reflects your business rules, making r...
The past few years have brought a sea change in the way applications are architected, developed, and consumed—increasing both the complexity of testing and the business impact of software failures. How can software testing professionals keep pace with modern application delivery, given the trends that impact both architectures (cloud, microservices, and APIs) and processes (DevOps, agile, and continuous delivery)? This is where continuous testing comes in. D
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is foundational - something customers routinely cite as a top pain point. In his session at @DevOpsSummit at 21st Cloud Expo, Bill Borsari, Head of Systems Engineering at Datera, explored how organizations can reap the bene...
In a recent survey, Sumo Logic surveyed 1,500 customers who employ cloud services such as Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP). According to the survey, a quarter of the respondents have already deployed Docker containers and nearly as many (23 percent) are employing the AWS Lambda serverless computing framework. It’s clear: serverless is here to stay. The adoption does come with some needed changes, within both application development and operations. Tha...
Digital transformation is about embracing digital technologies into a company's culture to better connect with its customers, automate processes, create better tools, enter new markets, etc. Such a transformation requires continuous orchestration across teams and an environment based on open collaboration and daily experiments. In his session at 21st Cloud Expo, Alex Casalboni, Technical (Cloud) Evangelist at Cloud Academy, explored and discussed the most urgent unsolved challenges to achieve f...
With tough new regulations coming to Europe on data privacy in May 2018, Calligo will explain why in reality the effect is global and transforms how you consider critical data. EU GDPR fundamentally rewrites the rules for cloud, Big Data and IoT. In his session at 21st Cloud Expo, Adam Ryan, Vice President and General Manager EMEA at Calligo, examined the regulations and provided insight on how it affects technology, challenges the established rules and will usher in new levels of diligence arou...
In his general session at 21st Cloud Expo, Greg Dumas, Calligo’s Vice President and G.M. of US operations, discussed the new Global Data Protection Regulation and how Calligo can help business stay compliant in digitally globalized world. Greg Dumas is Calligo's Vice President and G.M. of US operations. Calligo is an established service provider that provides an innovative platform for trusted cloud solutions. Calligo’s customers are typically most concerned about GDPR compliance, application p...
Mobile device usage has increased exponentially during the past several years, as consumers rely on handhelds for everything from news and weather to banking and purchases. What can we expect in the next few years? The way in which we interact with our devices will fundamentally change, as businesses leverage Artificial Intelligence. We already see this taking shape as businesses leverage AI for cost savings and customer responsiveness. This trend will continue, as AI is used for more sophistica...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
In his Opening Keynote at 21st Cloud Expo, John Considine, General Manager of IBM Cloud Infrastructure, led attendees through the exciting evolution of the cloud. He looked at this major disruption from the perspective of technology, business models, and what this means for enterprises of all sizes. John Considine is General Manager of Cloud Infrastructure Services at IBM. In that role he is responsible for leading IBM’s public cloud infrastructure including strategy, development, and offering m...
Most technology leaders, contemporary and from the hardware era, are reshaping their businesses to do software. They hope to capture value from emerging technologies such as IoT, SDN, and AI. Ultimately, irrespective of the vertical, it is about deriving value from independent software applications participating in an ecosystem as one comprehensive solution. In his session at @ThingsExpo, Kausik Sridhar, founder and CTO of Pulzze Systems, discussed how given the magnitude of today's application ...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
The 22nd International Cloud Expo | 1st DXWorld Expo has announced that its Call for Papers is open. Cloud Expo | DXWorld Expo, to be held June 5-7, 2018, at the Javits Center in New York, NY, brings together Cloud Computing, Digital Transformation, Big Data, Internet of Things, DevOps, Machine Learning and WebRTC to one location. With cloud computing driving a higher percentage of enterprise IT budgets every year, it becomes increasingly important to plant your flag in this fast-expanding busin...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...