@CloudExpo Authors: Kelly Burford, Karthick Viswanathan, Xenia von Wedel, Yeshim Deniz, Dana Gardner

Related Topics: @CloudExpo, @DevOpsSummit

@CloudExpo: Blog Feed Post

Gene Kim on Automated Dev/Test Environments | @DevOpsSummit [#DevOps]

Gene Kim on the Need for On-demand Access to Automated Dev/Test Environments

Gene Kim on the Need for On-demand Access to Automated Dev/Test Environments

​Noel: Hello, this is Noel Wurst with Skytap, and I am speaking with Gene Kim today. At Skytap, a lot of us are big fans of Gene and his work, and we just finished reading The Phoenix Project, which we will talk about, as well as The DevOps Cookbook. I'm curious to know how that's coming along, and when we can expect to read it. Gene, how are you today.

Gene: I'm doing great, and it's great being here with you.

Noel: So, to get started, being the DevOps aficionado that you are, we recently had a contest on our blog where we asked people to submit their best definition of DevOps, and we knew we would get a bunch of different definitions.

What I thought was the best definition, had a line about how “communication was the responsibility of both Dev and Ops.” I know sounds really simple, but if you extend that definition of “communication” I thought that did a good job of showing that DevOps is absolutely both departments’ responsibility. It’s not just relying on one team to change culturally. I’d love to get your short definition of how you define DevOps today.

Gene: This is my personal definition: I would define DevOps by the outcomes. In my mind, DevOps is those set of cultural norms and technology practices that enable the fast flow of planned work from, among others, development, through tests into operations while preserving world class reliability, operation and security.

DevOps is not about what you do, but what your outcomes are. So many things that we associate with DevOps fits underneath this very broad umbrella of beliefs and practices—which of course, communication and culture are part of them. I love this notion that it takes more than just a bunch of nice people communicating well to achieve DevOps outcomes.

For organizations to be able to do tens, hundreds of thousands of deploys per day, while preserving great reliability, security, availability, it takes amazing technical practices, as well.

Otherwise, we're just a bunch of nice people getting suboptimal outcomes.

Noel: Right. That reminds me of the way we talk about Agile sometimes. There's so much focus on culture. Obviously culture is very important, but you can have the greatest culture in the world, but without the technology to enable you to put all that communication and culture into play—it's not going to get you more than having a bunch of nice people at work.

Gene: Exactly. In fact, one of my heroes is Elizabeth Hendrickson. She was one of the speakers at the DevOps Enterprise a couple months ago. We were having this exact same conversation at Agile 2014 is that some of our best friends are Agile coaches and Agile practitioners, but a talk about a success story that doesn't involve the actual technical practices, is in some ways unsatisfactory.

Exactly what you said, it takes more than just people communicating, and it takes more than just trust. Those are necessary, but those set the stage for the really hard things, which is how do you get a level of productivity and reliability that we didn't even think possible five years ago.



Noel: Exactly. That really leads to a great quote from The Phoenix Project. It's a quote from Erik, who has so many great lines, but there's a line that says:

Allspaw taught us that Dev and Ops working together, along with QA and the business are a super tribe that can achieve amazing things.

He did this by ensuring environments were always available when they were needed. He automated the build and deployment process, recognizing that infrastructure could be treated as code, and says that enabled him to create a “one step environment creation and deploy procedure.”

Here at Skytap, we’ve seen firsthand that concept of a one-step environment creation and deployment procedure is not something that everyone is even aware exists. What do you think will make that technology common sense for everyone? Is there something that will come along where it won’t be so surprising for people to find out they have access to these things?

Gene: This is one of my favorite lines in the book, because it's actually one of my own personal “ah-ha” moments. This whole notion, what we see in high performers, are several things that substantiate how important this capability is. One, this is in the DevOps Survey of Practice that I did with Jez Humble and the great folks at Puppet Labs. We found that certain behaviors were not only a high predictor of performance, but in some cases, they were, I would elevate them to be prerequisites of performance.

One of them was that there is an automated environment creation process that can build the dev, test, and the production environments all at the same time. This is so amazing because it eliminates a whole category of errors that most of us grew up with, which was that those environments that actually match. You only find out during the deploy.

A second piece is that they have an automated deployment process, as opposed to a six-week, thirteen hundred step deploy process, which is, all the steps are manually error prone.

The third is, where do these environments come from? There are amazing technologies like Skytap, but I think what's important is that the same build mechanism that creates these environments are all under some sort of version control, so we can repeatedly, reliably, build these environments and I think the constituency, who would benefit from this most is Developers.

That means that Developers can now work, write the code in, and run their code in production-like environments, even at the earliest stage of the process.

That is such a game changer. Because it eliminates the whole phenomenon around, “hey, it worked on my laptop, why doesn't it work in production?” I think so much of our experience has been, “hey, I would like a test environment” and the answer is, “I can get one for you in thirty weeks.”  The whole notion that you can get one on-demand in minutes changes everything.

Noel: There's a great survey that was recently done by voke, Inc., and voke's founder Teresa Lanowitz uses the phrase “lifecycle virtualization” to describe the benefits of extending virtualization into dev and test, and shifting everything left. I've seen her speak on this a couple times and she talks about the fact that it's 2014. It shouldn't be so mind blowing that you have the ability to do these things.

We're not sitting around waiting on environments and provisioning and hardware anymore. There's just no need.

Gene: John Willis said something I found startling. He said these capabilities have been around for nearly a decade around virtualization. I think the magic was virtualization where you could spin up environments in seconds or minutes. The problem was that the provisioning and code deployment process was still manual, so the whole notion of “infrastructure as code” has elevated our sights on not only what's possible, but what is actually required to achieve world class performance.

The notion of virtualization, and being able to spin up environments, it's not just good for Ops. The fact you can shift everything left dramatically improves productivity for developers. Jez Humble has this great quote. He said the best proxy measure, for him, of developer productivity, is lead time.

Lead time would be measured by, “how quickly can we go from code committed through the test cycle were everything is running in a production-like environment and being deployed into production?”

This is such a provocative statement. Development productivity is not about how many features can you complete, it's about how quickly can you go from, it runs on your laptop, to where you have confidence that it's going to run in production, and then it's actually running in production. I love that quote.

Noel: That leads to another quote from The Phoenix Project. “Until code is in production, there's no value actually being generated.”

Gene: Right. In fact, one of the sayings from the lean manufacturing community that I think about so often. There's this deeply held belief that lead time is the most accurate predictor of customer satisfaction, quality, and employee happiness. This is one of the most cherished expressions.

The fact that as we benchmark over 14,000 organizations in the last two years, we found that was absolutely true. Lead time correlates with change success rate, mean time to repair, deployment frequency, employee happiness and even organizational performance, meaning how likely is the organization to exceed productivity, profitability, and market share goals.

They're twice as likely to exceed profitability, market share and productivity goals.

Noel: That’s awesome.

Gene: It’s true of manufacturing, and apparently true in DevOps as well.

Noel: That's very cool. So, talk about The DevOps Cookbook for a little while. I noticed on your website it says that your intent behind it was, "To catalog what the high performing DevOps organizations all have in common. Attempting to describe all the necessary and sufficient steps to create the culture of values, processes, procedures, and daily work behind their transformations.”

That sounds like the most massive undertaking. It's so much to cover. How long have you been compiling that research and then how many organizations did you collect that information from?

Gene: The DevOps Cookbook was actually supposed to come out before The Phoenix Project. Arguably, it's about two years late. I've promised everybody and myself that a first draft manuscript will be complete this year.

I would say in some ways I'm very glad that we were doing this, coming out now as opposed to two years before.

Noel: I think it works well.

Gene: Yeah, we've been able up to now to benchmark 14,000 organizations and really prove to a level of confidence that we wouldn't have been able to do, had you not done that research.

It's been now at least, almost over two years in the making.

Noel: That sounds like a pretty short time for the number of things you're attempting to show in there. Did you find that many of those who were “high performing” as you call them, were “doing DevOps” in a similar way, or kind of like Agile, in all its ambiguity and all the definitions of DevOps—did you find people doing DevOps in their own unique way, yet still getting that same goal.

Gene: No, I would say our experience has been the opposite. There's a certain uniformity about “the emergent behavior.” It’s as if all these different organizations have simultaneously and independently derived the same sort of systems of work of how exactly does dev, test, InfoSec, and Ops work together to get these amazing outcomes.

We now know through the research that there are seven predictors of performance.

First off is the use of version control by operations. In other words, not only do you have a build mechanism, that can build the Dev, test, and prod environments all at the same time, but this is under version control. There's a continuous integration and continuous tests in integration practices.

There's, develop the checking code into a chunk every day, and automated test suites to validate that it’s actually working as designed. There's automated deployment processes, there's proactive monitoring of the production environment, and there's a high trust culture. The presence of a high level of trust seems to be a prerequisite for their performance, as opposed to a culture fear, a cultured bureaucracy.

Those seem to be the emergent behaviors and beliefs that drive all organizations whether it's Google, Amazon, Etsy or whether it's Disney, Atheon, or the U.S. Department of Homeland Security.

It's always been very cool to be able to show that it's not an amorphous bunch of nice people doing random things. Instead, it's a common philosophy, a common set of practices that lead to high performance.

Noel: It's almost a relief to find out that, one, there are 14,000 that are doing something similar to this. You look at the track record of some of those bigger names you were talking about and to know it's a set of things that are hard to argue against.

It almost takes a little bit of the weight off of, “can we do this, will this work here, are we too big, are we too small?.” Those are all things that would be great for everyone to be doing.

Gene: I think one of the reasons for the separate research we do, or you benchmark 14,000 organizations, it's called “a cross-population study.” That's type of research they used to show how they linked smoking with early morbidity.

You cast a very wide net, and then you try to figure out what is the link between early death, behaviors, and environmental factors and so forth.

One of the reasons that one does cross-population studies is to negate objections. People say, “Is it really true that large organizations can't be high performers?” No, it turns out that that’s not true.

How about, “Does it really matter if you have a DevOps team or not?” Jez thought yes, I thought no. We tested it, and it turned out it didn’t matter. Does it matter who does your deploys? We measured change success rate,  Jez thought it would be better if Dev does deploys, I though it was better if Ops does deploys, it turns out it didn't matter. The two populations have a statistically identical performance.

These are the kind of awesome things you can do when you can do benchmark studies like this.

Noel: That’s awesome. To move to another bit of writing that you've done, you have a whitepaper on your site that I really enjoyed and would suggest other people check out. It’s, “The top 11 things you need to know about DevOps.”

One of your favorite DevOps patterns resonated with me a lot. You talk about the need for an automated environment creation process, to go back to that again.

You said that it makes environments not available just early, but you said perhaps before a project even begins. Like these things you just listed, your seven principles, this automated creation process has become almost essential at this point.

There's not a lot of room for saying, “Our wait time isn't that bad, or we're not being that hampered.” They should be available before you've even started.

Gene: These are certainly, out of the set of set of about seven things we found, that were the strongest predictors of performance. I think one can argue that, it's hard to argue that you can get high performance if Developers are building their own environments. Environments that are different from the production environments, and even if they used production environments, or it takes a long time, or they're lagging behind what their real production environment is. It's one of these sort of logical, satisfying conclusions. Absolutely, yes.

I think it matches most people's experiences. I love this phrase that someone's said to me. He said, “The goal of science is to explain the most amount of observable phenomenon with the fewest number of principles, to confirm deeply held intuitions, and to reveal surprising insights.

I think this is one of those. It makes a lot of sense.

Noel: Absolutely. I've got one last question for you. In that same list of your eleven things to know about DevOps, number eight asks a question: “How do InfoSec and QA integrate into the DevOps work team?” I think this is a great question and whenever you've been talking about it in this conversation, you've always mentioned reliability and security as well.

But, you mentioned something I hadn't heard before, and it was:


When software is delivered as a service, and defects can be fixed very quickly, then QA can reduce its reliance on testing and instead rely more on production monitoring to detect defects in production, as long as they can be quickly fixed.

I just wanted to get clear on that. I'd always thought about testers wanting to test more often and the whole move to test earlier, but I wanted to find out  the risks or the rewards of being able to detect defects in production and not sooner.

Gene: There's an adage in the DevOps community of, “Architect not for meantime between failures, but instead architect for meantime to repair.”

In other words, instead of spending our time trying to create the perfect system that never fails, let's instead create systems that are designed so that we can detect problems early and fix them fast.

I guess what I might have imprecisely stated in that sentences was, if you adopt this philosophy, it means that we don't have to find and prevent all the issues before we release. If we can have confidence that we'll find issues fast enough, and that it's safe to make changes, and also that it then restores quickly—then let’s consider the opposite of that. If we have this fragile, big monolithic application that takes two weeks to deploy, and whenever something goes wrong, three hundred people have to drop what they're doing to nurse a quick, hot fix in production.

It means that the desire is that we want to do more testing to try to prevent future failures. This whole notion of the safe environment where it's safe to make changes and can rely upon production monitoring to find issues quickly, it alleviates a huge burden for developers, and quality engineers and information security.

I think that's another game changer in technology.

Noel: That reminds me that video of the Jeep.

Gene: Yeah!

Noel: They completely, they didn't just break it, the broke it into every single part imaginable, but like you just said, they rebuilt it on the road.

It didn't have to go back to the body shop or back to the testing team. They rebuilt it on the road in just as quick a time as it took to break. It was back up and running in no time.

Gene: Exactly right. Imagine if they had to take it back to shop and they had to go through a two-month certification process in order to be back on the road. What we're talking about is the exact opposite.

Noel: Exactly. Awesome. That’s about all I had for you today, thank you so much for your time. And if anyone wants to learn more about Gene, to learn more about The Phoenix Project, or The DevOps Cookbook, I recommend doing all three.

Gene: Hey, no problem. And if anyone wants to subscribe to updates about the DevOps Cookbook release and receive an advance draft of the manuscript, you can sign up here.Gene is a multiple award winning CTO, researcher and author.  He was founder and CTO of Tripwire for 13 years. He has written three books, including “The Visible Ops Handbook” and “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win."  Gene is a huge fan of IT operations, and how it can enable developers to maximize throughput of features from “code complete” to “in production,” without causing chaos and disruption to the IT environment.

Read the original blog entry...

More Stories By Skytap Blog

Author: Noel Wurst is the managing content editor at Skytap. Skytap provides SaaS-based dev/test environments to the enterprise. Skytap solution removes the inefficiencies and constraints that companies have within their software development lifecycle. As a result, customers release better software faster. In this blog, we publish engaging, thought provoking stories that revolve around agile enterprise applications and cloud-based development and testing.

@CloudExpo Stories
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settle...
The use of containers by developers -- and now increasingly IT operators -- has grown from infatuation to deep and abiding love. But as with any long-term affair, the honeymoon soon leads to needing to live well together ... and maybe even getting some relationship help along the way. And so it goes with container orchestration and automation solutions, which are rapidly emerging as the means to maintain the bliss between rapid container adoption and broad container use among multiple cloud host...
Blockchain is a shared, secure record of exchange that establishes trust, accountability and transparency across business networks. Supported by the Linux Foundation's open source, open-standards based Hyperledger Project, Blockchain has the potential to improve regulatory compliance, reduce cost as well as advance trade. Are you curious about how Blockchain is built for business? In her session at 21st Cloud Expo, René Bostic, Technical VP of the IBM Cloud Unit in North America, discussed the b...
Blockchain. A day doesn’t seem to go by without seeing articles and discussions about the technology. According to PwC executive Seamus Cushley, approximately $1.4B has been invested in blockchain just last year. In Gartner’s recent hype cycle for emerging technologies, blockchain is approaching the peak. It is considered by Gartner as one of the ‘Key platform-enabling technologies to track.’ While there is a lot of ‘hype vs reality’ discussions going on, there is no arguing that blockchain is b...
Imagine if you will, a retail floor so densely packed with sensors that they can pick up the movements of insects scurrying across a store aisle. Or a component of a piece of factory equipment so well-instrumented that its digital twin provides resolution down to the micrometer.
"IBM is really all in on blockchain. We take a look at sort of the history of blockchain ledger technologies. It started out with bitcoin, Ethereum, and IBM evaluated these particular blockchain technologies and found they were anonymous and permissionless and that many companies were looking for permissioned blockchain," stated René Bostic, Technical VP of the IBM Cloud Unit in North America, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Conventi...
As DevOps methodologies expand their reach across the enterprise, organizations face the daunting challenge of adapting related cloud strategies to ensure optimal alignment, from managing complexity to ensuring proper governance. How can culture, automation, legacy apps and even budget be reexamined to enable this ongoing shift within the modern software factory? In her Day 2 Keynote at @DevOpsSummit at 21st Cloud Expo, Aruna Ravichandran, VP, DevOps Solutions Marketing, CA Technologies, was jo...
While some developers care passionately about how data centers and clouds are architected, for most, it is only the end result that matters. To the majority of companies, technology exists to solve a business problem, and only delivers value when it is solving that problem. 2017 brings the mainstream adoption of containers for production workloads. In his session at 21st Cloud Expo, Ben McCormack, VP of Operations at Evernote, discussed how data centers of the future will be managed, how the p...
"Since we launched LinuxONE we learned a lot from our customers. More than anything what they responded to were some very unique security capabilities that we have," explained Mark Figley, Director of LinuxONE Offerings at IBM, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
The cloud era has reached the stage where it is no longer a question of whether a company should migrate, but when. Enterprises have embraced the outsourcing of where their various applications are stored and who manages them, saving significant investment along the way. Plus, the cloud has become a defining competitive edge. Companies that fail to successfully adapt risk failure. The media, of course, continues to extol the virtues of the cloud, including how easy it is to get there. Migrating...
Nordstrom is transforming the way that they do business and the cloud is the key to enabling speed and hyper personalized customer experiences. In his session at 21st Cloud Expo, Ken Schow, VP of Engineering at Nordstrom, discussed some of the key learnings and common pitfalls of large enterprises moving to the cloud. This includes strategies around choosing a cloud provider(s), architecture, and lessons learned. In addition, he covered some of the best practices for structured team migration an...
In his session at 21st Cloud Expo, Raju Shreewastava, founder of Big Data Trunk, provided a fun and simple way to introduce Machine Leaning to anyone and everyone. He solved a machine learning problem and demonstrated an easy way to be able to do machine learning without even coding. Raju Shreewastava is the founder of Big Data Trunk (www.BigDataTrunk.com), a Big Data Training and consulting firm with offices in the United States. He previously led the data warehouse/business intelligence and B...
Smart cities have the potential to change our lives at so many levels for citizens: less pollution, reduced parking obstacles, better health, education and more energy savings. Real-time data streaming and the Internet of Things (IoT) possess the power to turn this vision into a reality. However, most organizations today are building their data infrastructure to focus solely on addressing immediate business needs vs. a platform capable of quickly adapting emerging technologies to address future ...
In his general session at 21st Cloud Expo, Greg Dumas, Calligo’s Vice President and G.M. of US operations, discussed the new Global Data Protection Regulation and how Calligo can help business stay compliant in digitally globalized world. Greg Dumas is Calligo's Vice President and G.M. of US operations. Calligo is an established service provider that provides an innovative platform for trusted cloud solutions. Calligo’s customers are typically most concerned about GDPR compliance, application p...
You know you need the cloud, but you’re hesitant to simply dump everything at Amazon since you know that not all workloads are suitable for cloud. You know that you want the kind of ease of use and scalability that you get with public cloud, but your applications are architected in a way that makes the public cloud a non-starter. You’re looking at private cloud solutions based on hyperconverged infrastructure, but you’re concerned with the limits inherent in those technologies.
Is advanced scheduling in Kubernetes achievable?Yes, however, how do you properly accommodate every real-life scenario that a Kubernetes user might encounter? How do you leverage advanced scheduling techniques to shape and describe each scenario in easy-to-use rules and configurations? In his session at @DevOpsSummit at 21st Cloud Expo, Oleg Chunikhin, CTO at Kublr, answered these questions and demonstrated techniques for implementing advanced scheduling. For example, using spot instances and co...
DevOps is under attack because developers don’t want to mess with infrastructure. They will happily own their code into production, but want to use platforms instead of raw automation. That’s changing the landscape that we understand as DevOps with both architecture concepts (CloudNative) and process redefinition (SRE). Rob Hirschfeld’s recent work in Kubernetes operations has led to the conclusion that containers and related platforms have changed the way we should be thinking about DevOps and...
The need for greater agility and scalability necessitated the digital transformation in the form of following equation: monolithic to microservices to serverless architecture (FaaS). To keep up with the cut-throat competition, the organisations need to update their technology stack to make software development their differentiating factor. Thus microservices architecture emerged as a potential method to provide development teams with greater flexibility and other advantages, such as the abili...
Product connectivity goes hand and hand these days with increased use of personal data. New IoT devices are becoming more personalized than ever before. In his session at 22nd Cloud Expo | DXWorld Expo, Nicolas Fierro, CEO of MIMIR Blockchain Solutions, will discuss how in order to protect your data and privacy, IoT applications need to embrace Blockchain technology for a new level of product security never before seen - or needed.