How do you know when a DevOps strategy has succeeded or failed?

Sanjay Poonen

COO at VMware

VMware has been fortunate to have a ringside view of DevOps evolution across the marketplace over the last five-plus years. We measure success, or lack thereof, across DevOps using our simple 5S framework, which has been very effective in an objective way, and is modeled after the best practices of some of the industry's best talent.

The first S is for Speed — How fast is a business idea translated into working and deployed code in production? Best-in-class teams deploy to production many times a day while reducing lead time for app infrastructure to just minutes.

The second S is for Stability — How quickly can a customer bring back an impacted business-critical application? Best-in-class teams can do that in less than 30 minutes.

The third S is for Security — Security is intrinsic at all levels in the software stack. The most secure organizations have Zero High CVEs across the entire app stack.

The fourth S is for Savings — Lowering the cost run to the same application is a sign of automation at all levels. It is possible to shrink costs by as much as 80% in the most efficient models.

The fifth and final S is for Skills — DevOps is more about the people than tools and processes. The best companies will focus on growing employee skills and fostering a culture of curiosity and learning.

Abby Kearns

CTO at Puppet

Experience has shown that the technical practices of DevOps are important, but if those practices are isolated to just a few teams, this simply isn't enough to help organizations achieve a successful broad-reaching, long-term DevOps strategy. Ultimately, there must be meaningful cultural shifts across an entire organization for a DevOps strategy to succeed. Here are a few key things organizations should look for when measuring their success with a DevOps strategy.

At the end of the day, DevOps is really about enabling collaboration. First, does your team have the tools and technologies necessary? Specifically, do developers have access to a self-service platform that provides a foundation for automation, standardization and team autonomy? This is key to ensuring teams can operate more independently, efficiently and, most importantly, quickly.

Second, do you have a platform team with a product mindset? The platform team provides the organization with the infrastructure, deployment pipelines and other internal services that enable internal customers — usually application development teams — to build, deploy and run applications. The most successful platform teams treat their platform like a product, meaning they build a roadmap, receive feedback from their consumers, establish metrics and continually engage with their consumers to make their platform — their product — better.

Finally, are there cross-functional teams organized around a common goal or outcome? This part is the most important, and usually the most difficult as it requires making organizational changes, as well as establishing common and consistent practices across new teams. New communication and collaboration patterns are hard to create, and can often slow down the progress as organizations try to move DevOps practices beyond an initial team.

While not inclusive of everything, these are a few identifiers that speak to the future success of a DevOps strategy across an organization.

Dave McJannet

CEO at HashiCorp

The true measure of the efficacy of your DevOps strategy is fairly simple: How long does it take for your organization to deploy applications or application updates that see live traffic? That requires a handshake between the "dev" side and the "ops" side of the organization.

An organization that does this well has generally decomposed the problem: On the "dev" side, this means standardization on a manageable number of CI/CD pipelines that allow for the safe and consistent development and delivery of the application artifact. And on the "ops" side, there must be an implicit sign-off from the operations, security and networking teams that ensure if an app is delivered in a particular manner, it will see traffic as soon as deployed. Much of the focus is understandably on the "dev" side of the domain, but, in fact, the constraint for most organizations is the "ops" side.

For those that do it well, operations teams provide codified infrastructure templates that can be spun up on demand. Security teams provide a centralized identity-based security model for secrets. And the networking teams use service networking to enforce networking rules fully in software.

Most large enterprises I speak with have done a good job of solving their operations and security issues, but are still reliant on older networking models and that's where they often stumble. It doesn't matter how fast you can deploy the app if you're waiting for two days for someone to manually update the firewall and networking rules.

Thevi Sundaralingam

Director of Product Management, Edge Software Hub at Intel

Product competitiveness and customer satisfaction are the barometers of a successful DevOps strategy. DevOps capabilities are integral to modern software product strategy and business goals, but the strategies can fail when implemented in a silo, without alignment to the product roadmap, or when they are not measured through product performance indicators.

At the Intel® Edge Software Hub product team, our product vision to provide countless software packages for edge computing use cases and deliver easy-to-use and up-to-date content is central to our market competitiveness. However, these packages integrate software modules from tens of publishers, making it challenging to stay current. A scalable DevOps strategy is crucial to our vision and delivering on our commitments. We integrated the DevOps function within our development scrum team and invested in infrastructure, tools, processes and training to scale from day one. We also collect feedback from the sprint retrospective and integrate it into our roadmap and minimum viable product definitions.

Edge computing is an emerging market that combines complex software technologies from the Internet of Things, software-defined networks, and the Cloud; thus, a simplified developer experience and faster response time are essential in delighting our developers. We benchmark our response time to new updates and continuously improve our DevOps while also investing in modular and microservices-based architecture across the developer community to create a high-velocity edge computing software ecosystem.

Learn more.

Nicole Forsgren

Vice President of Research and Strategy at GitHub

A successful DevOps strategy adapts to an organization's current conditions and improves dynamically. These strategies don't offer predefined steps, where teams follow a maturity model telling them they're done once they reach Level 5 — these approaches leave companies vulnerable to competitors who keep improving. The pandemic demonstrated this, as organizations of all types and sizes pivoted almost overnight to survive unpredicted conditions.

DORA's research spans thousands of organizations, and shows a good DevOps strategy includes technology, process, and culture. Knowing what success and failure look like will help organizations know if they're on the right path:

Doing it right looks like:

• Delivering value to your stakeholders, not only focusing on things like "lines of code"
• Shipping software with both speed and stability
• Improving the right things at the right time by focusing on constraints, not just following a prescriptive path for every team regardless of their challenges

Doing it wrong looks like:

• Shipping software happens infrequently, and often causes failure
• Delivering software that requires brute force and results in burnout
• Focusing only on tech and ignoring important process and culture improvements

Using good technology and practices, teams can continue to develop and deliver software from anywhere. Our recent 2020 Octoverse Report backs this up: When work shifted to home, developers around the world were able to continue working, and even worked more. There is no gold standard for success, but broadly speaking, a good DevOps strategy results in great software, all while empowering teams to do their best work.

Wendy M. Pfeiffer

CIO at Nutanix

At the heart of DevOps is the ability to continuously use code to create, operate and manage infrastructure as part of a software service. Because this entire operation is software-defined, this work can be performed iteratively and remotely. Over time, a company that is pursuing a DevOps strategy should see reduced delivery cycles, increased use of automation, improved resource capacity and improved access to self-service for all of those who utilize technology within the ecosystem.

Within my IT team at Nutanix, for example, we began measuring baselines in these areas as we began several of these initiatives three years ago. Today, well over 80% of our service incidents are detected and resolved autonomously — a significant increase as compared to our starting point of around 5%. This has led to a 30% increase in our team member capacity and, in some cases, a 10x improvement in our delivery times.

In true DevOps fashion, I don't advocate that you attempt to create a "perfect" DevOps strategy at the outset. Expect to collect and measure the data generated from your operations, and use that data to iterate on your strategy to meet the unique and changing requirements of your organization.

Ken Gavranovic

Head of Platform Strategy, Engineering, DevOps and Solutioneers at Unqork

To understand how successful your DevOps strategy will be, ask yourself: Does my organization talk about which teams "Have the DevOps"? Are we measuring success in terms of tasks completed?

If the answer to these questions is yes, failure is soon to follow.

In a recent Gartner poll, 88% of enterprises said their DevOps initiatives are not delivering everything they hoped for. Too many enterprises think the biggest stumbling block to DevOps success is simply bridging the gap between development and operations. They measure success with task-based metrics, such as "every team has an automated build pipeline," or "all dynamic workloads have been migrated to the cloud." The assumption is that if tasks are being completed, the strategy is working.

What those metrics won't tell you is whether DevOps is actually delivering value to the business and its customers. To truly understand how effective the strategy is, you must prove a positive business impact in terms of internal and external key results. External metrics include: better customer NPS scores, higher reliability, enhanced speed, better software, and faster enterprise agility and innovation. Internal metrics include: substantial productivity gains for critical functions, meaningful cross-team alignment and accountability, continuous delivery with confidence and higher eNPS scores from DevOps employees.

Measuring success against these metrics will help you understand how effective the strategy really is. From there, you can adapt the strategy to better deliver business value, and join the 12% of enterprises that get everything they want from DevOps.