For Infrastructure & Operations Professionals

April 15, 2015

Six Trends That Will Shape DevOps Adoption In

2015 And Beyond
Benchmarks: The Modern Service Delivery Playbook For 2015
by Amy DeMartine
with Eveline Oehrlich and Megan Doerr

Why Read This Report

Infrastructure and operations (I&O) organizations are in various stages of adopting the DevOps
methodology. Where does your organization fall in comparison with other organizations? This report
presents strategic benchmarks that I&O professionals can use to see where they are in their DevOps
adoption compared with their peers. These benchmarks represent a subset of data collected by our Q4
2014 Global Modern Service Delivery Benchmark Online Survey, which highlights six major trends.
Although the data tells us that I&O professionals have the essentials in place for success, DevOps execution
is lacking in some key areas. I&O professionals should use this data to determine what DevOps methods
to adjust, attack, or continue so that they can make significant headway in 2015 and beyond.
assess your Devops adoption by using the calmsS model
Organizations such as Amazon, Etsy, and Netflix have successfully adopted DevOps to achieve quicker and
higher-quality service and application delivery.1 However, the DevOps movement is in early days, and how
well I&O pros are adopting DevOps to achieve modern service delivery in comparison with their peers is
a common question Forrester fields.2 But how do you know what to benchmark against? Many modern
service delivery experts use the CALMSS (culture, automation, Lean, management and measurement,
sharing, and sourcing) model to mark the differences between old methodologies and new DevOps
methodologies.3 We are using this model to determine where companies are in their DevOps journey.
new service delivery methods ARE EVOLVING Too slowly
The age of the customer forces organizations to increase their release cycles of applications and services
to sustain or achieve better customer experience. DevOps is a set of practices and cultural changes
supported by automation tools and Lean processes that creates an automated software delivery pipeline,
enabling organizations to deliver better-quality services and applications faster to ultimately win, serve,
and retain their customers.4 From the results of the Q4 2014 Global Modern Service Delivery Benchmark
Online Survey, we found that six trends are currently dominating DevOps adoption.
Trend 1: The Foundation For Success Is In Place . . . Mostly
You must have certain foundational processes in place in relation to the CALMSS methodology to ensure
modern service delivery (AKA DevOps that leads to continuous delivery). One key requirement is an
integrated product team that delivers products or services.5 I&O professionals are engaging in several
positive trends for success; they are:

Headquarters
Forrester Research, Inc., 60 Acorn Park Drive, Cambridge, MA, 02140 USA
Tel: +1 617.613.6000 Fax: +1 617.613.5000 www.forrester.com

For Infrastructure & Operations Professionals

Six Trends That Will Shape DevOps Adoption In 2015 And Beyond

Open to relationships beyond I&O. Trust is the foundation of modern service delivery.

An integrated product teams successful interaction in the new life cycle depends on the
teams ability to openly communicate with and trust each other. We found that 78% of I&O
professionals strongly agree or somewhat agree that they trust their colleagues and believe
that their motivations are for the benefit of the business and customer. Collaboration is a key
to success, and 55% of respondents say that team collaboration is expected or expected and
rewarded. In addition, 76% of respondents state that they work in an environment in which new
ideas are welcomed by management.6

Ready to begin automation. I&O professionals are automating release, configuration, and

change management processes. The really encouraging fact is that all possible automation areas
are at least somewhat automated and that release, configuration, and change management
are starting to see full end-to-end process automation versus simple tasks or sub-processes.
Seventy-one percent of respondents say that their organization uses release management
automation. Within this group, 20% have fully automated the release process. The adoption of
change management is at 74%, where 13% within this group have fully automated the change
management process (see Figure 1). I&O pros are also using automation across the different
life-cycle phases of development, test, and production. I&O shows higher automation in its
traditionally owned phase of production, except where it interacts between various phases of the
life cycle, where 31% of the respondents are automating in testing versus 26% in development
and 24% in production (see Figure 2).

Willing to reach out across the integrated product team. I&O showed strong results in solving
production environment issues. As organizations invest in business technology to support
the customer life cycle, the responsibility for the availability, performance, and usability of
these customer-facing business services will be increasingly shared with other members of the
integrated product team. Sixty-four percent of survey respondents indicated that when an issue
affecting a customer occurs, they collaborate across the integrated product team either all of the
time or most of the time.7

Confident in their ability to source. Finding service partners or solution vendors to provide

expertise, enable missing functionalities, or simply reduce costs is typically the responsibility of
the I&O professional, and they believe they are successful at it. Additionally, 52% of respondents
believe that service partners or solution vendors that they regularly work with understand their
business needs.8

2015, Forrester Research, Inc. Reproduction Prohibited

April 15, 2015

For Infrastructure & Operations Professionals

Six Trends That Will Shape DevOps Adoption In 2015 And Beyond

Figure 1 I&O Has Begun Automating Key Processes

Please choose the following life-cycle stage(s) that are automated in relation
to customer-facing business services and applications.
Based on the types of automation that you identified that your organization uses,
please select the level of automation that most closely resembles your
environment today across all life-cycle stages.
Individual process
Release

Multiple processes

18

Configuration management

16

Change management

16

Interactions between 2
life-cycle stages

10

Workload adjustments
Knowledge collection

11
7

Full process

6
8

N = 30
N = 28

11

N = 31

N = 19

N = 20

9
14

N = 21

Base: 42 global I&O professionals

Source: Q4 2014 Global Modern Service Delivery Benchmark Online Survey
120481

Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited.

2015, Forrester Research, Inc. Reproduction Prohibited

April 15, 2015

For Infrastructure & Operations Professionals

Six Trends That Will Shape DevOps Adoption In 2015 And Beyond

Figure 2 I&O Uses Automation Across The Life Cycle

You have indicated that the following types of automation are used by your organization.
Please choose the following life-cycle stage(s) that are automated in relation to customerfacing business services and applications.

Development

Testing

Production

25
20
15
10
5
0

Release

Configuration

Change

Interactions Workload
Knowledge
between adjustments collection
various
life-cycle
stages

Base: 42 global I&O professionals

Source: Q4 2014 Global Modern Service Delivery Benchmark Online Survey
120481

Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited.

2015, Forrester Research, Inc. Reproduction Prohibited

April 15, 2015

For Infrastructure & Operations Professionals

Six Trends That Will Shape DevOps Adoption In 2015 And Beyond

Trend 2: Fear Of Failure Will Hamper Advancement

Although I&O professionals are willing to transition from yesterdays service delivery model to
todays DevOps methodologies and behaviors, several trends will hamper their success:

Fear and lack of accountability. Seventy-six percent of respondents either try to avoid failure,

see it as disruptive, or dont tolerate it.9 This attitude can prevent you from trying new methods,
making innovation more difficult. Failure is an opportunity to learn and improve.

Not sharing accountability for success or failure. Modern service delivery requires all teams

to feel accountable for the success or failure of customer-facing business services or applications
to ensure that everyone is driving toward the same goals, even if other teams are making the
decisions. However, only 45% of respondents feel this is true for them.10

Not using knowledge to improve the life cycle. Modern service delivery success relies on

collaboration across the life cycle and feedback and feed-forward loops; however, 57% of
respondents state that they either sometimes, not that often, or never get invited to attend the
design, planning, and release activities.11 This will prevent organizations from truly optimizing
the life cycle.

Trend 3: Monitoring And Analytics Strategies Must Make A Big Leap Forward
Monitoring and analytics provides a single source of truth from which you can predict performance
issues of an application or service. Unfortunately, I&O professionals do not trust monitoring and
analytics enough for it to be useful. The lack of usable real-time dashboards prevents I&O from
understanding and resolving issues before they affect customers or business users. Specifically, we
found that:

Monitoring and analytics lacks consistent usage. Its disappointing to see that only 14% of

respondents state that monitoring and analytics is always seen as a vital requirement by their
business in relation to the design and development of customer-facing business services and
applications. A key trait of modern service delivery is fast and high-quality releases. To maintain
quality, solutions to measure the customer expectation gap should be designed into the product
from the requirements phase by the integrated product team, not implemented ad hoc during
delivery. In addition, 50% of respondents state that they have no dashboards available to them
that contain real-time monitoring and analytics information, or if they do, they are updated
infrequently in regard to release and change pipelines.12

Many organizations lack a single source of truth. When the tech management organization

uses multiple or no tools to monitor business services or applications, I&O professionals

working with application developers dont know what data to trust to help pinpoint problems
customers are experiencing. Leveraging a single tool so that I&O and AD&D professionals

2015, Forrester Research, Inc. Reproduction Prohibited

April 15, 2015

For Infrastructure & Operations Professionals

Six Trends That Will Shape DevOps Adoption In 2015 And Beyond

can monitor and analyze performance and availability of business services or applications
is essential for creating a single source of truth. However, at best, 45% of respondents trust
their monitoring strategy only sometimes.13 Integrated product team members cant pinpoint
problems quickly without a single source of truth.

Customers are finding problems. Nearly a quarter of respondents (24%) say that I&O teams

are very frequently notified of problems or incidents around business services or applications by
their business customers. The majority (55%) of respondents say that customers notify I&O of
problems or incidents occasionally.14 This lack of proactive monitoring and measurement leads
to unpredictable customer experience and has significant effects on the organizations ability to
stay competitive.

Trend 4: The Focus On Customer Experience Is Not Second Nature . . . Yet

The successful adoption of modern service delivery will be measured by how well the integrated
product team delivers the applications and services to improve customer experience. Unfortunately,
I&O professionals report that they dont know whether their firm measures customer experience and
dont have shared common goals to meet customer experience objectives. Over a third of respondents
(38%) state that they dont know if their firm assesses whether customer business services and
applications meet customer experience goals. Furthermore, 43% of respondents either state they are
not measured on common goals related to customer experience or dont know if they are.15
Trend 5: Change And Release Processes Are Not Delivering Business Needs
The goal of modern service delivery is small, fast, high-quality releases of application and services.
While tools are in place for rapid release and change automation, you should authorize and
implement low-risk changes and releases quickly. There will be delays in the release process if every
release and change needs authorization. Currently, 48% of our survey respondents disagree with the
statement that only high-risk changes require approval. Over half of respondents (57%) state that
their business is either very dissatisfied or somewhat dissatisfied with the time it takes to release new
features or changes.16
Trend 6: You Must Prioritize And Focus Sourcing Strategies
Today I&O professionals may work with a variety of vendors, such as cloud service and outsourcing
providers. The complexity driving todays modern applications design and delivery, which includes
mobile applications, means that you cant achieve modern service delivery by yourself. Tech
management requires a sourcing strategy that identifies, defines, and builds strategic partnerships with
vendors supporting all stages of the life cycle. Currently, I&O pros find that their sourced vendors are:

2015, Forrester Research, Inc. Reproduction Prohibited

April 15, 2015

For Infrastructure & Operations Professionals

Six Trends That Will Shape DevOps Adoption In 2015 And Beyond

Not strategic. To be a strategic partner, a sourced vendor must support business innovation
by building new solutions and improvements into the modern service delivery life cycle.
Unfortunately, only 43% of respondents state that their relationship with service partners or
solution vendors is a strategic partnership.17

Not integrated into the product team. Vendors must be a part of the integrated product

team and must be held accountable for service design, plan, build, and run activities. Only
29% of respondents either somewhat agree or strongly agree that service partners who they
regularly work with are integrated into processes for service development, support, and delivery.
Furthermore, only 28% of respondents either strongly agree or somewhat agree that the external
service partners or solution vendors that they regularly work with are accountable for service
design, plan, build, and run activities.18

R e c o m m e n d at i o n s

Accelerate your Devops practices now

Now is the time to discover where your organization falls relative to other organizations so you can
make cultural and practical process and technical changes. I&O professionals can take the following
next steps to gain traction in their DevOps practices:

Baseline your current state. Ask your own integrated product team to rate itself against

questions from our survey, and use the outcome to determine the current state of your
DevOps methodology or initiative. This will provide you with a baseline for applying
improvements to your culture, automation, Lean, management and measurements, sharing,
and sourcing practices.

Prioritize your improvements. Prioritize improvements to your DevOps practices by

finding where you deviate the most from your peers, which will tell you where your greatest
weaknesses are. This will help you gain traction and confidence quickly in targeted areas.
You can also find one issue to address in each area of the CALMSS model to help you
improve your DevOps practices holistically.

Measure your future state. When youve made improvements, ask your integrated product
team to rate itself again. Refer to your baseline to show how improvements have affected
your results.

2015, Forrester Research, Inc. Reproduction Prohibited

April 15, 2015

For Infrastructure & Operations Professionals

Six Trends That Will Shape DevOps Adoption In 2015 And Beyond

Supplemental Material
Survey Methodology
Forrester hosted the Q4 2014 Global Modern Service Delivery Benchmark Online Survey to 400
individuals. These individuals work in IT roles (not including application development, enterprise
architecture, security, or sourcing/vendor management) and are involved with customer-facing
business services/applications for their opinions on the current state of modern service delivery
and DevOps. These respondents were narrowed down to those in managerial positions in IT/tech
companies, worldwide, with over 1,000 employees.
Endnotes
For more information on how to execute DevOps methodologies, please see the The Seven Habits Of
Highly Effective DevOps Forrester report.

For more information about what is new in modern service delivery, please see the What Makes Modern
Service Delivery Modern? Forrester report.

For more information on the evolution of the CALMSS model, please read the following blog. Source:
Eveline Oehrlich, DevOps Now With CALMSS, Eveline Oerhrlichs Blog, March 2, 2015 (http://blogs.
forrester.com/eveline_oehrlich/15-03-02-devops_now_with_calmss).

For more information about Forresters definition of the CALMSS model, please see the What Makes
Modern Service Delivery Modern? Forrester report.
To learn more about how to effectively implement DevOps and explore best practices, see the Modern
Application Delivery Demands A Modern Organization Forrester report; see the The Seven Habits Of
Highly Effective DevOps Forrester report; and see the The Eight Tenets Of Faster Application Delivery
Forrester report.

For more information on integrated product teams, please see the Playing Musical Chairs For Staffing
Modern Service Delivery Forrester report.

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

10

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

11

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

12

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

13

2015, Forrester Research, Inc. Reproduction Prohibited

April 15, 2015

For Infrastructure & Operations Professionals

Six Trends That Will Shape DevOps Adoption In 2015 And Beyond

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

14

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

15

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

16

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

17

Source: Forresters Q4 2014 Global Modern Service Delivery Benchmark Online Survey.

18

Forrester Research (Nasdaq: FORR) is a global research and advisory firm serving professionals in 13 key roles across three distinct client segments. Our clients face progressively
complex business and technology decisions every day. To help them understand, strategize, and act upon opportunities brought by change, Forrester provides proprietary
research, consumer and business data, custom consulting, events and online communities, and peer-to-peer executive programs. We guide leaders in business technology,
marketing and strategy, and the technology industry through independent fact-based insight, ensuring their business success today and tomorrow.
2015 Forrester Research, Inc. All rights reserved. Forrester, Forrester Wave, RoleView, Technographics, TechRankings, and Total Economic Impact are trademarks of Forrester
Research, Inc. All other trademarks are the property of their respective owners. Reproduction or sharing of this content in any form without prior written permission is strictly
prohibited. To purchase reprints of this document, please email clientsupport@forrester.com. For additional reproduction and usage information, see Forresters Citation Policy
120481
located at www.forrester.com. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change.