The Challenges in Adopting DORA Metrics for DevOps success
The content here is under the Attribution 4.0 International (CC BY 4.0) license
The advent of DevOps changed how businesses face the integration between developers and operations leading to a shift in the cultural mindset of organizations focusing on the outcome and collaboration to provide customers an increased outcome. As such, new challenges have arisen in the approach of getting the metrics and adjusting the direction of the software development challenges.
There are two main points that we can refer to in the history of DevOps: the first was the launch of Accelerate, the book that became a reference in the field, and the second one was the launch of The Phoenix Project, another book that exposes through examples how the day-to-day of business and its limitations with operations were. Through history, the three ways of working were introduced and quickly became another reference in the field.
The foundation laid by those two examples and many others left the software industry with a sense that more could be achieved and better outcomes could be realized through the change in the ways of working as well as adopting the 4 key metrics.
The metrics
The metrics Deployment Frequency, Change Failure Rate, Mean Time to Restore, and Lead Time were the metrics used to differentiate what successful businesses were achieving in comparison with less successful businesses. Those metrics, according to Accelerate, were derived from the capabilities of software development teams to deliver working software. Those turn out to be the four metrics that differentiate them.
Where the metrics comes from?
The starting point for the metrics is described in the book Accelerate, the methodology used and the data gathered to support those are described there.
Once those metrics were established, they became quickly the ideal metrics to be measured, and as such, the desire to automate them also increased. In that advent, open source projects that offered those features became popular.
- Metrik - A cross-platform measurement tool that pulls data out of CD pipelines and analysis the four key metrics.
- FourKeys - Platform for monitoring the four key software delivery metrics of software delivery
Despite efforts in the adoption of automated tracking for the metrics, the challenge is a mix of infrastructure and business. It is possible to automate part of it using open source tools, however, it will be most likely that some adjustments will be needed as each business operates in a different fashion.
The ways of working
Not only did the change in the metrics disrupt the software industry, but the ways of working also suffered a mutation to adapt in an attempt to deliver value for customers faster. Adapting to a single framework (like Scrum) does not provide the benefits and guidelines required to achieve such flow. There is a need to incorporate new practices that will allow such flow to be introduced in the value stream of software delivery.
- Test-Driven Development
- Postmortem analysis
- Hands-on infrastructure
The practices in addition to the agile framework of choice will also provide the foundation that is needed to achieve the differentiator for business when comparing them to the competition.
The challenges
Despite automation, the challenges on the technical aspect of it are blurry for newcomers in this process. As you might have realized, the promise is big and the desired outcomes are even bigger, promising high stakes for business once they adopt it. However, the details are where the challenges become interesting. Each of the metrics focuses on different aspects of the technological stack, leading to the need for integrated tools that can collect the metrics. Despite efforts in the open source projects, what is found is a lack of standardized fashion to collect the metrics due to the segmented solutions that software has.
The metrics rely on pipelines, cloud providers, and version control (at a minimum). For each of those, many providers exist, and for each project in the industry, the mix of technologies is unique.
Adopt a pipeline
The first challenge to be noted is that, if you are not using CI/CD for software, this should be the first step, regardless of the technological stack. For current standards, there are multiple vendors that can be used. Some of the notable ones are GitHub Actions, Jenkins, and GitLab.
Which cloud provider
The traceability of the metrics and the adoption rate is coupled with the metrics. At the same time, the metrics are coupled with the pipeline and the infrastructure used. In that sense, for each software project, adapting the automation might be required.
Commits standarization
Parts of the collection of metrics come from the git history. Git plays a central role in the process. It is the place where hooks get triggered to start pipelines, rollbacks, or deployments. In that sense, standardization of commits is needed. Doing so will enable the automation of pipelines based on them, helping the metrics to be collected.
Which platforms can help us?
Recently, I have come across the backstage plugin that collects the DORA metrics and displays them in a dashboard.