The ROI of OKRs: Sears Study Quantifies the Impact of OKRs
More and more people are embracing Objectives and Key Results (OKRs) at work. However, I’ve not found any research suggesting that the use of OKRs impacts financial results. Until now. So, what is the ROI of OKRs?
Chris Mason completed his Ph.D. in Industrial-Organizational Psychology at DePaul University. He is Sr. Director, Strategic Talent Solutions at Sears Holding Company (SHC). In the fall of 2013, Chris and his team led one of the largest deployments of OKRs to date. SHC rolled out OKRs to their entire salaried population of roughly 20,000 associates. Research based on the use of OKRs over the past 18 months at SHC suggests that OKRs can have a major impact.
Joe Kutter was the lead researcher who analyzed the data. Even a minimal use of OKRs led to higher levels of performance. Specifically, we found an 8.5% lift in hourly sales at a call center. Consistent use of OKRs led to an 11.5% increase in the chances of moving to a higher performance bracket across their corporate associate population. These may be the first experiments concluding that using OKRs makes an impact. This impact includes both personal performance as well as the organization’s bottom line.
I recently spoke with Chris and his team. He shared how they stumbled into something that made me feel a lot better about my chosen profession as an OKRs coach. This may be the first evidence that OKRs impact the bottom line!
How OKRs drove revenue 8.5%
Here’s a summary of our discussion.
Lamorte: I’ve not seen a single formal study addressing how and if OKRs improve an organization’s bottom line. Have you seen any such research?
Mason: To my knowledge, there is no research that demonstrates if, and to what extent, the use of OKRs actually improves an organization’s bottom line. Certainly, decades of research in behavioral sciences such as the work of Locke and Latham who developed Goal Setting Theory, suggests that setting shorter-term goals, giving people more autonomy so they commit to the goals, and including a stretch component, drives higher achievement. We could assume OKRs are effective based on this general research. However, I’ve not seen any research supporting that OKRs, as a specific methodology, is effective.
Lamorte: Given you knew of no formal analysis documenting the effectiveness of OKRs, why did you deploy OKRs? And why did you bother to develop an in-house software tool to track and monitor OKRs?
Mason: We actually were inspired to deploy OKRs based on the Google Ventures video in 2013. Our Chairman and CEO, Eddie Lampert, shared this video with senior leadership at SHC. That video and the continued leadership of our Chairman and CEO are the driving force behind OKRs at SHC. More recently, members of our HR leadership spoke with Google about their approach. While I don’t have all the details, my understanding is that they’ve not conducted any formal research into the effectiveness of OKRs. In the book How Google Works, the argument for OKRs is simply that they make sense and support their culture.
Lamorte: Describe your process for researching the effectiveness of OKRs.
Mason: After using OKRs for over a year, we are beginning to analyze the data. Preliminary results indicate that using OKRs does indeed create a positive impact. We have two studies so far. The first one happened by accident, which actually makes it a great experimental design complete with pre and post-measures and a control group. Our outbound call centers are divided into teams of outbound call agents at various locations. They focus on add-on sales, measured by metrics like the number of calls per hour and total sales per hour. OKRs were formally launched to salaried associates at the centers, but not to these outbound sales agents (though everyone was given access to the system that houses OKRs). For reasons that are not entirely clear, a portion of the call agents opted in. It’s particularly interesting since agents opting in were spread across multiple locations. They started with OKRs in various months, which helps control for any effects of location or seasonality in our data. We started OKRs in 2013. Over the next year and a half, agents opted in organically.
Lamorte: How did you compare the performance of each group of sales agents over time?
Mason: We went back to the moment when they first started using OKRs. We then compared their sales performance from the month prior to using OKRs with the month when they first entered OKRs and their first month after entering OKRs.
Lamorte: Can you summarize the results?
Mason: Yes. For the group who used OKRs, we saw an increase in their average sales per hour from $14.44 per hour to $15.67. This translates to an average increase of 8.5%. This increase is not only statistically significant, but practically significant. We also saw a similar increase in the number of calls they made per hour, which helps explain the increase in revenue. Most importantly, the control group who did not use OKRs averaged the same sales each month. In other words, the group that did not use OKRs did not show an increase.
Lamorte: Does this 8.5% result control for the possibility that agents who decided to use OKRs might happen to be the high performers?
Mason: Yes, in fact, we controlled for this. It’s not possible that agents opting in simply improve because they are high performers. The agents improved relative to their own performance. It doesn’t matter who you were or when you started. Any agent who used OKRs averaged an 8.5% lift in their own sales performance compared to their rate prior to starting with OKRs.
Lamorte: This may be the first study that measures the impact of OKRs on bottom line. If so, we’re talking about an 8.5% lift in revenue. This is a big deal!
How consistent use of OKRs drove performance
Lamorte: This might have major implications for getting more sales out of a call center. However, this is just one of the many groups of employees using OKRs. Do you have any data that indicates OKRs can help workers be more effective who are not in sales roles?
Mason: Yes, we next looked at the broader corporate population. Now the sample size here is roughly 12,000 workers across dozens of functional areas. This study is particularly interesting because we were also able to look across a longer time period. We can see how often people used OKRs between the late summer of 2013 and the summer of 2014.
Lamorte: Given you’re looking beyond a sales team, you can’t simply look at revenue per sales agent as an outcome variable. What outcome variable did you use to measure the general impact of OKRs on effectiveness?
Mason: We used the outcome of general performance level. We use a typical 9-box tool to classify this population on the dimensions of performance and potential on an annual basis. For this study, we looked only at performance. We classified everyone into either high performance (roughly top 20%), middle performance (middle 70-75%), or low performance (bottom 5-10%) in the summer of 2013 and again one year later in the summer of 2014. It is hard to move from medium performance to high performance. But if you can make this move, good things such as rewards and promotions are more likely to happen. We then categorized each worker into three groups based on their use of OKRs: 1) Consistent: entered at least one OKR consistently all four quarters of that year, 2) Inconsistent: used OKRs at least one time in the year, but failed to enter them every quarter, and 3) No use of OKR at all during the year.
Lamorte: So, did the use of OKRs impact a given worker’s chances of getting classified at a higher performance level?
Mason: Yes, and the results are very interesting.
- Group 3 – No use of OKRs showed no change in performance level.
- Group 2 – Inconsistent use of OKRs showed some improvement. Even just using OKRs once made it 3% more likely to move up a whole performance level. In this group, we did not even consider the quality of OKRs and we still saw a statistically significant, but small impact.
- Group 1 – Consistent use of OKRs were 11.5% more likely to move into a higher performance bracket.
Lamorte: OK, so what’s the bottom line conclusion here? Can we now say research supports that OKRs are effective?
Mason: We’re not saying we have definitive proof that OKRs works for everyone. However this second study may have some big implications. This result translates roughly to: “If you knew there was a tool out there that takes just a few hours each year to use which could increase your chances of high performance by 11.5%, wouldn’t you at least try it?”
Conclusion – The ROI of OKRs
Sears Holding Company has conducted research on the use of OKRs with 20,000 associates for the past 18 months. They found evidence that even minimal use of OKRs led to higher levels of performance. Results include an 8.5% increase in hourly sales at a call center and up to an 11.5% increase in the chance of high performance across their corporate associate population. These may be the first experiments concluding that the use of OKRs makes an impact. This impact appears to be at the personal performance level as well as the overall organization’s bottom line.
If you know of any research on the effectiveness of OKRs or any other specific approach to Enterprise Goal Management, please share it here!