When Anthony Cardone was first hired at Shutterstock a little under four years ago, his role was supposed to be program management—to work with teams getting new features out the door. A component of his job was supposed to be coaching teams about how to improve their working methods and efficiency.

After becoming one of a small set of Jira administrators, Cardone started ripping through the records in Jira to find trends and understand how teams were working. “That’s how it all started,” Cardone says. “We started realizing more ways to improve how teams were using Jira and other tools. We started seeing certain good practices some teams used. They’d write ‘how we work’ documents about how the team should work during a sprint. We saw that if we could just teach these good practices across teams we could scale this up and start seeing efficiencies.”

Prior to his arrival, most planning was done by business unit. This meant that the plans and styles of work were specific to each unit, with little commonality in the ways plans were tracked and reported, and limited sharing about what was working process-wise, and what wasn’t.

“The main value of this role comes from identifying pain points within your teams, and across teams,” Cardone says. “Then you start pulling, analyzing and using data to find underlying causes, measuring and confirming—or changing—your plan after testing your hypothesis, then implementing process improvements based on this plan, and repeat.” It’s an end-to-end view of engineering operations.

From gut feel to data-driven

Cardone was the first into a new team dedicated to product and tech operations. He and his manager started diving into the Jira data more, trying to decipher which teams were hitting their targets and which teams weren’t… and why. “In the absence of data, leadership makes decisions based on gut,” Cardone explains. “When a team keeps missing deadlines, the gut reaction for some types of managers is that the team is slow.”

After crunching the numbers, Cardone was able to see that a team that perpetually missed deadlines wasn’t necessarily underperforming. Sometimes it was because the team always planned too much. Sometimes it was because other teams were throwing them too many requests, and the distractions prevented them from focusing on the planned work. Leaders started using these metrics to make more informed decisions, and asked for more. The more they dug, the more questions they had. Every time they looked at a metric, they would ask, is this relevant? Is it a symptom or a cause?

“The first metrics we released publicly for the whole company was planned work versus unplanned work,” Cardone explains. “When you said you were going to do ten things, and at the end of the sprint you had done ten things, how many of those ten were part of the original plan?” Shortly thereafter, Cardone started tracking how much of a team’s time is spent developing new features versus fixing bugs versus background tasks.

“There’s always something uncomfortable when another person comes in and says they’re watching you and tracking metrics,” Cardone admits. From the beginning, there was clear communication from the CTO on down that this was not a punitive tool. In fact, Cardone says, it was really being used to address pain points being articulated by development teams, who sometimes couldn’t understand why they didn’t finish all the projects they had planned in a sprint. The goal is to get better continuously—including updating the metrics with something better based on feedback.

Metrics are a means, not an end

Although Cardone uses data to understand how teams are doing, his job isn’t just about crunching numbers. In many cases, the numbers on their own aren’t enough to understand what’s going on—they are a starting point to have a conversation with the team about why, for example, the team only gets 30% of its planned sprint work done. Talking with teams can also help Cardone understand whether or not the metrics he’s looking at are relevant. Most importantly, it’s his job to translate the metrics into actionable recommendations to help teams plan better.

“When a team said, ‘we’re going to deliver this in three months,’ we’d look at the team’s planned versus unplanned work breakdown and ask them, ‘do you realize that two thirds of the work you do isn’t new features, so you’ll only be able to work on it a third of your time,” Cardone explains. “It was like a light bulb moment. Getting this awareness earlier allows teams to provide better, more realistic ideas of how their work goes and when they can deliver.”

As Cardone works with teams, the goal isn’t to compare team A to team B, because different teams have different roles and one might be designed to tackle more unplanned work. It’s about analyzing trends of a team over time, but it’s also about working together to see what works and what doesn’t. One of the challenges, Cardone says, is that while it’s easy to see when something’s wrong—say, a team only completing 20% of what it set out to—it’s hard to pin down a ‘right’ answer.

Coaching can be relatively high-level, where Cardone discusses numbers with the team and the team works to improve. “Then, there’s what I like more, which is in-depth coaching, where I embed with a team for a couple days,” Cardone says. After spending a couple days seeing how the team works, attending their meetings, and asking questions, Cardone will work with the team to come up with short-term and long-term actions to implement.

The return

The process isn’t necessarily easy. Organizing and sorting through the Jira database is a slog, and connecting all the tools whose data Cardone and the team need is time-consuming. There’s also the interpersonal challenges: It’s uncomfortable to feel watched, and people often candidly tell Cardone that they don’t like or feel comfortable with him looking at their team’s numbers. But the metrics also show concrete results.

“Unplanned work is down. There’s better cross-team collaboration and communication,” Cardone says. “We run scrum-of-scrum meetings and release management meetings on the bigger projects where we get to facilitate and hear what’s the plan for the coming month? Those meetings show everyone what’s coming and it holds people accountable. We have more consistent deliverables, and if something’s going to be late, we know ahead of time. It’s a lot easier to react and adjust.”

Get the data science behind high-performance teams.