Pinpoint Engineering

The TL;DR from my session on AI and EngOps at AIDe...

This morning I had the opportunity to chat with software engineers and data scientists at the AI Dev World Conference on a topic I just happen to be v...

Pinpoint Book Club Reads: The Tyranny of Metrics

One of the most gratifying (and often frustrating) features of life in a software startup is the “everyone owns everything” philosophy. Borne of necessity as much as the inherent curiosity and industriousness we all share, this spirit of knowledge-sharing is a cultural value we honor at Pinpoint. In fact, we set a goal to enhance our knowledge-sharing across the company in 2020.

Why a Book Club?

From this aim came the idea of a book club as a platform for our team to explore and exchange ideas. We realized this activity could have several desired outcomes that contribute to our cultural goals, including:

  • Active communication across departments
  • Challenging our mental models and expanding thinking
  • Spurring creativity, innovation, and connection
  • Creating alignment through conversations we normally wouldn’t have

So, we assembled a diverse group across our sales, marketing, engineering, product, and data science departments, and began brainstorming what material to include.

What’s on Our Reading List?

Each book we read this year will help us think about our business differently, brainstorm solutions, or improve an aspect of how we work. Our first selection is already challenging our thinking around key product features. Let’s dig into our inaugural book review on The Tyranny of Metrics.

The Tyranny of Metrics at a Glance

Review by Nick Kvlatine

Not long after launching our product, we received feedback from engineering leaders who loved our aggregate metrics summarizing engineering performance, progress, and projections. Engineers, on the other hand, were skeptical. We heard concerns from some about the “Big Brother” effect, as they worried that metrics may fail to account for complexity, lack meaning, or even encourage undesirable behavior.

As engineers building a product for engineers, we were quick to assure users that the tools and dashboards in the Pinpoint app are built to help them work transparently and efficiently, but we wanted to dig deeper into this issue all the same.

A key demand we strive to meet is balancing the business need for understanding engineering performance while acknowledging the complexity of the work and the needs of engineers. The Tyranny of Metrics, from J.Z. Muller, would facilitate a discussion on how to tell good metrics from bad, and when to apply them.

Author and historian Muller applies a sociological lens to metrics and demonstrates how our appetite for data can quickly become problematic, even mirroring an addiction. While metrics can offer extraordinary benefits when they’re used properly, he asserts, they can also be designed poorly, interpreted manipulatively, gamed and corrupted by those with something to gain, and ultimately can become toxic to the cause they were designed to support.

What We Learned from The Tyranny of Metrics

While the book is not written for or about engineers, we identified several takeaways that can help us build better software. As we read, we asked ourselves, “How can we create a product that doesn’t suffer from the tyranny of metrics, but can instead help engineers become more informed and efficient, help managers become better coaches and leaders, and help executives become more empathetic and fluent in our language?”

Here are just a few of the concepts that resonated with our team:

  • Metric fixation happens when we become addicted to data, replacing judgment with numbers. Not everything that matters is measurable; and not everything that’s measurable matters.
  • Focusing on a single metric leads to imbalanced and undesirable results.
  • When hitting numbers becomes the sole focus, gaming the system runs rampant.
  • Bias is everywhere – from which metrics are valued over others, to how data is gathered, to how information is ultimately interpreted.
  • When rewards and penalties are arranged behind metrics, it’s an immediate red flag that puts accuracy into question.
  • When developing performance metrics, it’s important to consider which can drive intrinsic versus extrinsic (such as monetary) motivation.
  • Measuring outcomes based on inputs is tricky, especially for a software product. Our output is disconnected from the outcomes of business stakeholders prize, so there may be many competing definitions of “quality.” We can decrease bugs and measure customer satisfaction, but who’s to say which metrics are the definitive source?
What Nick found most interesting:

How widespread the misapplication of metrics is and the ingenuity people show in subverting them. From police underreporting and misclassifying crimes to surgeons only taking on the cases most likely to success, the abuse of metrics reaches across any place metrics are being used.

Despite all this, the author made it clear that there was still a place for metrics to be judiciously applied.

Building Better Software with Meaningful Metrics

These challenges are exactly the issues we think about as we continue to expand and improve Pinpoint. The Tyranny of Metrics has helped us further shape these fundamental principles of our approach to building software and metrics for engineering:

  • Simply bringing data together across varying tools and systems delivers significantly more insight than siloed analysis. Beyond understanding how one variable affects a single factor, aggregate data science delivers a “1 + 1 = 3” effect.
  • We can use trends in data, and even outliers, to improve processes that have a larger organizational impact in facilitating efficiency and shaping desired behavior.
  • Team and organization performance is often a more meaningful and intrinsic indicator than individual performance.
  • Making data visible to the people producing the work – engineers – is critical, because they have the power to identify and enact solutions.
  • Well-designed performance metrics can help individuals understand and celebrate their contributions to team efforts, as well as provide a basis for personalized feedback from managers to improve each individual’s skill set.
  • OKRs can connect company, team, and personal goals to measurable results. This ensures that everyone is running in a unified direction while keeping engineers interested and demonstrating their impact on larger goals and outcomes.
How Nick plans to apply this insight:

Thinking more carefully about when we choose to use metrics. There were such a range of interesting examples of abusing metrics, it has helped me to imagine ways in which the work I'm doing might be manipulated. We've always had the goal to make software development better for developers, so the things I learned will help me to think about the best ways to do that.

Related Post:

The TL;DR from my session on AI and EngOps at AIDevWorld

This morning I had the opportunity to chat with software engineers and data scientists at the AI Dev World Conference on...

Our 12 go-to Python libraries for data science

We use data science — machine learning, natural language processing, etc. in Pinpoint to correlate data from all the too...

Two data science life hacks to improve your workflow

Data science is fundamental to Pinpoint’s application. But, like most startups, we are still in the process of building ...

Subscribe to the Pinpoint Engineering Blog

cat developer