Editor’s note: This article was originally published in 2017. We’ve updated it to be more relevant.

Back in the bad old days of battleship grey UIs, rounded corners made up with GIF files and “Best Viewed With Netscape Navigator 3.0” badges, businesses just wanted their software teams to ship – quality be damned!

In today’s modern software world, things are much different. Technology leaders are expected to deliver on time.

Software development teams are also expected to meet modern demands:

1. Quality software that doesn’t crash
2. Software that is fast
3. Software that communicates how users are using the software, and what their experience is
4. That the software delivery process is repeatable, and the team is agile

Competition is fierce in the software industry, team members are expensive, and doing great work keeps the lights on.

So which software team KPIs would lead to delivering better end products?

What gets measured, gets managed

It’s important to measure software quality metrics to help the entire team manage the user experience.

Let’s get real for a second: your team is employed because you have customers. If your software is terrible, you will lose customers. (71% of users will stop using software after an error!)

At Raygun, we see many software teams who come from no system at all. They’ve had issues, and appreciate that Raygun would help streamline their processes, make customers happier and grow their businesses.

It’s understandable – our industry is maturing, and expectations increase with maturity.

What should you measure, and how can you measure it?

At Raygun, we hold the senior development team members accountable to several metrics that Raygun can automatically track:

1. Users affected by bugs
2. Median application response time
3. P99 application response time
4. Resolved bugs => new bugs

We’ve all been in situations where a metric was important, but it was hard to compute. What happens? You stop tracking it because it’s too hard (hey, there’s a reason we like computers right?)

Here’s how we measure software quality at Raygun.

1. Users affected by bugs

Frankly, error counts are misdirection. If you have 10,000 errors that affect one customer, it’s not as bad as 500 errors affecting 250 customers. Measure the affected customers every month, with a goal to reduce.

At Raygun, we like to use our Users dropdown to sort error counts by the number of users affected. That way, we can prioritize our customers.

2. Median application response time

Forget averages, they lie. The median response time is what 50% of your customers experience. Track that, and hold the team accountable for achieving at least 3 seconds (or faster) for important pages. Performance makes money – 40% of users will leave a website that takes more than 3 seconds to load!

You can use a few tools to measure median application response times. Our frontend team uses real user monitoring because it’s helpful to see which page assets are causing the problem.

3. P99 Application response times

Medians are great, but we also need to appreciate the upper limit. We choose to track the P99 – the time taken for the 99th percentile of users. This will normally be slow, but we want to make sure it’s more like 5 seconds slow, not 25 seconds slow. We don’t often track P100 as it’s the domain of total timeouts, bad bots that hold connections open and is generally misleading about real users.

If you’re working on larger, more popular websites, you’ll want to maintain these load times even during periods of high traffic (say if you work with e-commerce and you need to ensure site stability over Black Friday sales weekend.)

4. Resolved bugs >= new bugs

The team should be fixing bugs at least as quickly as they are creating them (in a brownfield project, ideally resolving more than are created).

At Raygun, we use error grouping to group errors by their root cause which makes it much easier to manage bug count, (not the crash count, which is instances of a bug encountered). Measuring how many bugs are being introduced is tough without deployment tracking, so here at Raygun, we have it set up so all we have to do is hover over the deployment to understand how many bugs are new, resolved, and still occurring with each deployment.

Accountability

It’s not enough to just measure these metrics. If one thing has made a difference to our team, it’s surfacing these metrics on a dashboard TV display for the whole team to see, giving tech leads the ability to take a quick glance at software performance.

What about KPIs like monthly active users, geographic breakdown, and feature tracking?

These metrics are crucial to track but are typically tracked by product managers and marketing managers.

Building great software takes a lot of effort, but tracking the right metrics makes it easier. If you’re getting pressure from the board about software quality, track these metrics and you’ll be able to provide an answer.

Wrap up

These four metrics will make your software stand out as being best-in-class. It will oblige your software teams to:

1. Improving user experiences with fewer crashes
2. Enhancing user experiences with faster software
3. Reduce technical debt accumulation. It won’t fix all technical debt, but fast bug-free software is nearly always better to work on than slow buggy software

Without them, you’ll have a hard time improving the user experience of your application or proving the business value of investing dev resources into performance gains.