Highlights from the Raygun and AWS tech leaders' panel: Closing the gap between code and customer

The Tech Leaders’ Tour is a series of events bringing tech leaders together to learn from each other about improving software quality and customer experience.

“How much of our time are we spending on building new features versus tending to and improving what we have, versus making sure we’re investing enough to add the capabilities that teams need to scale, grow and deliver things with high quality?” Asks Josh Robb, SVP Product and Engineering at Pushpay, to over 60 tech leaders last week in Auckland.

It’s a relevant and timely question—to stay ahead, leading product teams must have a laser focus on their end users. But in the fast-paced world of engineering and development, how can we achieve a balance between product delivery and preserving software quality?

We attempt to answer this question and more in our inaugural interactive panel. Here’s a look at three key learnings from how Xero, Vend, and Pushpay prioritize user experience and performance in their product development workflow. There were plenty more, so make sure to watch the full discussion below.

Measure and communicate technical debt through a different lens

“When I first joined Xero, I kept hearing about technical debt and the monolith, but I’m thinking, this company is 10 or 12 years old, so how come (that’s the case)?” Starts Simona Turin, Executive General Manager of Product at Xero. This challenge is one that many tech leaders are looking to tackle. So how do top-performing teams balance the release of features while maintaining excellent software quality?

At 9:55 in the panel recording at the start of this article, Josh Robb, SVP Product and Engineering at Pushpay offered some insight. “For me, it’s a portfolio allocation decision that starts with asking ‘How much time am I willing to spend on quality?’” Josh continues, “There are two ways of looking at it—one is to capture actuals, and the other way is to think about it from a strategy perspective, in terms of where we want to invest this quarter or this year.”

But how do teams measure technical debt? Most agree that technical debt can’t be measured itself, but the impact of that debt can be. Josh suggests that instead, we look at technical debt through the lens of “performance, reliability, scale and security,” which opens the door to engineering time being spent on software quality metrics like NPS and the number of software errors.

This perspective also lends to better communication with the executive team to avoid the negative connotations the term ‘technical debt’ brings.

Strengthen the relationship between customer and engineering

The central theme to the panel was bringing code closer to customers, so naturally, we asked for insight into the most significant practice that tech leaders implement to bring engineers closer to the end user.

At 23:17, Cara Fonseca-Ensor, Director of Product Engineering at Vend, reinforces that in her company, delighting the customer is at the core of their success. Therefore, it’s an essential part of Vend’s development workflow to take the time to talk to customers and get their feedback, which is then collected and displayed in Redash. The product team then uses that data to inform the product direction. At 19:12, Simona explains that Xero goes a step further and unites teams under both SLOs (Service-Level Objectives) and OKRs (Objectives and Key Results) that reinforce customer happiness and product usage.

Josh builds on this point further by suggesting that engineering ideas that have no connection to the customer are dangerous. “In my view, everyone who’s working on software should know why they are working on a particular feature, they should have someone that can articulate to them not just, ‘Do the thing!’, but ‘This is the underlying need that needs to be fulfilled.’”

At 40:01 minutes into the panel discussion, an audience member suggests one way to strengthen this relationship is to get engineers answering support tickets in their first week. “It just gives them so much more context about their jobs like, ‘Why are we building this?’ Keeping in touch with customers is really, really key for engineers,” concludes the audience member.

Software quality is part of company culture

A common thread throughout the discussion is that software quality must be aligned with culture, and that starts with the hiring process. Our panelists look for qualities like caring about the customer experience in a candidate’s first interview. Cara from Vend says at 23:49 minutes into the panel discussion, “We hire good engineers who care about this stuff.”

Company-wide internal values are also helping Xero, Vend, and Pushpay to keep the emphasis on the customer.

It was agreed among the panel and the audience of over 60 tech leaders that there needs to be metrics and goals in place to keep the whole company accountable. OKRs are the tool of choice for our panel. But how do most tech leaders measure software quality?

We asked our audience, and the results are in the graph below:

Empowering engineering teams with the right metrics that focus on performance and reliability is vital to Pushpay’s success.

“Reliability is super important. Consumers have high expectations these days that apps will work reliably." Josh Robb, SVP Product and Engineering at Pushpay

“When you open something on your phone, and it crashes, it will get uninstalled quickly, and there is plenty of empirical data to support that. For apps that take more than three seconds to open are 60% likely to be uninstalled within five days. So even just performance is really important.”

Simona also echoes the emphasis on the customer at Xero, “I think every company has to start balancing how many features they push. At Xero, (We’re looking) for a balance of customer focus and performance.”

Monitoring performance, software quality, and customer experience matters

If the learnings from these tech leaders can be summarized under one theme, it would be that software quality and performance for end users will continue to be a challenge for tech leaders.

As we’ve learned from our esteemed panelists and audience members, closing the gap between code and customer is a matter of identifying the right KPIs around software quality, enabling good communication between teams and real customers, and empowering a strong performance culture with great hires and strong company values.

Make sure to watch the full talk. It was full of provocations about how to improve and mature the software industry with a laser focus on the customer. Serge van Dam, our panel host concludes nicely, “And I think we’re all passionate about that.”

To watch past editions of the tech leaders series, and get a rare glimpse at how companies like Alexa, Nike, Microsoft, AWS, Tableau Software, Raygun, The Standard, Xero, and Vend, monitor software quality, catch the recordings below.

• Wellington panel - Trade Me, Xero, Raygun, and Sharesies on maintaining an excellent standard of software quality.
• Seattle panel - Amazon Alexa, Tableau, AWS, and Raygun cover customer feedback mechanisms for better quality software.
• Portland panel - Scott Hanselman leads a panel of speakers from Microsoft, Chef Software, Nike, and The Standard talking about tools, people, and processes when building world-class software.
• Auckland panel - Xero, Vend, Lancom, and Tend on prioritizing customer experience in today’s digital climate.
• Read the Tech Leaders’ Report and learn how other tech leaders measure software quality and user experience.