How to monitor deployments for performance problems with Raygun

Many engineering teams (including ours) use Raygun’s Deployments feature to monitor deployments and version releases. But how do they use it in their day-to-day operations to watch for performance problems and identify the bad releases? We asked our customers so we could bring you some practical examples.

TL;DR: Teams mostly use the Deployments feature as a visibility tool. It is most helpful when investigating which errors have been introduced with the new release, which are still occurring, and which have been fixed.

Get visibility in the deployment pipeline

The consensus around teams we asked was that visibility into the deployment pipeline is the largest use case for monitoring deployments. What most teams want to avoid is errors slipping through your processes undetected. For example, it’s common that your pipeline reports nothing wrong, but there could be a problem with a server which prevents files from loading in your app. Raygun would detect these issues, acting as a safety net.

Although Raygun alerts you to errors as they occur, we recommend configuring Raygun with deployments so that you can correlate error spikes with releases.

You can then click on the Deployments chart to show version releases:

Check which version created the error, at which time, and who made the deployment:

“The Deployments tracking feature brings a whole new level of insight data for better diagnostic and resolution of errors, resulting in better quality for your app,” says Daniel Conde of Intechnica.

Use the Deployments feature for release management

Once you’re all set up, it’s time to release your code and play the waiting game. If something goes wrong with the deployment, Raygun will alert you to the error via email, or via your dedicated Slack or HipChat channel. Teams then use the error details page to see what’s wrong.

To help identify which commit caused the problem, head to the ‘Commits’ page under the ‘Deployments’ window. If you’re using GitHub, GitLab or Bitbucket and have it integrated then the commits tab will show you what commits were released in this deployment.

The version link sends you through to the commit in your source repository so you can see the code that changed in that commit.

The next step is to go and fix the problem. While the build is running, head back to the error and mark it as ‘Resolved’ in your version. Raygun will then stop emailing to say the error is still occurring. (Unless the version is greater than or equal to the last.)

Once the build is done and deployed, go back to the Deployments page and make sure that the new release is much healthier.

To roll back, or not to roll back?

Some team leaders we spoke to use the Deployments feature to make a calculated decision on whether to roll back a deployment.

You’ll need to decide what to do. If it’s a quick fix and you’re in a position to deploy quickly, then you may leave it, or deploy a fix. But if it’s affecting customer experience, then speak to your development team as they will know what to do.

As an extra warning sign of a deployment gone wrong, we recommend taking a look at your Real User Monitoring dashboard. Look for warning signs like:

• Increased P99 load time (the time it takes for the page to load for the 99th percentile of users)
• Shorter customer sessions

At Raygun, we also use the Real User Monitoring Compare feature to check for performance deltas. The following screenshot shows how you can see each version release’s performance side-by-side (in this case 1.0.0.41433 vs. 1.0.0.41402):

Compare deployments for performance and error improvements over time

Usually, it falls on the team lead to create a performance analysis of the previous deployment. Monitoring deployments for performance will make this job much easier, as you can see a timeline of events going back six months. You can use the data to cross-reference a deployment’s performance over time and create a performance analysis based on the results.

You’ll also be able to correlate errors to version releases meaning much less time tracing bugs back to specific deployments.

Building reliable products for customers is where deployment monitoring helps the most.

As Indy Griffiths, Co-Founder and CEO of Parent Interviews says, “Reliability is a key factor in ensuring a great product.”

You can use the Deployments API to configure Raygun to monitor deployments.

Related articles

Get more visibility into deployments with Raygun and Codeship

What is Software Intelligence, and how can it help your team?