How to Measure Developer Success
In most companies the ultimate measure of success is if you’re growing and/or making money. For developer-first platforms that can start and end with enabling developers to get their job done. But how do you know if what you’re doing in DevEx and DevRel is working? We’ll examine what metrics to focus on to measure developer success and which you should steer clear of.
North Star Metrics
There are many different opinions on what’s the ideal north star metric that measures developer platform success. Some DevRel metrics are valid, while others can be what are called vanity metrics. The goal of DevRel is to ensure third-party developers are able to leverage your platform to create something of value. Value can be subjective, but some examples include shipping a new integration or plugin that increases the usability of your products or integrating your APIs and SDKs into their web or mobile apps to deliver a better experience for their customers.
False North Star Metrics
Because this is hard to quantify, some developer relations programs naively fall back to marketing metrics like page views and sign ups. The problem with these metrics is a new user who signs up for your platform didn’t actually create any value. You can easily hack these metrics by running a large Google AdWords or Facebook campaign to drive an influx of random sign ups. Yet those users would never integrate and leverage the platform.
To really measure the success of a platform, you should look at two north star metric:
- Weekly Active Tokens (or Weekly Active Users)
- Time to First Hello World
1. Weekly Active Tokens (WAT)
In order to measure the success of a platform, we should instead look at the usage. This can be from the various APIs and SDKs that you have released for third party developers to interact with your platform. The mere act of generating an API key does not measure usage since even non-developers can view API keys without any real use for them. Thus we turn to Weekly Active API Tokens. Since most APIs limit access to authenticated users, we are able to track how many distinct tokens are accessing our API platform in a given week.
Since a single customer can create multiple API keys with various scopes and expiration dates, we can further narrow this down to Weekly Active Integrated Companies. Everyone on your developer relations team should be looking at this metric when deciding on where to invest more time. In order to do so, you should have the instrumentation in place to tie metrics like tokens to developer demographic info such as any marketing channels that brought the developer to your site, company affiliation and size, and even social info such as GitHub stars. With this info, you can break down WAT to find what is contributing the most to your north star metric. Read more on other metrics a platform team should know
2. Time to First Hello World (TTFHW)
Followed by Weekly Active Tokens, Time to First Hello World, is a good proxy for the overall success of your platform. Not only does it include adoption and usage like WAT, but it also includes other functional areas outside DevRel such as marketing and support.
The longer a developer takes to get started with your platform, the less likely they will be successful. Documentation that is hard to understand, unclear next steps in onboarding, or API and SDKs that require a six hour lecture just to understand the basics are all items that can contribute to higher than wanted TTFHW. Read more on TTFHW and the Developer Funnel.
An effective DevRel program should be working to increase Weekly Active Tokens (WAT), while reducing Time to First Hello World (TTFHW).
Where Does DevRel Sit?
Since the goal of developer relations is to ensure developers are able to create something of value with your platform, extra care must be taken to ensure the DevRel goals are aligned with the greater department. Being a relatively new field, each company has its own idea where DevRel should reside and where the budget should come from.
At some companies, developer relations is an extension of marketing. Due to this, these DevRel teams have the luxury of a large advertising budget and can promote the platform everywhere from Facebook Ads to a large presence at conferences. Key metrics for the marketing team include page views, sign ups, and Marketing Qualified Leads (MQLs). However, these metrics are misaligned with the developer relation’s team in helping developers engage with and find something of value with the platform. Thus, these teams are effectively a technical or developer marketing team without much of the relations part.
At other companies, developer relations is an extension of the engineering team. These teams are responsible for up to date API documentation and potentially the SDKs themselves. They are also able to debug and fix issues developers may be running into or create additional guides and examples that can aid their developer community. These teams unfortunately spend most of their time engaging with other teammates at their company rather than the greater community due to the constant stream of SDK changes, documentation updates, debugging, etc.
The product’s mission is to build a product customers (or developers) actually adopt and use. They are constantly testing various hypotheses and building out a feature road map based on product analytics data and qualitative feedback from customers. Developer relations teams are constantly collecting feedback from developers to influence the road map, while they are the first to know if there is any resentment from their community on product changes. In this capacity, developer relations is an extension of product and more “in the community” rather than focused on the building side and collaborating with internal stakeholders. While this is a biased belief from working with multiple platform companies and developer relations teams, we’ve seen DevRel can perform best when aligned with product rather than other areas like marketing and engineering. After all, product is also held accountable by engagement metrics similar to Weekly Active Tokens and Time to First Hello World (which product messaging and onboarding experience can have a large impact on).
Any new program needs to have the right metrics that guides the mission and goals of the program. Developer relations teams are not immune to accountability. Yet, the limited operational history of developer relations teams have created an ad hoc system of borrowed metrics from other departments based on where the DevRel budget comes from. Creating product metrics like Weekly Active Tokens ensure DevRel is aligned with helping developers engaging with and finding value in a new API or platform.