Developers are the strength of any API platform; they not only contribute to its success, but they enable your product with pathways to novel, actionable use cases you can turn into new product features. When developers embrace an API platform to create a vast range of experiences, they unleash their creativity and ingenuity and put it at the heart of what they built. The quicker they get to validate their effort, the quicker they can go to market, iterate, and enable both themself and your platform to succeed.
Growth poses new challenges that get in the way of that creativity. Increased usage can bring considerations about access control and rate limiting; monetization may include a pricing tier that may upset organic developers; legal and policy consideration may rightfully protect all users and significantly alter roadmaps and slow down releases and product adoption. As the number of endpoints also increases, developers may find it increasingly difficult to get started, or even to understand what set of functionalities should be used for the use case they want to implement.
When the developer experience stays the same or moves slowly, platforms become slower to act upon developer feedback. And developers who are not heard are likely to fall into neutral or negative NPS territory; lower NPS means more churn; more churn means less organic traction.
Each of those issues is a barrier between developers and their first meaningful API request to your platform. The time developers take to overcome those barriers is usually called Time To Value, or Time To Hello World.
Time To Hello World
Developer Relations and Product organizations are equally fascinated with Time To Hello World. For the former, it’s a measure of engagement and a proxy for quality of code tools (like SDKs or template projects) and documentation; for the latter, it’s a helpful metric to understand how to drive a coherent platform design that it’s easy to consume and understand.
Interestingly enough, there is little agreement on the standard definition of Time To Hello World. Both literature and online resources do not seem to converge on what to measure, and how. Should there be a set limit of time? What is the right amount? What’s to do when developers get started after that time, and should we get a hold of developers who gave up trying?
Understand your funnel
Most organizations look at Time To Hello World as a single number; for them, it’s almost like record developers need to break. This mindset is common, and it’s one of the reasons this metric still looks a little nebulous. If Time To Hello World is truly about measuring time to overcome obstacles, and to reduce that time, then we need to start looking at the journey developers go through between landing to your platform and provisioning access.
Just like any marketer maps a customer journey, every API platform should map its developer journey. Nearly every platform has a signup process where developers sign up and get provisioned with API credentials; all users will likely need to go through documentation and code examples, debug, and make a request. There are few exceptions (if any) to this pattern, and it’s hard, if not impossible, to complete a request in seconds. This explains why we shouldn’t be focusing on an industry-standard amount of time in the first place.
Determining when the clock starts ticking
When do we start measuring Time To Hello World? The starting point is not the same for all platforms. Enterprise platforms, where registration flows may be largely dependent on trial periods, may need to apply a different approach than direct-to-consumer platforms. Some of these platforms implement signup and credential provisioning as two separate stages.
Because of the variety of flows and processes, it’s important to determine what’s the most meaningful moment to start measuring Time To Hello World, knowing that there are at least three important moments:
- Signup. It’s the moment developers have general access to the developer platform as users. Developers may sign up to take a look around, and their intentions of actually using the API may not be consolidated.
- Access. This is the moment when developers are actually enabled with all they need to make their first API call. This is usually a combination of credentials, code and documentation.
- Hello World. The developer’s first request!
It’s important to understand how these three moments are placed in our timeline to first request. Because some platforms review access requests before issuing credentials, the time between Signup and Access may be shorter than the gap between Access and Hello World.
Capturing these considerations into your Hello World measurement is crucial to determine what steps your platform will need to optimize. It’s likely that developers may be still into their intent stage when they signup; this may be the case if certain developer resources are only available after signing up. Generally speaking, signing up is an occasion to further engage with developers, for example by exposing them to a newsletter, a developer community, or other resources.
When developers can finally get access to their API credentials and make their first request, the time to value may be significantly longer than expected. You may already have inferred that the playful moniker of Time To Hello World implies the developer can get started quickly and easily; what’s less obvious is that developers should also see the value of the underlying developer experience. This is where oftentimes we see the typical disconnect between developer marketing and developer relations; during the access phase, developers are expected to evaluate tools and technologies, client library implementations, and to make a number of unsuccessful API calls before finally succeeding. It’s arguable that some of these decisions will need more time than others, and this time will be different depending on the developer audience. For example, choosing the right SDK will be more of a concern for a professional developer than a student – you need to make the right decision now or waste time later in a potential refactoring later.
This is one of the key reasons why smoothing the transition between intent and action smoother is a delicate but impactful optimization. If your platform does not offer first-party SDKs, compile a list of supported clients you endorse, and keep it updated. This way, developers will find guidance from a source they already trust (yes, your platform). This also means less user dispersion, because the developer will not need to leave your platform and spend time searching for answers that may potentially lead them to a competitor.
Each audience has its pace
Platforms – especially those with a global reach – cater to a large multitude of audiences and developer segments. Yet, most measurement frameworks consider time to value as an average metric. Even if averages are easy to compute, they do not discriminate between outliers and general behavior. If a small number of developers ends up taking a very long time to Hello World, your overall average is going to be much larger, and you may end up dimensioning your efforts without seeing the expected improvements.
This is why things like request latency are measured using medians or percentiles, not averages. Users all over the world may access content with the most disparate combinations of bandwidth, devices, and network conditions; sometimes even the same user may not be able to have the same quality of service (it’s the case for many commuters who may have access the same service from home and from a subway train with two different network experience). This way, we can understand exactly what percentage of the overall user population connects in a given time range.
It sounds like percentiles can be useful to measure Time To Hello World. After all, if users all over the world may not have the same experience, why should your developers be different?
Here’s a practical example of why percentiles matter. Suppose you have a population of developers getting started with your API platform. Suppose you can measure each individual’s Time To Hello World (which you can).
You can immediately notice two things:
- The general population takes less than 10 minutes to get started
- There are two outliers
If we were to compute the average (the sum of all the values divided by the number of size of the population), our Time To Hello World would be 24.45 (24 minutes and 27 seconds). This number clearly does not reflect the general trend. A median, on the other hand, takes the middle point of the distribution as a reference point. It allows us to say that 50% of developers took 3 minutes of less to get started. Compared to an average, it pictures the reality more closely by using an actual value in the distribution (nobody took 24 minutes to get started, if you noticed). It also allows us to partition our user buckets incrementally, and find more precise areas of optimization.
Quarterly percentile increments (25%, 50%, 75%, 99%) are often used to bucket users. This way, we can correlate time to value with the share of the developer population. More importantly, it allows us to better understand outliers and find areas of improvement to make everyone’s experience more uniform.
