Rolando Andrade
Back to Content

Time to first value: what a week of onboarding data taught me

How to measure time to first value in a developer product, why a better onboarding flow did not raise first API calls, and the developer onboarding best practices I took from a week of real data.

· 9 min read
Time to first value: what a week of onboarding data taught me

Time to first value is how long a new user takes to get the first real result from your product. In a developer product, that result is usually the first API request that works. To measure it, pick one event that proves value. Give every new account the same time window. Then count how many accounts reach that event inside it.

At SignatureAPI, our event is the first API call within 48 hours of signup. In October we spent a week rebuilding the first screen that new accounts see. When the first week of new signups had gone through it, the share of accounts making that first call had not gone up. It went from 47% to 45%.

I described what we built in What a week of data says about our new onboarding on the SignatureAPI blog. This post is about the other half of the work. It covers what I learned about measuring time to first value. It explains why a nicer first experience did not move the number. And it lists what we still cannot answer.

What first value means

The idea comes from SaaS products in general. A person signs up because they want something done. Until it is done, they are still deciding whether to stay. Time to first value in SaaS is the gap between “I signed up” and “this works for me”.

For a signing API, I think the moment is clear. A developer sends a request, and an envelope comes back. An envelope is the package that holds a document and the people who need to sign it. Reading the docs and clicking around the dashboard help, but they are not the integration. The integration starts when that first request goes out.

So we track two numbers together:

  1. The share of new accounts that make a first API call within 48 hours of signup.
  2. The share of those first calls that return an error.

The second number matters as much as the first. A request that fails is not value. It is the opposite. A developer who gets an error on their first try has mostly learned that your product is hard.

Measuring time to first value

Measuring time to first value sounds easy. Pick an event and count it. In practice, every choice inside that sentence changes the answer. These are the four choices we had to make.

  1. Pick the event. We count any request made with the account’s keys. That includes requests sent from the dashboard. This choice looks small, but it becomes the most important detail later in this post.
  2. Pick the window. We use 48 hours after signup. A short window gives fast feedback. The cost is that it cannot see anything that happens later, such as a second request a few days after signup.
  3. Pick the cohort. A cohort is a group of accounts created in the same period. We compared accounts from the two weeks before the release with accounts from the week after it.
  4. Decide who to leave out. We removed accounts opened by people who already had a SignatureAPI account. We also removed people who first used SignatureAPI as a signer. Neither group onboards the way a new customer does, so they would blur the result.

The window has a cost that is easy to forget. Our goal was to ship the changes in one week. But no number about a cohort can be read until its last account has had its full 48 hours. You ship, and then you wait. If you plan a one week sprint around this metric, plan the waiting too.

Our starting numbers were wrong

At the start of October, our numbers said that fewer than half of new accounts made a first call. The goal for the week was a call rate of at least 50% and first call errors under 20%.

That goal was set against a call rate of 39% and an error rate of 42%. Those numbers came from an earlier measurement. When we looked at the data again, we could not reproduce how they had been calculated. Measured with the definition above, the starting point was 47% calls and 15% errors.

So the goal was set against numbers that were already out of date. An error rate target of under 20% sounds ambitious next to 42%. Next to 15%, it means “do not make things worse”.

The lesson for me is simple. Write the definition of the metric down, in enough detail that someone else can run it again and get the same number. A baseline you cannot reproduce is not a baseline.

A shortcut we threw away

The first design we built was a welcome dialog with one button, “Create your first envelope”. One click sent a real request with the account’s test key. A six second animation played while the envelope was created. Then the user went to sign it on a separate page.

This version would have produced a first API call for every account that clicked the button. On our metric, it was close to perfect.

We replaced it before it reached customers, for product reasons. The request was the thing we most wanted people to see, and it was only visible for a few seconds inside an animation. Signing happened outside the dashboard. And the user never saw what the API returns after an envelope is signed.

Looking back, that design also shows a risk in the metric itself. When your own onboarding can send the first call, the metric can be satisfied by a button. The number goes up, and the developer has learned nothing about writing a request. Keep that in mind for the data below.

The flow we shipped

The version we shipped turns the same request into stages that the user moves through at their own pace. Every stage after the welcome has a Skip link. For a developer, I think these parts matter most:

  • The request is real. The screen shows the exact request body the dashboard sends. The same code builds it and sends it. Only the API key is hidden. Five tips explain the important parts, but nobody has to open them.
  • Signing stays in the dashboard. The signing session is embedded with embeddable_in. A customer uses the same option to show signing inside their own product. The first envelope is also a working example of that feature.
  • The result is visible. At the end, the signed document sits next to “What your app receives”. One card shows the webhook event (the message our servers send to your app when an envelope completes). The other shows the envelope as the API returns it, with the user’s real answers.
1 Welcome The first screen a new account sees. 2 The request The real request types in, with five tips. Skip 3 Creating The envelope is created from that request. Skip 4 Signing Signing is embedded with embeddable_in. Skip 5 What your app receives The webhook event and the envelope data. Skip
The real request comes right after the welcome, before anything is created. Every later stage has a Skip link.

Building the flow on our own public API also made us a customer of it. We found a bug that affected every customer using radio groups. If a signer left an optional radio group empty, the signing session could not be completed. We also found that dropdown answers were missing from the envelope data the dashboard reads. Both are fixed now. This is a useful side effect. Your onboarding goes through the same paths your customers use, so it finds their bugs first.

Reading a week honestly

This is what the first week showed.

Before After
Accounts that made a first API call 47% 45%
First calls that returned an error 15% 20%

Neither number moved toward the goal. It is tempting to explain this away, and it is also tempting to say the new flow made things worse. I think both are wrong, and here is how we read it.

First, we checked the result with a second definition. We used a stricter rule for spotting people who already had an account, and we left out throwaway accounts. With that rule, the call rate went from 49% to 43%, and errors went from 11% to 22%. The numbers changed, but the direction did not.

Second, we looked at the size of the groups. One week of signups is a small cohort. A difference of a few points is within what a couple of accounts can change. So I would not claim the flow lowered the call rate.

First API call within 48 hours 0% 20% 40% 60% Goal: at least 50% Our definition Before 47% After 45% Stricter rule Before 49% After 43% First calls that returned an error 0% 20% 40% 60% Goal: under 20% Our definition Before 15% After 20% Stricter rule Before 11% After 22%

One week of signups is a small cohort. A few points is what a couple of accounts can change.

Under both rules, calls went down and errors went up. Neither number moved toward its goal.

Third, we asked the narrow question the data can answer. Does it show the new flow raising the call rate? It does not, under either definition. That is the honest conclusion, and it is enough to decide what to do next.

Why nothing moved

The totals hid the story. We had to look at individual first calls.

In the week after the release, most first calls were the guided envelope’s own request. All of those succeeded. The few first calls that came from customers’ own code all failed.

The guided envelope's request Sent by the dashboard. Most first calls. All succeeded. API Requests from customers' own code A few first calls. All of them failed. { } API
The guided requests all worked. The few requests from customers' own code all failed, and every first call error came from them.

Remember the event definition. Any request made with the account’s keys counts, including the one the dashboard sends. So the guided request counts as a first call. But the accounts that used it seem to be mostly the same ones that would have sent a request anyway. They would have used the playground or the quickstart.

The flow changed how those people made their first call. It did not bring in new people. And it did nothing for the first request a developer writes in their own code. Every first call error in the new cohort came from those requests.

This changed how I think about time to first value for developers. The value is not “a request happened”. The value is “my code sent a request, and it worked”. A guided demo can teach the shape of a request. It cannot replace that moment.

Developer onboarding best practices

These are the practices I would take to any developer product after this week. They come from what we built and what the data showed.

  • Show the real request. Build the example from the same code that sends it, so it can never drift from the real API.
  • Let people go at their own pace. Typed text that pauses, optional tips, and a Skip link on every stage.
  • Show what comes back. The webhook event and the API response teach more than a success message.
  • Count your own requests separately. If your onboarding sends requests for the user, track the developer’s own first request as a separate number.
  • Make the first error useful. An error the developer cannot act on ends the integration before it starts.
  • Write the metric down. Event, window, cohort, and exclusions, in enough detail to reproduce.

What we still don’t know

A lot, and I want to be clear about it.

We have not measured how many new accounts finish the guided envelope, how many skip it, or at which stage people leave. That is the first thing we will look at.

We also do not know what happens a few days later. Are people who sign their first envelope this way more likely to send a second request from their own code? That second request is what the flow was built to lead to. A 48 hour window cannot see it, so the metric will need a longer window to answer.

The clearest problem is the errors on first requests from customers’ own code. A recurring one is a validation error whose response says nothing beyond its name. The developer gets no clue about what to fix. We plan to fix that first.

The full story of the flow, with videos of each step, is in the original post on the SignatureAPI blog. Our next number to move is the first request a developer writes on their own.