Validation Is the New Bottleneck (Also, the Old One)
Early-stage startups have limited options for rapid validation. That makes validation timelines the bottleneck of product development.
We just went over the metrics dashboard with our client. They predominantly focus on the acquisition funnel, which matters for the product now, but we read the temperature for conversion and retention, too. Finally, we have a compound metric, a.k.a. the North Star metric, that covers a range of customer behaviors.
Then, we have a whole list of product experiments queued up.
The most obvious path forward is to prioritize those experiments, try the simplest prototypes, and see how they affect the metrics. If readings go up, we double down on the idea. If they’re meh or go south, we roll back and pick another thing from the list.
Except it’s not that simple.
The Metric Liquidity
In finance, we talk about market liquidity. If the market is liquid, it means we can easily buy and sell the asset. What follows, the prices of these transactions are frequently and continuously updated to reflect how the “market” values the asset.
In a similar manner, we can consider metric liquidity. When Amazon deploys a feature to production, they know almost instantly whether it helps or harms their KPIs. Their metrics are liquid.
With our client, it’s not the case. Mere weeks after the initial release, the traction is, well, I’d love to use the word “modest,” but it’s not even that. Week-to-week readings may bounce up or down by as much as 100%. And it’s not because something significant has changed about the product. It’s because of how few data points we’re dealing with.
To stick with the financial market analogy, it’s as if transactions were few and far between. In such a case, the market is not liquid, and we can’t infer much from the most recent price. It might have been a one-off thing. Or it happened too long ago to tell much about the current status of the asset.
By the same token, metrics that are not liquid enough will tell us little about how well a product is doing. Add a game-changing feature, and if there isn’t anyone to notice, the metrics won’t budge.
Timelines of Validation
Fundamentally, the more liquid a metric is, the faster the lead time for experiment validation. Amazon can automatically roll back underperforming features and keep those that positively impact their KPIs. Anthropic can embrace the spaghetti strategy to see what sticks with their users. Netflix famously A/B tests the shit out of everything before settling on what’s the new norm.
For contrast, imagine a fledgling startup with a handful of early users. Unless the idea under scrutiny truly blows up, which happens but is extremely rare, it’s gonna be a long wait till the metrics show traction.
Those 3 new users who used the product a week after we released a new feature may suggest a 38% engagement increase thanks to the new idea. Or it can be a natural fluctuation that tells literally nothing. How do we know? We wait. If the change is real, the traction growth over time will compound and outweigh the natural engagement flux. We will know. Eventually.
Metric liquidity defines validation timelines and available strategies. As much as we’d always like to have short experiment feedback lead times, without enough data liquidity, it’s not an option.
Why Long Validation Lead Times Suck
There’s an important reason to push for short feedback loops whenever we experiment. OK, there are many. Like the fact that humans don’t like uncertainty. We don’t like it all. We actually prefer the certainty of receiving an electric shock to a 50/50 chance of getting away without the painful experience. Curious creatures we are, indeed.
Electric shocks aside, long validation lead times suck in product development because they serve as a pace-maker. We either wait for the outcomes of one experiment before trying another, or we risk the outcomes of different attempts interfering with each other. In the latter case, odds are we’ll have the right change and never know it, buried under lots of other, not necessarily helpful, stuff.
Even more crucially, the outcomes of earlier experiments inform later decisions. It’s the old Build-Measure-Learn cycle from Lean Startup. Skip the “learn” part, and you drive in the fog. You can make one turn or another, but it’s mostly guesswork. Good luck trying to navigate like that, I guess.
When you run a startup, you’re playing against the odds already. No need to make your situation even more miserable by making your experiments random. So yes, long validation lead times suck big time (to use a technical term).
“We Wait” Should Be a Default Strategy at the Earliest Stage
Unless we subscribe to the notion that building a successful startup can be predesigned (which I called the destination hypothesis in the past), the metaphor we operate under is exploration.
We try things, stick with what works, drop what does not. In product development, we learn which is which thanks to validation.
Put differently, validation timelines are really what define how fast we can go at a startup. Those timelines, in turn, are constrained by the traction.
At the earliest stage, we have little-to-no traction, and that’s precisely when we need the most rapid validation cycle. Except that’s not an option. Let me rephrase. It sucks to be an early-stage startup.
We can reverse the picture, though. If the only available tactic for experiment validation is “we wait,” then faster development doesn’t do much for us anyway. We arrive at the same conclusion: development speed is not a bottleneck.
It is not today, when we can generate a ton of code with AI. It wasn’t so in the past either. Even if we acted as if it was.
The Guessing Game
Imagine a game. You’re supposed to guess a 4-digit number. You pay $4 for each try. If you win, you get $5,000.
Alternatively, you can play the same game, but you “buy” each digit separately for $1 each. So you start by trying to guess the thousands digit for a buck. You know whether you guessed correctly before you decide whether you want to “buy” the guess on the hundreds digit. And so on.
Which version would you play?
Obviously, the latter. Duh! It splits one big decision into four separate small calls, where each informs the following ones. If we miss the first guess, which is likely, we save ourselves $3 and stop trying.
We play the same game with our product experiments. That is, save for the fact that most companies claim to choose the second variant, and then don’t wait to learn how the small bet went and commit to the following ones right away.
The rationale is along the lines of “we have developers, and we need to keep them busy.” It’s like betting again before we even know how the last one landed. That’s a compulsive behavior of people with a gambling problem.
Also, a default in the startup world.
Validation Has Been a Bottleneck All the Way
When we consider the relationship between metric liquidity and validation lead times, it is clear that the effort required to build an experiment doesn’t materially affect the picture.
We should, in fact, subordinate development effort to the pace of experiment validation, and not vice versa.
Interestingly, our client, who I mentioned earlier, has done precisely that. Once the MVP went live, instead of speeding up, we slowed down.
We could now be adding stuff like there’s no tomorrow. After all, there’s no shortage of ideas. But it’s not a casino, and we don’t try to lose our life savings. Our context is way less movie-worthy product development. And we understand that it’s not building that’s a bottleneck. It’s validation.
It has always been so.