Code Is Not the Outcome, Learning Is
There's a build-learn trade-off. We can't optimize for both. For early-stage startups, the choice should be a no-brainer.
At Lunar, we’ve helped build some 200 digital products. None was a slam dunk. There was no “we shipped it, and it went wild.” Hell, I’d have settled for “it performed as intended.”
Nope. Not one.
There’s a neat explanation coming from a recent podcast with Eric Ries of Lean Startup fame. Aside from some promotion of his new book, which I guess was the point of the interview, he got asked how Lean Startup fares in the AI era.
The centerpiece of his answer is something that I think still few companies claiming to be lean startups (or Lean Startups) get.
Learning is where the value is. Not the artifact.
In other words, the point of building an MVP is not the MVP. It’s everything we learn on the way.
Building a Product Is Never a Home Run
As in the Anna Karenina principle (”All happy families are alike”), all the products we helped build that eventually succeeded share the path. It was a long haul of changes, adjustments, discovery, and exploration to eventually find the right ingredients that clicked. And after that whole effort, “clicked” still doesn’t mean getting to millions in Annual Recurring Revenue (or whatever placeholder we now use instead) in months.
If we boiled it down to one meta pattern, it was optimizing for learning. My old mantra all over again:
Try stuff. See what works. Stick with what does. Drop what does not.
We don’t know up front what might work. We can’t know it. It’s unknown and—more importantly—unknowable. If the odds of a home run are but a fraction of a percentage point, what actually constitutes an MVP doesn’t matter at all. What matters is whether we improve our understanding of our surroundings and set ourselves in a better position for another swing.
Learning. Again.
In the Complex Domain Recipes Don’t Work
Cynefin offers a neat explanation why there are no home runs in the startup world. Here’s one of the filters that helps to figure out whether we’re in a complicated or complex domain.
If good solutions to a problem exist and we can scrutinize the problem enough to pick the right one, we are in the complicated domain. We can analyze our way through the issue.
If, on the other hand, we can’t possibly know the course of action unless we start probing the environment around us, we are in the complex domain. We can only play the sense-and-respond game here as we learn the novel environment. There are no recipes that can reliably work in this land.
So where do early-stage startups live? It’s the complex domain all the way. If they were complicated, we would be able to devise a path to success up front. Somehow, I’m not seeing these bulletproof plans turning into anything but a pipe dream. And, believe me, I’ve seen a lot of ‘em plans.
Again, we can’t know what to build from the outset. It’s not building pace that is the constraint. It’s the learning pace.
More Swings, More Hits, Right?
“But Pawel, we build 10x faster now. That’s 10x the attempts and 10x the learning!”
Um, it’s not 10x. Faros crunched the numbers, and the productivity gain is way below 2x. Packaged with deteriorating quality. If there is a more impressive change somewhere, it’s the size of a task. I looked at Lunar data, and the average PR might be more than 3x as big as it used to be. Packaged with deteriorating quality. Different data. Different metric. Same outcome. Coincidence?
Coding musings aside, we talk about two fundamentally different activities. Think of it as a difference between shooting as fast as you possibly can versus practicing to improve your technique. It’s not the attempts that count. It’s whether you score. And you don’t see teams trying to shoot the second they get the ball. One has to wonder why. They’d literally 10x their volume, wouldn’t they?
By the same token, getting another MVP or feature out the door at breakneck speed does virtually nothing to improve a startup’s odds.
Fine, say you can 2x the build speed. You can’t double the traffic by sheer willpower, though. You can’t summon twice as many customers willing to run interviews. Most importantly, you can’t extend anyone’s attention limit. It is the hard constraint.
But you sure can overwhelm said attention with the new feature overload. Break a leg.
The Learning Cycles
One of the useful things introduced by Lean Startup is the Build-Measure-Learn cycle. The gist of it is that you go in rounds. First, you build something. Then, you measure its impact against the expected results. Finally, you draw conclusions that inform the next iteration. Repeat. Over and over and over again.
While Eric Ries explicitly labels one part “learning,” the meta concept is anything but new. In the 70s, John Boyd introduced the Observe-Orient-Decide-Act loop, drawing on experiences from multiple dogfights. Before that came W. Edwards Deming’s Plan-Do-Study-Act method, which in turn built on Walter Shewhart’s Specification-Production-Inspection cycle. That was the 1930s already. Like, when dinosaurs roamed the Earth.
David Campey has another version of it. Idea-Struggle-It Works! It works just as well for kids as for grown-ups when they aspire to learn something. And he calls it the joy cycle. I used Discovery-Development-Validation in the product development context.
While each of these models tackles the issue from a slightly different perspective, the common parts are evident. The cyclic nature. Continuous validation. New insight at the very core. Past outcomes informing new decisions/actions.
These are all learning cycles. Not build cycles.
Do We Optimize for Learning?
Isn’t it curious how consistent the pattern is? Across the century, from Bell Labs (Shewhart), through Toyota (Deming, Ohno), the Air Force (Boyd), to the modern startup landscape (Ries), it’s the same message all along. It’s learning where the value is. Not the artifact.
If we want to optimize for the effectiveness of this process, the crucial skill is validation, or whatever you want to call it. Remove that, and you don’t have a cycle. You just have a Gantt chart of activities, which you can execute one after another. Or all at once.
If the latter was the case, with all the AI capabilities, we’d be golden. We’d just orchestrate a swarm of agents and call it a day. Alas, the problem is not complicated. It’s complex. We need that learning cycle.
And it’s just so easy to get AI to dismantle it.
“People’s ability to evaluate the quality of their own work becomes impaired by using [AI].”
That’s Eric Ries again. The more of the learning cycle we outsource to the tools, the less insight we extract. Hell, it’s detrimental to our learning muscle. It harms us short and long term.
We start with generating code. Soon, there’s too much of it to keep up with the reviews. Not long after, the product work follows suit. By the point where AI writes specs, the game is already lost. It’s a feature factory, not a product startup. We act as if we knew where we were heading when, in fact, we don’t.
We optimize for building rather than learning. We make the wrong trade-off altogether.
The Build-Learn Trade-off
I ain’t a luddite. I don’t argue for AI-free product development. I am genuinely impressed by how much we can achieve with coding agents. There’s no going back to 2020. It’s just that I am painfully aware that code is not a product, and product is not a startup.
If building was all there was, I’d be all like “Outsource the shit out of that to coding agents and call it a day.” But it’s not a Build-Build-Build cycle. It’s Build-Measure-Learn.
We need to rebalance the priorities. It’s like youth team sports. The best teams are not those that win petty games when kids are 10 or 12 years old. The best ones are those that develop these kids into successful professional athletes (and decent human beings, let me add). These goals are at odds with each other.
Winning when you’re 10 means you get the biggest and strongest and let them muscle their way over less physical teams. A couple of years later, these lads and lasses are nowhere near their counterparts who focused on understanding the game, learning the technique, and getting used to the teamplay.
The former is the equivalent of build focus. We muscle our way with increasingly powerful AI capabilities. The latter is learn focus. We don’t take shortcuts in the short term. In the long run, we beat the crap out of the brute-forcers, no sweat.
And those 10-year-olds with an overambitious coach aren’t even winning anything that matters. Save for the coach’s ego boost. Which isn’t something you’d put into your resume ten years down the line.
It’s the ultimate build-versus-learn trade-off. For an early-stage startup, it’s no contest. Learn. Duh!
How to Optimize for Learning?
The big question, then, is what can we do to preserve the learning component of the cycle? Better still, how can we enhance it?
Here’s Eric Ries one last time (I promise).
“The major antidote to this is to be getting real-world feedback.”
Yup, it’s as simple as getting feedback from actual customers and/or users. And no, now that you asked, synthetic personas for running user interviews are not a suitable substitute. You can’t cheat your way to real-world feedback without real-world activities. Bummer, I know.
Again, it’s the same intuitive frame. We put a different label on it, but it’s the same thing. Eric Ries packages it as real-world feedback. Teresa Torres might call it continuous discovery. Johanna Rothman would go with frequent customer interaction. I’d be all about validating relentlessly.
Essentially, figure out how to learn. Build only as much as you need to facilitate that. If you optimize building, the learning will come too little, too late.
Pick your side.
Some sources in this post come from fellow companies operating in the same niche.
The joy cycle comes from David Campey, who runs Afrolabs, which is very much Lunar’s African “brother from another mother.”
The podcast with Eric Ries was run by Kuba Filipowski from Netguru, which might now be in a very different place than Lunar, but we grew from the same root (and collaborated on a couple of projects years back).
If you ever consider Lunar as your product development partner, do consider them as well.