AI Won't Generate a Good Product Idea
While creativity remains a popular use of AI, we shouldn't expect it to come up with a good product idea on its own.
Creativity remains one area where we utilize AI quite extensively. When HBR published their 2025 update on AI usage patterns, the first thing that made the news was how we had started treating Dr ChatGPT as our personal therapist even more than we had.
‘Generating ideas’ use case fell from the top position. However, ‘Creativity’ rose through the ranks to be included in the top 10.
If we stick with the professional applications of AI, we have only two contenders:
Coding: generating code and improving code categories (ranked 5 and 8).
Creativity: generating ideas and creativity categories (ranked 6 and 9).
Now, obviously, in the context of early-stage product development and (especially) ideation, creativity plays a pivotal role. There’s no shortage of guides telling you how to generate product ideas with LLMs.
Given how much we rely on AI for our creativity, how good is it in ideation? Can we get it to generate a genuinely good product idea?
AI as a Probability Machine
At a risk of boring you with covering the absolute basics, we need to start with a general idea of how an LLM works. The whole ChatGPT brilliance that conquered the world is basically a probabilistic machine that guesses the most likely answer to a prompt.
Not even the whole answer. It’s just the next fitting word. And then, given the prompt and the first word, it guesses the second. And then, with the inclusion of the second word, it guesses the third. And so on.
Given the prompt: “Who was the first king of the USA?”, an LLM guesses the first word. It seems that Gemini chose “The” as the first one.
Then it asks itself the following. Given the prompt and the fact that the first word is “The,” what is the most likely second word? It’s “United.”
And, given the prompt, what’s the most likely word after “The United?” Isn’t it “States?” Duh!
And so it goes.
It’s like an improv game, One Word Story, except an LLM simulates all the players.
An LLM can do that because it was trained on an immense amount of data. It read the entire internet, all the books that have been digitized, and more.
AI Has No World Model
As impressive as AI One Word Story can be, it’s also the source of LLMs’ biggest weakness. Since it’s essentially an extremely well-trained guessing machine, it doesn’t have a world model.
In the question about the US king, it doesn’t know what a king is or what the USA is. Or why it makes sense to mention a president in that context. And it’s fine. That is, as long as it operates in the context where there is a lot of training data. And sure enough, we have plenty of records covering the history and political system of the US.
What would happen, however, if we were trying to explore a territory where there’s little to no training data? Like something really esoteric?
An LLM would still be guessing the most likely words. Which would be smart-sounding gibberish delivered with confidence. Because in this case, “most likely” would be very unlikely, but hey, still more likely than everything else.
We call it a hallucination.
Because an LLM has no actual world model, it can’t know when it hallucinates. A great example is playing chess. An LLM can recite the game rules. When asked explicitly, it will tell you that a specific move is illegal. Then, it will promptly make that illegal move.
The reason? Not enough training data about that very game and that very position. And no understanding of what a queen, bishop, or chess in general is.
But why does that matter for creativity?
AI “Creativity”
If we stick to this (admittedly oversimplified, but roughly accurate) picture of AI as a probability machine, we can assess how creative it would really be.
Imagine a context of a prompt—any prompt, really—and a range of possible answers. The more likely an answer is, the closer it is to the top of the distribution.
Unless we do anything extra with context management or prompting, our probability machine provides the most likely answer.
In other words, we’re landing with something that would be the most popular option if we asked enough knowledgeable people. That’s not creativity.
If, as Cambridge Dictionary suggests, creativity produces something original or unusual, it has to happen on the fringes. Anywhere but where AI directed us.
The Beginner in Awe, The Expert Not So
”But Pawel, if that is true, why then are we so often amazed by the ideas we get from AI?”
The shortest answer? Because, as likely as AI’s answer is, it’s still very likely broader than our narrow and shallow knowledge of the topic. We don’t ask about stuff we know. We know it, for heaven’s sake. Why would we ask about it?
If we start with a beginner’s mind, the most common answer will put us in awe. If we know almost nothing, almost anything would teach us something.
Here’s a common observation, though. When we probe AI in areas where we have deep expertise, they tend to fall apart. We spot hallucinations way more often (we are experts, after all, we know stuff). The answers we get are underwhelming.
It’s because “the most probable answer” loses a lot of its appeal when confronted with deep and broad knowledge.
In fact, it’s not only the deeper and broader knowledge of an expert that changes the outcome. It’s also the question context. After all, a historian is unlikely to ask about the USA’s king.
That brings us to a handy lever we have when working with AI. We can manage AI’s context.
Shifting the Context
Since we can flexibly manage the context and prompt, we should be easily able to shift the answer toward less likely ones. Think of it as adding the ”give me only unlikely answers,” or ”suggest 20 different answers to this question,” or ”imagine you fulfill [a specific role]” system instruction to the prompt.
While technically it changes the query and—ironically—we still get the most likely answer, if we tried to plot it against the original context, it would shift.
Whoa! We’ve just gotten more creative, haven’t we?
Yup. With two caveats.
We are still limited by our (human) creativity. We can shift the context only as far as we can imagine the right questions, prompts, and context. A beginner won’t get to the expertise areas.
We pushed the context window into an area with less training data, and thus, less reliable answers. Even if an answer may sound more creative, its potential trustworthiness is way down.
On the one hand, we want to push the context window as far from our knowledge as possible. Hell, that’s how creativity works, doesn’t it?
On the other hand, the further we push it, the more likely AI is to hallucinate (because of less training data), and the less likely we are to spot it (because we’re far from our knowledge base).
Creativity Happens on the Fringes
Oh, and we didn’t even get there. If we want to get something original and unusual, we need to shift the context to the extreme.
First of all, doing that rules out beginners. By this point, we need to be able to ask expert questions, and that, in turn, requires a degree of knowledge. There’s no way to send AI to explore the fringes from a safe distance of an ignorant mind.
”But Pawel, we all read stories about how people use AI to close the gap, to learn what they need to learn on the way. It’s then a matter of a bit of effort and willingness, right?”
Not really. As much as doing anything serious with vibe coding requires more, not less, technical skills, relying on AI with creativity requires more, not less, innate understanding of the domain.
The reason is trivial. We need to be able to validate the AI output. By now, by definition, we are in an area where training data is scarce to non-existent. We have to assume that an LLM will be hallucinating like hell.
It doesn’t have a world model to actually think creatively. Word by word, it will generate the most likely answer. With great confidence, of course.
In the end, we might want to redirect our efforts to our own creativity, instead of verifying why AI output is gibberish. It’s going to be a much more effective way of investing our time.
AI Creativity in Product Ideation
We can’t outsource our creativity to AI models. We can create an illusion of AI creativity by sending LLMs just outside our area of understanding to fetch us something that looks new, but only because of the limits of our imagination and knowledge. It’s not innately creative.
When we send it too far, it fetches literal shit.
It helps to remember that asking an AI model is akin to having a way to ask the entire internet: “Given the context, what’s statistically the most likely answer to this question?”
Not “the best,” not “the most reliable,” not even “actually true.” It’s just “statistically most likely.”
Oh, and the internet as a whole isn’t the most truthful source itself, and it is getting worse as I write (and you read) this.
Limits of AI Product Ideation
Having all that in mind, it’s not difficult to imagine the limits of how AI can help us with product ideation:
It can help us orient if we explore a relatively popular domain. What are the existing solutions and products, and how are they marketed? What’s the size of the addressable market, what are its demographics, etc.? We can get a quick pointer to at least relatively reliable data.
It can help us challenge an idea if we don’t need too much depth. Look at [this idea] and figure out ways why it will fail. What are the less common challenges that products in this niche face? It’s like double-checking whether our intuition didn’t leave a blind spot or whether our love for the idea didn’t blind us. It won’t take us too far from what we already know.
It can help us seed our own creativity. Some creative techniques rely on tearing apart other ideas or generating actively bad solutions just to jog our creative muscle. There’s no shortage of bad ideas an AI agent may offer.
Think of AI as a tool that may be contextually useful when humans do the thinking. As impressive as LLMs sometimes are, they won’t do it themselves. They’re incapable of that.
At its current state, AI will not come up with a product idea by itself. It isn’t creative. It just creates (pun intended) an illusion of creativity.