(Disclaimer: AI wrote this blog post based on my YouTube video here.)
Using AI to Test Product Ideas
We’re seeing more clients using AI to test product ideas. It’s fast, cheap, and offers instant feedback. But just like with most things AI-related, there’s a catch.
Below is a real example of a simple service idea we ran through several AI tools – not to sell you on the idea, but to show you what happens when different models respond to the exact same prompt. It’s a lesson in how easily we can be led down very different paths if we’re not asking the right questions – or worse, if we’re not thinking critically about the answers.
The Idea We Tested
A simple product concept: A service to vet HubSpot candidates for companies hiring internally.
Why? Because marketing managers often aren’t equipped to assess HubSpot-specific skills, and we’re occasionally asked to do it informally. So we wondered – should we turn it into a proper offering?
Then we ran it through 4 different AI models – using the same prompt. Here’s what happened.
What the AI Models Said
- Claude Sonnet
- Thought it was a “strong concept” with a clear market need.
- Gave pricing: $300–$500 per candidate, $3–8K monthly.
- Suggested 90-minute technical interviews, rush options, and retainers.
- Sensible structure, if slightly simplistic.
- Claude Opus
- Same prompt, totally different response.
- Estimated: $1,500–$3,500 per assessment.
- Introduced “tiered” offerings and deeper dives.
- Same idea, higher complexity, and cost.
- Perplexity
- Brought in external references and current market tools.
- More realistic: $600–$1,500 per assessment.
- Gave bulk discount suggestions and source links.
- A hybrid of Claude’s two personalities.
- Manis
- Took 20+ minutes to respond with a 30-page report.
- Included case studies, pricing models, service structure, add-ons, onboarding, and more.
- Sounds impressive… but who’s reading 30 pages from a tool that didn’t ask you any clarifying questions?
The Gotchas (and Why This Matters)
- Same question, wildly different answers
- If you’re relying on one model, you’re playing with fire. You’ll get something that ‘sounds right’ – but may not be right.
- False precision
- The tools throw out specific numbers ($8K/month, $1,500/candidate) that sound authoritative, but they’re not sourced, validated, or contextually relevant.
- Overconfidence in output
- AI doesn’t know your business. It can inspire, but it cannot think. Don’t let a 20-page PDF trick you into thinking it did.
- Bad questions = bad outcomes
- A vague or general prompt leads to fluffy answers. Without context or constraints, AI fills in the blanks however it likes.
- Most users don’t question the output
- They paste in a rough prompt, get an answer, and think, “Wow! That saved me an hour.” But did it? Or did it just lead them down a rabbit hole of noise?
A Quick Reminder
Don’t outsource your thinking!
Outsource research, inspiration, or exploration – but not the decisions.
The real value lies in how you interpret the AI’s answers. Take the best bits, ignore the noise, and refine the questions until you’re getting something useful. That’s the skill.
Next Steps
If you’re dabbling with AI to test ideas (as you should be!), here’s how to do it better:
- Use multiple models, not just ChatGPT. Compare answers, spot patterns, and notice inconsistencies.
- Refine your prompt. Include context, constraints, your goals, and what success looks like.
- Always review critically. Ask: Does this advice simplify or complicate things? Does it make sense for my stage, audience, and capability?
- Look beyond the hype. A flashy answer isn’t always a useful one. Focus on clarity, not volume.
- Bring in expert thinking before shipping or launching anything. Use AI to accelerate ideas, not replace sound judgment.
