Lessons Learned from 500+ AI Pocs
AI: The Wrong Expectations
No matter the size of the company, companies always want to make optimizations; the more in trouble the company is, the more optimizations the company wants to make. Does anyone look for cash flow? There is nothing wrong with optimizing things, but sometimes companies can throw the baby out with the bath water. We are seeing this to an extreme degree, from companies believing that software is solved and doesn't need engineers anymore to the fallacies of productivity.
Technologists are expensive, engineers are expensive, architects are expensive; however, building the wrong product and having a bad user experience is even more expensive. AI, generative AI, and agents, in fact, speed things up; however, it's not so simple.
Software development has become more democratized due to agents and tools like Claude Code and Codex. However, just because anyone with a credit card can vibe code something does not turn them into professional engineers. A good chunk of people who are not engineers would not want to be on-call, debug production issues, and be accountable for making software run smoothly like a clock every day, 24x7. A lot of people are vibe coding their own solutions, and that's cool and awesome; however, I also see many stories of people and companies starting something and giving up in less than 3 months. Guess what, maintaining software is hard, even with AI.
Wrong Expectations: Productivity Gains
It's obvious to anyone that AI has productivity gains, however that is not so simple as well. The closer you are to infrastructure and constrained problems like migrations, the more benefit you get, and the faster you can benefit. I had experiences in the Christmas of 24, early 2025, migrating a good-sized codebase from Kotlin 1.x to Scala 3. x in less than 3 weeks. Back then, I was using the arcane GPT-3.5 Turbo in VSCode. However, during that migration, many bugs were introduced, and the hard troubleshooting had to be done by humans. YES, AI got better and might still get a bit better, but not in all senses. Back to 2026, we see cases of impossible migrations happening, like Bun: from Zig to Rust.
The closer you are to the user, the more ambiguity you have, the more departments are involved, the more friction and less speed AI gives you. Companies never measure things properly. I'm a big believer in Lean. I've always believed you need to measure end-to-end, from idea to production; that's not how companies measure, so they cannot really tell for sure whether things go 2x, 2x, or 10x faster. It's only vibes and perceptions of productivity. The more you think you can one-shot things, the less you know what's going on. Recently, the Primeagen introduced an interesting term: "AI Demo p*rn". However, 9 months ago (which in AI terms is 3 years :-)), Karpathy explained why DEMOS was not really that interesting. Because you can get from 0 to demo in one shot, but demo to production is not that fast.
The more impressed you get from one-shots, the less you understand what's going on. We need to understand that AI is a token approximation machine and will produce averages. It can do amazing things; the more you try to steer it, the more you see it does not go that well. I never liked the term vibe coding, because not paying attention to the details or the code or feature could never lead to something great or even better, calling extraordinary (which is above ~avg). You can get amazing productivity gains with AI if you want to be 100% the same as everybody else. Is that what we want/need?
The discipline
By going too fast, we are cutting some important elements. Mainly what some people call the craft, or the discipline(Dex); I just call it the proper way of doing software. In a few months to weeks, depending on where you live, we are moving so fast that we can't digest things.
Start with Architecture
People hear "500 POCs" and imagine me firing prompts all day. That's not it. A POC without a spine is just slop with a repo. Here is the actual loop:
Research → Design Doc → POCs → Implementation → Iterate
The design doc is not bureaucracy, and it's not for management. It's for me. It's where I figure out what problem I'm actually solving, what the tradeoffs are, and what "good" looks like — which, as I said, is exactly the part AI cannot do for you. Writing it down is how I find out if I understand the thing or only think I do. Can I explain it? Can I defend it? That's the discipline, and it costs an hour.
Then the POC becomes an experiment against a hypothesis instead of a wish. And this is the part people miss: the model produces better code dramatically when there's a design in front of it. Same model, same prompt quality, completely different output. When I skip the doc, I get 3000 lines of confident garbage that compiles and means nothing. When I write it first, I get something I can actually reason about.
The order matters, too. POCs come before implementation, not instead of it. The POC is where you're allowed to be wrong. Implementation is where you bring back what you learned and build it properly — with design, testing, observability, automation, all of it. Vibe coding collapses those two phases into one, and that's precisely why it produces distributed monoliths, just faster.
Research, design, build, iterate. Nothing new here. It's the same discipline we always had; AI just made the POC step nearly free. That's a reason to do more of them, not a reason to skip the doc.
Lessons Learned
I group it into 4 categories of learning.
Passion and Self-Motivation
I do things because I want to and like it, not because someone told me to. I need to always be learning, always discovering and expanding; that's who I am. AI is created to feed my passion because it allows me to do things that before I would never ever do, like:
- A video game for my daughter: Seagotchi. Plenty more here.
- A Hacker News Chrome extension so I do fewer clicks and save time.
- An app to track my games, the ones I finished and am playing: Game Stand.
Skill and Judgment
I was doing a POC with Playwright and discovered that Playwright records video of the test run. I did not read that in the docs first. I hit it, saw it, and immediately my brain reorganized itself: if the browser session is recorded, then an agent that can watch the recording can debug the UI. That's a completely different capability than "Playwright can click buttons." One little feature, discovered by doing, unlocked an entire class of harness design for me. Now what I can do is:
- Make AI record bugs (so it's easy to reproduce)
- Auditability and reproducibility for free
- Also can make AI fixes for those same bugs :-)
Multiply that by 500. That's the point. You are not learning 500 things; you are learning 500 things that connect to each other. The compounding is the value, not the individual POC. Turns out I made a generic skill with the playwright feature discovery.
When you look at several things and iterate, your brain starts picking the patterns. POCs and experimentation are great for that. It's how you acquire taste.
Discovery and Delivery
Emergency: repeat → discover patterns
Here is a real one. During an emergency, I needed to poke at Redis, then Cassandra, then MySQL. Different clients, different syntax, different mental context switches, all while the clock is running and people are on a bridge call.
I did the same painful thing enough times that the pattern jumped out: I don't want three consoles, I want one generic console. So I built it as a POC. If you work with many different stacks, you can see the value of a centralized developer console.
That's the pattern of the pattern: repetition reveals structure. But you only get the repetition if you are actually in the arena doing things. If you outsource all your hands-on work, you never feel the friction, and if you never feel the friction, you never build the tool. Pain is data. POCs are how you turn the data into patterns.
Discovery: 4 years → 4 weeks
This is the number that still messes with my head. Work that took 4 years in software could be done in 4 weeks today. But be careful with how you read that. It is not 4 years of thinking compressed into 4 weeks. It is 4 years of typing compressed into 4 weeks. The delivery got roughly 100% faster. The discovery got 0% faster.
Discovery is still bounded by human understanding and retrospection. You still need to know what problem you are solving, what the tradeoffs are, and what "good" looks like. The model does not know that and will not tell you. As I said in AI Shift Left, we have a two-phase SDLC, one for discovery and one for delivery. AI absolutely demolished the delivery phase. Discovery is untouched.
That's a big IF — only if you do it right. Either way, the constraint moved. If delivery is cheaper and discovery is not, then discovery is now the whole game. Guess what makes you better at discovery? Doing 500 POCs. AI does not automate product judgment, but it dramatically lowers the cost of producing evidence for discovery.
AI can accelerate:
- Prototypes.
- Alternative implementations.
- User-testable interfaces.
- Technical feasibility checks.
- Competitive experiments.
It does not automatically accelerate:
- Understanding customers.
- Resolving organizational conflict.
- Choosing tradeoffs.
- Developing taste.
- Deciding what matters.
Culture Change
Re-wire your Risk Perception
This is the subtle one, and honestly the one I did not expect. After a few hundred POCs, my sense of what is risky changed completely. Things I would have called "too big" or "too risky" three years ago now look like a Tuesday. My baseline for what's possible moved considerably.
But the more interesting effect is the opposite direction: how fast can you say NO?
That's the real skill. Everybody talks about how fast you can build. The senior/staff move is how fast you can kill. When a POC costs you an afternoon, you can run the experiment, get a real answer, and say "no, this doesn't work" with actual evidence instead of vibes. You are no longer arguing from opinion. You are arguing from a repo.
Saying no fast, with proof, is anti-fragility. Every dead POC makes the surviving decisions stronger. You are not avoiding failure; you are buying failure at a heavy discount, on purpose, before it can hurt you in production.
Experimentation Culture
Experiments lead to discovery. More experiments lead to more discovery. This is not complicated, but organizations are structurally hostile to it. Most companies say they want innovation and then build a process where trying something new requires three approvals and a business case. Because bigger companies are risk-averse, they have a lot to lose; what they dont know is that doing nothing is not safer. That's not a culture of experimentation; that's a culture of permission. If the cost of an experiment is a meeting, you will have very few experiments.
The cost of an experiment right now is an afternoon. Teams that internalize this will pull away from teams that don't, and it will not be close. This is also where the transformation vs accumulation trap shows up again: buying Claude Code licenses is accumulation. Rewiring how the team decides what to build is transformation. Only one of those matters.
How can you get better?
- Explore: Build something you never did: a macOS app, a Chrome extension, a game, a real-time application, a new tiny tool that solves a problem for you.
- Try it out: Build something, then iterate, think about 2 features, use the system, think about 2 more; it's all smooth. Is anything in the experience better?
- Learn: Go understand the code; it's a pattern you don't know? Is there a technique? Would this break at scale? Did you try to get quizzed? Or try to explain?
- Throw Away: Throw away what you did; can you do something in the opposite direction? Can you think of a different strategy to solve the problem?
- Push yourself: Do something hard, do something you tought was impossible, then learn how it was done. IF AI does it fast, it's not hard enough.
- Find what's missing: It's not going well; does the AI need context to remember? Try build a game pointing assets, see the difference? Is there something missing in your ask? What is it?
