Posts

Evals need to be end-2-end

Image
Evals need to be E2E If you build AI agents, sub-agents, and agent skills. You know you need evals; that can be tricky. If you run agents inside your organization in our internal systems and services, it's pretty simple to collect evals. IF others and external companies run your skills/agents, then the story is a bit different, as you might not have control of the whole execution context. Its All under the testing umbrella There are 3 things that have a very close proximity: tests, observability and now evals. Testing includes classical pre-production and post-production tests, like unit tests, end-to-end tests, integration tests, chaos testing, stress testing, property-based testing, fuzzy testing, CSS testing, contract testing, and more. These can be done before production and in lower environWhich can be done before production and lower environments or in production. Production testing also encompasses things like canary and observability, which are forms things like canary and...

Burning Tokens

Image
Burning tokens is a double-edged sword. Powerful weapons can either give a lot of leverage or drag you down. It's easy to think that you are not shooting yourself in the foot with unlimited tokens. The thing is, you won't be shooting yourself in the foot; you will be expending a lot of company money and perhaps with lots of amplified waste. But only if you do it wrong or don't know what you are doing, and that's the very tricky part. Big Tech and big companies already have unsustainable token usage, like Uber . The list goes on and on: Microsoft , Walmart , Meta , JPMorgan , AT&T , and many others. For some, tokenmaxxing is over. Is it? Really depends on the cost of inference. Jev is showing that a classifier can be powerful and reliable, at least in the sense of low cost and speed, not saying anything about accuracy, which is not the score. The problem is not using tokens; that is the symptom. The problem is a lack of training; there is a huge AI enablement GAP ...

AI Slop

Image
AI Slop is everywhere. Technical debt always existed, but sure, slop will surpass it; somehow we managed to make things worse: Human-generated, handcrafted tech debt blended with synthetic AI generations. I won't say productivity is up because that's hard to measure; for sure, output is up, but so are bugs and incidents. The less you look at the code, the less you pay attention, the more slop you get. Attention is all we have; it's precious. The catch is, to produce more, you need to look less. It takes 5 minutes for the agent to produce the code and 5 hours for you to validate and understand. Multiply that by several PRs a day, and we have a big attention problem and slop proliferation. AI Slop Definition I think it would be fair to ask frontier LLM models what they define as ai slop. I had to ask to be max 2 lines; otherwise, it would be slop 🙂 GPT has the most interesting definition for me, specially because it mention judgement and quality. IMHO what we are also missin...

Lessons Learned from 500+ AI Pocs

Image
In the summer of 2024, I wrote 300+ AI POCs; in 2 years, I have 550+ AI pocs. Before I cover the lessons learned, we need to address somethings. I've been learning technology for the past 23 years since I started working professionally with software; I never stopped. I always learned multiple languages every year and always liked to explore , even before AI. For 19 years(since 2007), I have blogged about technology. Before AI, I always believed that in the software profession, you need to know what you are doing in order to do it right. I learned languages that I would probably never use in production, just for the fun and sake of learning. I always believed that when you're too narrow in your current needs, you would miss the big picture and opportunities. YES, I always learn and research problems I was solving, but never confined myself to only short-term features. So I was doing POCs before AI, and I will do them after AI. Now we need to understand why POC are getting a bad ...

AI Moves Fast. Decisions Move Slow.

Image
More tokens are not more intelligence. Still, it is easy to get hooked on token usage. Is it learning, experimentation, or just FOMO? Subscription models trained us to extract as much as possible from a fixed monthly payment. But when more usage creates more code to review, why do we need to max everything out? Are we getting more from AI? Or are we repeating an old consumption habit with a new resource? I was reading and watching Gergely Orosz’s conversation with Dex Horthy, “Context engineering with Dex Horthy” , and it made me think about fast loops, slow loops, token harder, and token smarter. Faster loops do not necessarily accelerate value delivery. Unless the lead time decreases and the value increases, we may be trapped in local optimization.

Handling Ambiguity in the Age of Agents

Image
Ambiguity is not simply having a difficult problem. It is not knowing exactly what the problem is, what matters most, or which direction will create value. It appears during requirements, scoping, analysis, and design—across the whole solution space. It also appears across teams: product, design, engineering, and the business. Ambiguity has always been one of the hardest parts of engineering. Agentic AI makes it worse at scale. A human engineer receiving an ambiguous request may push back, ask questions, make assumptions, or stall. An agent immediately produces something: code, commits, documentation, and a confident summary. The output looks like progress. It may have little to do with the outcome anyone wanted.  We always had this problem, but now agents are a huge employer.

Loop Engineering

Image
Software engineering keeps changing names for the same old problems. Before 2023, changes were introduced with some level os stability; now, changes happen much faster, to the point that we do not really understand the problems. We had those problems in the past, considering things like (agile/scrum, microservices,  devops), but never has it been a fast loop like this. First, we had scripting. Then automation. Then CI/CD. Then DevOps. Then Platform Engineering. Then AI Agents. Now we have Loop Engineering. However, companies have barely functional CI/CD due to a lack of incentives, poor vision, poor management, and other dysfunctions.  Loop engineering, the name is new. The problem is not. The problem is: how do you make a machine do useful engineering work without babysitting every step? That’s it. But "solution" and "waste" manifest in the same way, using the same tools. Loop Engineering is not just about prompting with a fancy jacket. It is not “write a bette...