Breaking the Debt cycle

Organizations need Lean more than ever. IT became a huge waste factory. Software scale works backward, the more money you throw in, the bigger the project the bigger your risk and bigger the waste you have. The technology work backward because If you think about Milk, the bigger the scale the cheap is the price. I was wondering how did we get into the position we are. Tech debt does not describe debt anymore, we are dealing with a totally bigger order of magnitude scale of debt. Tech debt can be big but I'm talking something beyond big. Maybe at the scale that some governments have debts with the IMF. Why did we go south without noticing? I have some theories. Technology is like a time bomb, companies might start breaking because of it at some point. The debt become so high that multiple 3-5 years of investments might not be able to pay then off and business might start failing because of the bad technology practices. You might think I'm crazy. Because of business operates with products and customers. However, we need to keep in mind today most of the products mean == Technology. Therefore even if you have an amazing user base and the working product you might get stuck in a very dangerous cycle.

Success Definition

PMI says that a project definition of success means to deliver on time, on budget and with the expected scope. The PMI definition of success is hardcoded in the brains of most business organizations.

This success definition requires Estimates. Estimates are guessed, they never work but we still use then as they were something reliable(which they are not, therefore we cannot trust then). Estimates are often done by managers who do not understand how modern software developers, product discovery works and often that happens 1 year before the project start targeting 3-5 years duration. Why? Because finantial / sourcing departments believe that buying more software is cheaper - they are wrong, it's not, It's the opposite. As a software engineer, you will be PUSHED to deliver somebody else guess. This practice is being around for decades(2-3) and companies are being around for decades, companies buy companies that operate with the same principles so debt piles up like crazy.  It's would be much better to have another definition of success, being more agile and product-oriented rather than being project-oriented.

This is my proposal for a better success definition. First of all, we don't care about OUTPUT, only about OUTCOMES. So it does not matter if we get or guesses right(nobody does). What matters most is deliver value(Agile says that for +20 years and companies still did not get it.) Value can be perceived as different things like user impact such as improving user organic growth or reducing churn.  Predictability is value as well, however, in order to have it, you need several things in place. I will talk about predictability latter on.

Acquire New Skills and Culture are value as well. Several times companies fail to perceive the impact New Skills and Better Culture can have on results in the long term. These factors often are not taken into account to judge initiatives. This really does not make sense because people do software, so if your product will be defined by your people, if you do not worry about what skills they have and culture you really are neglecting results and creating debt.

Reduce Debt side by side with Reduce Waste are more important than ever. Technology becomes about waste, inefficiency, slowness, lack of delivery. Money comes in debt comes out. Debt needs to be managed and tackle with much more serious discipline. Successful product companies know that. In theory, everybody knows that in practice companies prefer to prioritize best which kills their speed to product value in a long rung.

Better user experience, it might sound funny but the industry still does not prioritize UX. UX is not only about to look and feel but about the whole experience from the tone, the interactions, how it works, how to fix the user pains and problems to how reliable, fast and great the product(system) is. More features do not means more. Users care about fixing their problems, not about number of features.

Predictability Iceberg

Every single company out there wants predictability. That's the only thing they see. However thats only the tip of the iceberg. I would not have enough time and space to draw and the universe that exists under the predictability iceberg. For most of the industry, this iceberg is very shallow and could mean simple and linear things like estimates, writing specs, having and architecture. Unfortunately, thing are more complex.

Predictability depends on a stable system, Dr. Deming said that ages about, companies still did not get it. I share very interesting Deming experiments on one of my previous posts. The stable system requires talented people working on the right roles with the right practices. Today is almost impossible to be productive without a proper cloud platform in order todo Deploys, Observability, Tests, A/B Testing and much more.

SOA plays a very important whole in this picture. having contracts and proper ISOLATION allows you not only to contain blast radius in case of failures but also to contain debt. Hiding databases are one of the most powerful architecture techniques we have. The main issue is 20 years ago have a central monolith database was the way to go and since the industry never much attention on this "debt thing" you what happed.

Proper Discovery might be one of the most important things you need todo. Discovery disciple allow you to answer the most important questions for your business continuity:
* Are we building the right product?
* Are we fixing the right problems?
* Are we targeting the right users? Do we know our users?
* Are we doing the right strategy? It's our MVP too big?
* What should I modernize what should I re-create?
* What are the right priorities?

Proper Kamban is the bread and butter of continuous improvements. If you are not doing continuous improvements you are creating debt. Continuous improvement is a nice and beautiful world to say but in practice, few know who to do it. Why? Because in order to tune the system you need to understand it very well. Improvements are done in several fronts by?
* Running RCAs(Root Cause Analysis) and preventing bugs to happen again
* Using WIP limits and control utilization and see bottlenecks and act on them(Test/Code, Acceptance).
* Having flow / regular cadence by having more queues and always improving explicit queue policies like the minimum number of tests, documentation, and review your own code before submitting to the Code Review Queue.
* Training your team and teaching then how to be more effective and productivity
* Having retrospectives in challenge your tools, process, and practices.
* Having regular DEMOS to collect customer feedback and improve the product
* Re-thinking your architecture in order to reduce complexity and boilerplate
* Creating generic solutions which reduce developer workload
* Having platform services to allow engineerings to be focused on the business.
* The list goes on and on...

Automation, Observability, Automated Tests, Templates, Guides. I will stop because this list is much bigger than the one I add on the iceberg picture.

Too Much Debt: Vicious Cycle

In the beginning, teams tend to be super fast. As time pass, people leave the company and the business change the code became more and more complex as a consequence the ability to quickly iterate and release software becomes slower and slower due to the debts.

The picture above show how LED time deteriorates overtime and entropy takes over at some point. Adding more people actually makes it worse because we are only adding more VENON into the system so it might speed up this death spiral. IF I dont have a stable system with 2 people adding 20 won't make my life easier, actually will make worst as debt will be produced faster and the need to coordination will be a higher price. In order to add more people properly without suffering you need:
* Great SOA Architecture in place with proper ISOLATION
* Great Talent Density
* Stable System

Without these requirements we are only making the bad vicious cycle spin.

As you produce Features DEBT is created(Did you know?) DEBT increases complexity, complexity slow down you productivity and you produce fewer features. This cycle tends to repeat ever and ever and companies dont understand why they cannot meet the deadline. One interesting factor is that as you have more teams you have more dependencies and if this dependencies are binary(share same tables, code bad, open tickets) you slow you progress and debt even more. Your dependencies show be services which are deployed and reduce the debt, therefore, speed your progress.

So what do you want to do is reduce the investment and maximize the impact and also produce little debt.

Higher benefits, lower debt, and lower investment should be your top priorities. But what happens if my initiative takes 5 years or more. Well, work with 3-month budgets which will help you to reduce the waste and maximize the benefit.

Unfortunately what happens is the other way around, where companies invest a lot and the benefit is not that big and the debt is actually bigger than the $$$ investment. Companies often not prioritize refactorings, re-architectures, and prefer to "integrate" on what they already have, this only pushes the problem to be bigger.  The issue is that the product might not see the benefit in playing the debt system not "perceived" feature/capability will be added to the final user however this need to be taken into account otherwise the long-tern ability to add benefit might be compromised forever.

Breaking Free

First and foremost education is key. Do you know how much DEBT do you have? Could you DEBT be paid by your investments? If you dont know your debt and/or is not measuring it effectively, you will be in trouble or maybe you already are.

Making hard calls like refactoring / re-architecture / re-design instead of integration and migration as-is or lift-and-shift are very necessary, even if they dont have ZERO new features. The feature is not the only thing that matters, seeing just features create huge hazard to the business.

MVPs are great tools. The main issues people don't know how to run proper MVPs. in order to run a proper MVP, you need to focus on a "small" movement possible, doing discovery we do MVPs without code and this are the best MVPs. MVPs mean validating assumptions which can be done by talking with your REAL USERS or running UX Research or A/B testing in production. You can do a software MVP but you need to reduce features and inside features, you also can do less.

Fixed Scope projects should be avoided at all costs. Cheap and "obey-only" vendors also need to be avoided at all costs because they create debt bigger than the benefit. Total Tercerization also does not work because a software company might have part of the puzzle and the business org might have the other part so PARTNERSHIP is the only way.

Small Budget allocations like (3 in 3 months) help to ease the financial pressure and also produce less debt and waste, movements like OKRs, NoEstimates, Byond Budget, Continuous Digital enforce this culture

Ultimately breaking free is not easy because require to convince a lot of people but I deeply believe we need to evangelize for change otherwise software might kill itself and take the business with it.

Cheers,
Diego Pacheco