Software Architecture Hidden Costs
The Role of Debt
Enterprise software means Debt. It's impossible to have a successful company without Debt. There are different sources of debt. It could be a result of poor architecture decisions, wrong technical structure(not having architects or business dictating tech decisions) or just ignoring architecture at the team level, not paying the debt over time or maybe other sources. Some engineers think about the debt they have but very few think about the debt they are creating. In other words, are you really fixing a problem or are you making it worst? Technical Debt has a huge impact on architectural decisions. Making it some possibilities affordable or not. Debt has an "erosion" effect. Overtime Debt gets worst and really ends up killing you.
It's not enough to acknowledge the debt before making any decision but also have a concrete strategy to help to eliminate or at least mitigate the debt in place. I always like the Boys Scout Rule(I was never a boy scout btw): Let the place better than you found it.
The Role of Reversibility
This is a big Amazon principle. Avoid one-way doors. You always want to have versability, meaning if can recover, rollback, or move to something else. Having two ways means you have more flexibility and allow you to change your mind in the future. This aspect is important because it could drastically reduce the impact of a bad decision or could give you the flexibility to shift gears as you learn and move forward. Not having reversibility makes some decisions much more critically and requires much more thinking and due diligence on top of it.
The Role of Isolation
Microservices are about isolation. Great architectures are about isolation. Isolation is not only about your Data Stores(databases, caches, configs) but also about making your contract: simple, lean, and easy to use. However, your implementation can be FAT and COMPLEX and that's fine because complexity needs to live somewhere. Sometimes to think about proper isolation is really hard because we are pushed to this "micro" level by several movements like microservices, nodejs, docker, serverless and sometimes you need a DETOX JUICE and step back and take a different direction.
Having proper isolation also means your decisions have less impact on the whole(if you were sure and correct about it). The reason I say this with "some doubts" is because is so easy to do binary coupling and have contract by accident nowadays. IF you get it right you wrong decision cost will be minimized and also you will have the flexibility for future changes.
The four Architectural Hidden Costs
So far we talk about 3 BIG forces that will make our life easier or harder, being: Debt, Reversibility, and Isolation. Regardless of what you have or not for each of those(cases by cases) now let's look into 4 hidden costs moments that you might not be accounting properly. The big 4 are:
#1 Build vs Buy Hidden Costs
#2 Design Hidden Costs
#3 Upgrade Hidden Costs
#4 Decommission Hidden Costs
Let's take a deep look on each of the moments and hidden costs associated with them.
#1 Build vs Buy Hidden Costs
As Software Engineers we build software so the first thing we want is to build new software. Because new software is easy and sexy however not always is the right decision. Because not always you should build, for sure you could build anything but the question is should you? I believe one important aspect is how much this is close to your core business or how critical is this for the core business in terms of differentiation. This decision is not easy, oftentimes you need to cross-check several aspects on a matrix-like evaluation taking into account:
* How long would take it to build it?
* How much it would cost?
* How much cost in the long term are you gaining?
* You have so few things you can build and manage it, does any market/oss solution is better?
* What features are you gaining?
* How many uses cases can you cover or not?
Sometimes does not make sense to build it. This changes from business to business and the important aspect here is to consider because you build it. Several companies dont think about just buying it or they just consider buying it, very few companies I worked got this right and establish the right balance. Perhaps this is the biggest hidden cost, could you imagine a solution that would not require you to build any software to much less software to archive it? It's could really be the ultimate cost saver. This is not one-size-fits-all-folks, I'm not saying let's always buy neither let's always build. One sample that makes a look of sense to buy is observability tools for instances or cloud infrastructure(how much make sense or do you really need to build your own cloud?).
#2 Design Hidden Costs
Design costs tend to be huge hidden costs because several times, there are design gaps, design flaws, design limitations that will turn out to be a huge issue for the business in the long run. It's easy to think that you dont have design issues just because your solution is doable or will work. Find design issues requires teamwork and lots of os thinking and is hard exercise but you can practice and you can get it better. Business that just want some features delivered no matter what often pay a huge price for wrong or incomplete designs. Bad designs manifest them selfs in forms of:
* Complexity
* Difficult to Reason / Understand systems
* Slow Delivery Peace
* High bug Rate
* High Effort to change
At the end of the day is a like a leaking bucket. You solution might work but you might be paying too much for something that you should not. So your value(money) is leaking away.
Not all changes are equal, it does not mean all changes will be expensive but for sure PowerPoint, change is much cheaper than release something in production :-) Software is not hard to change as Hardware, Boats, Bridges, Buildings therefore if you are agile mean you change. Well, it really depends, sometimes the change is super expensive. Remember Debt, Reversibility, Isolation, and how much something is decoupled or couple will make a huge influence here.
#3 Upgrade Hidden Costs
This is a very interesting one. We always think about how long to take to build something. If it is a new microservices maybe it is fine, however, if it is a shared services, platform library, or data service/engine well it's a completely different game. Because you might need to Refactor your consumer's code in order to use it. Them we have a different tension here, you build a better service or tool but would people actually use it? This might sound silly but the history is full of interesting stories like:
* Python 2 vs Python 3 (How long it took to python 2 to die? a lot :-) )
* Java 8 to Java 11 (Correcto is here to say java 8 still a long way to die)
* IP V4 to IP V6
No matter if you are using a 3rd party lib like Spring or Quarkkus, this is beyond in-site vs external-site provider relationship. Inside or outside of you company it does not matter, the dynamics are the same. Meaning the is a cost of "code migration" people need to refactor and use your stuff. Let's look at Frontend world like Javascript frameworks, there is zero COST, right? (Kidding) what happens is that the community is so used to refactor every 3 months and is fine they are always using latest. If you look java is the other way around, the culture for backend is why the hell are you making me change my code?
Just realize you might need to be backward compatible or have a migration script/program or help consumers to migrate their code or any least make sure it is on their budget time to then to work on migrating to your new stuff. Engineering organizations are a great pattern in SV and you have budget to build, but do you have budget to upgrade?
Why this is important? Well is the key difference between Discovery and HIPPO. Value Vs Perceived value. It does not matter the value you "think" it has and matter what consumers "really perceived". So value is on consumer perspective, for architecture is the same, if people can't afford your upgrades or don't see the benefit it's a problem.
#4 Decommission Hidden Costs
Finally the last hidden cost. If you dont delete code you will pay a high price in the long term. Because it will affect your ability to reason about your codebase and figure out whats used and whats is not used. It mind sound silly because we got this Gmail-like culture of never deleting emails, iPhone app-like culture of not deleting apps, at cloud(AWS|Google) there is infinite storage(corporate credit card-backed) so why delete? You need to. Sooner or later you will be migrating old code, fixing bugs you dont have it, and will be dragging you down with past decisions. If you don't want to take my word for it it's fine to look Lean principles for instance. Lean has one principle called Motion, and motion is part of the 7 classical Lean Wastes.
Dead / Zombie code / Data is so bad that it actually will be classified as other wastes like(Overproduction and Overprocessing) for refactoring and upgrading things you dont need it and Defects for Bugs.
Architecture is not only about tradeoffs and the right design give decisions/requirements but is also how you navigate over cost complexities and what strategy you pick. Understanding architecture hidden costs is important aspect of being successful meaning: People are using your stuff and you actually make the world a better place rather than just thinking you make it.
Cheers,
Diego Pacheco
