Fighting Complexity
What is complexity?
Complexity can be seen as the quality, degree, or state of being intricate, complicated. Complexity can be a spectrum from 0 to infinity. The biggest issue with complexity is that is not fully visible, we only understand partially. Learning how to see complexity, matters, otherwise, how can we solve it?
Complexity is often confused with style and preferences. One engineer might prefer OOP over FP or vice-versa does not mean one is better than another. Complexity cannot be 100% solved in some cases, complexity needs to live somewhere, you can hide it but not always kill it 100%.
For me, complexity is like this:
Obscurity is the definition from the philosophy of software design. Is the degree how which things are dark, confusing, and difficult to understand. Obscurity is relative, one engineer might find something complex another might find ok, because there are different backgrounds and levels of knowledge.
Waste is often present or the result of complexity. Waste is not good, is burning money. Lean was always about seeing and managing waste, lean always manifested in fields and areas where waste was big, such as manufacturing, hospitals, and the technology industry.
Maintenance suffers from complexity a great deal. Simple systems are easier to maintain and are more efficient. Complexity creates inefficiency and increases cost at the same time reducing the speed of delivery.
Complexity affects your understanding of the system. Such understanding is not only to deliver features but also to perform migrations and troubleshoot complicated distributed problems.
Complex Systems are wanted!
The big problem is, we want things to be complex. We want more variables, we want music with many instruments being played at the same time, and we want businesses that are multi-regions. Companies want to have multiple businesses to diversify and be more resilient, and the company wants to buy other companies, which leads to complexity.
Complex != Complicated
Complex systems are wanted but at the same time design, code, architect, and maintain systems using simplicity as a strategy. Complexity is increasing and is wanted, so what can we do about it them?
BIG-O: We know complexity right?
Big-O notation is a form of fight against complexity. By understanding the time complexity of a algorithm or code we can see how that code will scale and know if will be acceptable or not. Big-O requires we understand how complexity manifests in the code (by loops).
Understanding time complexity using big-o allows us to see what is wrong or in other words, see the complexity and then reduce the complexity with a better algorithm. Now, how can we see complexity in software design? How do we see complexity in software architecture? How do we see complexity across a huge service ecosystem and countless decisions?
We do not have big-o for all these things, unfortunately. We can learn sources of complexity and hidden form of complexity like we learned that for inside for it not good.
Sources of Complexity
There are many sources of complexity, here are some:
Complex Interfaces: Interfaces matter a lot, having clean and simple interfaces matter a great deal. Having complex interfaces is worse them having a complex implementation. Interfaces are contracts and should hide the implementation. An interface that doesn't have the interface is not useful. Properly abstracting or "hiding" the complexity is not a trivial task and sometimes requires: rethinking the flow of the system, rethinking the UX, pushing back on requirements, and introducing restrictions.
Feature Bloat: Unnecessary features, rarely used features are sources of complexity. Consider having counters on all UIs(frontend, mobile, web) so you can tell if the features are being used or not. Very few companies remove features. When you go to a library, the books are managed. When you go to an art exposition or concert the content is managed, some things you need to delete, what is the last time you saw a feature being decommissioned?
Hidden Costs: Hidden costs are another interesting source of complexity. Companies think that just acquiring companies or buying commercial software is a free lunch. When you "buy" software you need to integrate it with your internal system, buy a lot of systems and you will have complexity.
Don't Build and Don't Buy: Companies that only buy or only build also generate complexity. We need to have some balance between build vs buy. It is very easy to build and not realize the integration problems that need to be solved. However at the same time is easy to fall in love and build everything and never buy, the result could be poor implementations of systems that are not even close to your core business.
Conway's Law: The law states that:
"Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure" -- Melvin E. Conway
Teams tend to produce architectures and solutions that mirror the team structure. Platform teams tend to produce libraries, frontend teams tend to just do UI code, and backend teams tend to create microservices for everything. Such an effect creates complexity.
Shifting Complexity
A good design can reduce complexity a great deal. The right tool and the right language for the job can and will reduce complexity. However complexity and shift to different layers. For instance, we might decide to code some feature or piece of code in a service. Another team might decide to push to the BFF. A third team might think this is generic enough and want to push it to a library. The PO might think is better just to buy some SaaS tools and code the integration. Complexity is bounding from all these "places" or "locations" but not really removed.
If you really want to remove complexity, you need to see the big picture, see end-2-end and understand the whole. If you are a frontend engineer is easy to just focus on the UI, if you are a backend engineer you could just focus on the backend code, but you need to do both, look from UX to Infra to them understand the feature deeply.
See the whole is a Lean principle. It's key to fight complexity and avoid local optimizations that make no sense seeing the global whole picture.
Fighting Complexity
So how do we fight complexity? Well, we understand it and approach it with an analytical mindset. Being humble is a good way to fight complexity, meaning assuming there is more, don't think you got it all, expecting the unexpected, and looking for unknown unknowns.
Proof of Concept (POC): A simple tool to keep us humble is doing code POCs. By doing POCs we will be able to see parts of the complexity we need to deal with. POCs are great tools for discovering and understanding what we are dealing with.
Trade-offs: Trade-off analysis is important to fighting complexity, by running multiple scenarios and simulations we reduce the chance of poor decisions, the fewer variables you see, the biggest the change you are not taking an informed decision and therefore would be making things worse.
Right Principles, Right Design: Principles are everything. Following the right principles to reduce complexity, principles like Loose Coupling, High Cohesion, Clean Interfaces, and SOA principles are the way to go.
Fighting complexity is like the last Avengers movie: Infinity wars, it never ends. Complexity is not going away anytime soon. Hope is not a strategy like Google SRE principles teach us. We need to manage complexity and fight at all times, but not with fear, not with just restrictions. Fighting complexity requires education and knowledge sharing, passion, and care.
More and more I'm convinced that our game has changed. The world is not what used to be, we can only win if we care, we need engineers that love what they do, have passion, and relentlessly improve no matter what.
Cheers,
Diego Pacheco
