Internal System Design: The Forgotten Discipline

Several years ago it was impossible to think about software development without having at least a business specification(usually in a word document) and a bunch of UML diagrams like package, class and sequence diagram. That was a pre-agile era where process and formality ruled the IT software development industry(yes folks, yes I'm old). Back on that time software engineering was pretty much conceived as "process".  Now we live in a more civilized era, however, back on that time, there were some interesting and valuable things(tools, techniques, principles and even practices).  Like the English expression, you need throw bath water but keeping the baby. I was wondering why these phenomena happened and still happen today. First of all, we are SOCIAL animals we tend to have some kind of "fashion" behavior where we follow the "flow". Needless to say that UML and Software internal design is not quite popular today with the new kids. I don't miss the process-formality-oriented times don't get me wrong. Back on that time, people used to create things for sake of creating things, lots of times we were doing simply raw CRUDs and really this was just a WASTE. However, sometimes you do have things that are very complicated to do. Plus the fact that we work in TEAMS and when you have to make several people CODE at the same time a COMPLICATED and DISTRIBUTED system, well things can get messy. For such times I think to make sense get back to the roots and revive some of this arcane tools and ideas but without the process-craziness :D

How to Build a plane on the Fly

So let's say your time need to work in a complicated system(green field) from scratch all together. So what do you do? Start coding like crazy? Well, that might be the worst thing you can do. Because is very likely you will create tons of technical debts. So if your team create a sort of mid-level design of the internal system would be possible to parallelize the work in a sane way. So if you need to build a plane you can do the right wing and somebody else could do the left wing without having the fuselage and that should work. The rationale behind it is the same core thinking we see in SOA(Service Oriented Architecture) and Microservices. The main difference is that one if EXTERNAL and the other is INTERNAL. So if we talk about microservices as long as you have the CONTRACT you can mock the service and build a dependent service without the original service being done. Same property can be applied to internal system design.

Economically speaking this approach will be way more efficient and productive way less technical debts. This does not mean to create a BDUF(Big Design Up to Front). Why not? Because you can and should validate the core ideas before creating this internal design. If you do some small-scale and focused POCs is possible to validate the plane on a small scale and then you can produce it on a larger scale. This is one of the main difference from the approach in describing now from the approach was used on the dark age(where folks design a BDUF via committee and then when some poor developer tries to code if they so that nothing works) so that's one big difference. When you stop to think about this you might realize this LEAN thinking. Look Lean Startup movement for instance where we validate the ideas, market-product-fit before building the solution. It's the very same principles.

In a Lean Startup, we analyze the landscape(industry) that we are targeting in order to compare the solution with other startups and solution providers. For engineering, we can and should do the same - that why we can compare with existent framework and solutions and analyze what they do well and what they don't so well and why. Doing so it will make our core value more unique in this case when we talk about engineering it will reduce the chance that we make same mistakes that other did - that's why is very important to read papers, see videos, look code and understand the problems you are about to face.

Typing is not the Bottleneck

the first time I came across 'typing is not the bottleneck' was in @GeePawHill's 2009 post: https://t.co/v1FNigZpxv - @sbastn then created the monkey stickers https://t.co/YqYcv3JcLr pic.twitter.com/iKLRP0ns14

— Philip Schwarz (@philip_schwarz) February 26, 2018

Don't get me wrong, I'm a hacker but I do have my think that as well. Some problems are fairly trivial and it would be wast add some internal design load on it, however, some problems are complicated and make sense to think before doing. Since thinking could be almost as fast and the speed of light, typing definitely has more latency. The code is just a way to EXPRESS the solution, we think with our brain not with our fingers AFAIK.  Rushing up to code won't make the problems go away. Thinking about the problems will help to address the problems. This might be obvious however in daily basic we often forget that. Several companies might also consider that you are not working if you don't have your IDE open and are not typing code.

When you just think about the code you often lose powerful abstractions. So let's just drawn diagrams and don't ever open an IDE? No, not at all. Extremes often lose important things. Without coding, there is no delivery. We deliver work via code. However, engineering is way more than just typing.

Packages, Interfaces and other Contracts

Packages group similar classes. Similar classes can be grouped by a "concept" this concept represents what the class is responsible to do. This is a SOLID and RDD design principles. These design principles are capture better when used with Color UML. Color UML we use different colors to represent different "responsibilities" or "roles". Having said that the number of packages can be a design smell that might say that if you have several packages, you have several concepts so you might have a module/package that is doing too much and you might:

• Overcomplicated things and shall refactor the code/design
• Should break the module in more modules
• Re-design since you might have a leak in your design, therefore, the wrong abstraction

Having said that you have several elements of your internal contract that is composed of packages, interfaces, and methods of course. All these items are like the plane assembling map that you saw at the beginning of this very post. This map important because allow your team to speed up and parallelize work in a sane way. 

Internal Design is really that important?

Yes, it is. That's the same thing as asking if you should have clean code or write unit tests. We all know this is not about quality but about economics actually. Saving money, time and energy in a long run. However we need to keep in mind we are living in a system of system world where we have cloud-computing, microservices, kubernetes, serverless. Why that matters? Because more and more we SHIFT responsibility from the internal systems(microservices) to the upper layers(cloud, kubernetes, serverless platforms) this is one of the big reasons why you can leave with a bad internal system design in a microservice world because that bad design is ISOLATED. However, if you are building distributed systems(a.k.a not only microservice this become more and more important).

A monolithic/duplicated code is better than bad Design

Doing a good design requires time and experience and study. So if you working on something that you don't have experienced is way-way-way better to avoid over-engineering and do simple and monolith things. It's much better to have a single class with 10 methods and 1k lines of code than 10 classes of 10 methods because the duplicated code is A) decoupled. B) easy to refactor and understand. A bad design might be way harder to refactor than monolith code. The same thinking is applied with microservices.

The evolutive design is about that. Doing things in waves is better than a single shot delivery. Needless to say, it's agiler, has less chance to fail, improve the sense of progress, since you are delivering more tasks. This can be achieved with a very simple kanban principle(reducing WIP limits to increase throughput) how? Limit your pull requests in 10 files. I did that with my team and that's was like pure magic.

cheers,
Diego Pacheco