Internal System Design: The Forgotten Discipline

Several years ago it was impossible to think about software development without having at least a business specification(usually in a word document) and a bunch of UML diagrams like package, class and sequence diagram. That was a pre-agile era where process and formality ruled the IT software development industry(yes folks, yes I'm old). Back on that time software engineering was pretty much conceived as "process".  Now we live in a more civilized era, however, back on that time, there were some interesting and valuable things(tools, techniques, principles and even practices).  Like the English expression, you need throw bath water but keeping the baby. I was wondering why these phenomena happened and still happen today. First of all, we are SOCIAL animals we tend to have some kind of "fashion" behavior where we follow the "flow". Needless to say that UML and Software internal design is not quite popular today with the new kids. I don't miss the process-formality-oriented times don't get me wrong. Back on that time, people used to create things for sake of creating things, lots of times we were doing simply raw CRUDs and really this was just a WASTE. However, sometimes you do have things that are very complicated to do. Plus the fact that we work in TEAMS and when you have to make several people CODE at the same time a COMPLICATED and DISTRIBUTED system, well things can get messy. For such times I think to make sense get back to the roots and revive some of this arcane tools and ideas but without the process-craziness :D

How to Build a plane on the Fly

So let's say your time need to work in a complicated system(green field) from scratch all together. So what do you do? Start coding like crazy? Well, that might be the worst thing you can do. Because is very likely you will create tons of technical debts. So if your team create a sort of mid-level design of the internal system would be possible to parallelize the work in a sane way. So if you need to build a plane you can do the right wing and somebody else could do the left wing without having the fuselage and that should work. The rationale behind it is the same core thinking we see in SOA(Service Oriented Architecture) and Microservices. The main difference is that one if EXTERNAL and the other is INTERNAL. So if we talk about microservices as long as you have the CONTRACT you can mock the service and build a dependent service without the original service being done. Same property can be applied to internal system design.

Economically speaking this approach will be way more efficient and productive way less technical debts. This does not mean to create a BDUF(Big Design Up to Front). Why not? Because you can and should validate the core ideas before creating this internal design. If you do some small-scale and focused POCs is possible to validate the plane on a small scale and then you can produce it on a larger scale. This is one of the main difference from the approach in describing now from the approach was used on the dark age(where folks design a BDUF via committee and then when some poor developer tries to code if they so that nothing works) so that's one big difference. When you stop to think about this you might realize this LEAN thinking. Look Lean Startup movement for instance where we validate the ideas, market-product-fit before building the solution. It's the very same principles.

In a Lean Startup, we analyze the landscape(industry) that we are targeting in order to compare the solution with other startups and solution providers. For engineering, we can and should do the same - that why we can compare with existent framework and solutions and analyze what they do well and what they don't so well and why. Doing so it will make our core value more unique in this case when we talk about engineering it will reduce the chance that we make same mistakes that other did - that's why is very important to read papers, see videos, look code and understand the problems you are about to face.

Typing is not the Bottleneck

Don't get me wrong, I'm a hacker but I do have my think that as well. Some problems are fairly trivial and it would be wast add some internal design load on it, however, some problems are complicated and make sense to think before doing. Since thinking could be almost as fast and the speed of light, typing definitely has more latency. The code is just a way to EXPRESS the solution, we think with our brain not with our fingers AFAIK.  Rushing up to code won't make the problems go away. Thinking about the problems will help to address the problems. This might be obvious however in daily basic we often forget that. Several companies might also consider that you are not working if you don't have your IDE open and are not typing code.

When you just think about the code you often lose powerful abstractions. So let's just drawn diagrams and don't ever open an IDE? No, not at all. Extremes often lose important things. Without coding, there is no delivery. We deliver work via code. However, engineering is way more than just typing.

Packages, Interfaces and other Contracts

Packages group similar classes. Similar classes can be grouped by a "concept" this concept represents what the class is responsible to do. This is a SOLID and RDD design principles. These design principles are capture better when used with Color UML. Color UML we use different colors to represent different "responsibilities" or "roles". Having said that the number of packages can be a design smell that might say that if you have several packages, you have several concepts so you might have a module/package that is doing too much and you might:

  • Overcomplicated things and shall refactor the code/design
  • Should break the module in more modules
  • Re-design since you might have a leak in your design, therefore, the wrong abstraction
Having said that you have several elements of your internal contract that is composed of packages, interfaces, and methods of course. All these items are like the plane assembling map that you saw at the beginning of this very post. This map important because allow your team to speed up and parallelize work in a sane way. 


Internal Design is really that important?

Yes, it is. That's the same thing as asking if you should have clean code or write unit tests. We all know this is not about quality but about economics actually. Saving money, time and energy in a long run. However we need to keep in mind we are living in a system of system world where we have cloud-computing, microservices, kubernetes, serverless. Why that matters? Because more and more we SHIFT responsibility from the internal systems(microservices) to the upper layers(cloud, kubernetes, serverless platforms) this is one of the big reasons why you can leave with a bad internal system design in a microservice world because that bad design is ISOLATED. However, if you are building distributed systems(a.k.a not only microservice this become more and more important).

A monolithic/duplicated code is better than bad Design

Doing a good design requires time and experience and study. So if you working on something that you don't have experienced is way-way-way better to avoid over-engineering and do simple and monolith things. It's much better to have a single class with 10 methods and 1k lines of code than 10 classes of 10 methods because the duplicated code is A) decoupled. B) easy to refactor and understand. A bad design might be way harder to refactor than monolith code. The same thinking is applied with microservices.

The evolutive design is about that. Doing things in waves is better than a single shot delivery. Needless to say, it's agiler, has less chance to fail, improve the sense of progress, since you are delivering more tasks. This can be achieved with a very simple kanban principle(reducing WIP limits to increase throughput) how? Limit your pull requests in 10 files. I did that with my team and that's was like pure magic.

cheers,
Diego Pacheco

Popular posts from this blog

C Unit Testing with Check

Image
When I start on IT, several years ago(+15), C was the first language that I learned. But was not the first language I mastered and deployed in production which was...surprisingly Visual Basic 6.0. However, when I learned C or any language back on that old and arcane time noddy was into testing. However now is impossible to work professionally in any language without tests. C is a low-level language but still kicks ass and has several interesting libs/frameworks. People use to fear C a lot because of lack of syntactical sugar but once you get used to it is not that hard at all. Today I want to show how you can do Unit Testing with C. I will not show how to do Unit Testing with C++ but with C ANSI actually. We will be using Check a unit test lib for C ANSI.
read more »

Having fun with Zig Language

Image
Zig is a general-purpose language created in 2016 by Andrew Kelly .  Zig aims to fix the issues C language has while being pragmatic and simple at the same time. Zig really shines as being simple by not having any hidden control flows, not having hidden memory allocations, no pre-processors, and no macros. Basically, it's one language and one language only. Since pre-processors and macros can get really complex and become their own universe. Sure it's not the Zig Philosophy.  Zig Compiler(Zig cc) also is a C compiler, and the interoperability from C to Zig is very good. Clearly, we can see influence from other languages like Go and Rust; however, Zig is pretty unique and different from Go and Rust.  Zig is fast. Zig is perfect for system programming. I bet we will see a lot of solutions being built in Zig. As a languages aficionado, I like to learn at least one language per year and sometimes more; Zig was my 2021 pick, did not regret it. Even if you dont use all language...
Keep reading

HMAC in Java

Image
In 2018 AWS CTO(Vogels) said : "Security is everyone's job". For the last 2 years, there was so many famous and big data leaks and breaks that made that statement be very true more than ever. Security often could mean worst performance and worst user experience so in order to get it right you really need to think about the designs before jumping into to code and consider performance and user experience has main requirements.  I was thinking about writing about how to do HMAC in Java for a while and recently Redis 6.0.0 come out and to my surprise, there was a refactoring on the password part in order to use HMAC.  Security easily could scare engineers often because it is not something well spread yet but I believe this will change soon. It does not matter if you have to deal with PII Data or not, security is super relevant because everybody is running their workloads at the cloud or with IoT and Edge devices which means more distribution, more code, more points of fail...
read more »