Refactoring: Making sense of the invisible

Refactoring is so underrated and misunderstood. There will never be enough books or enough blog posts about it. It's something that we need to keep revisiting from time to time. Very few people deeply understand it, but one thing is for sure: genuine and spread across the whole industry like wildfire: fear of refactoring. Fear can be a powerful feeling and an excellent way to keep you in check with reality, especially in difficult economic times. Still, fear can also be a significant source of blindness, inertia, and illusion, leading to bad economic decisions. Refactoring is not a binary decision; you either do it all and re-write the system or do zero-to-nothing. Companies and engineers need to apply more refactoring daily, and this matter is definitely unbalanced. Software at scale makes complete rewrites almost impossible. No company can stop the business for years and still succeed; many companies have tried and failed financially and in rewriting. So don't do more refactoring? No, that's not the way either. Some opportunities can exploited if you learn to see them. Most of the time, opportunities are not visible, and people and companies lose opportunities to improve things. However, if you know such opportunities exist, you might start behaving differently and making different decisions. Let's start seeing the problems.

Features vs Refactoring Ratio

Bare with me for a second. Let's say your team doesn't do any refactoring in a couple of sprints or even for a quarter(maybe more?). But you have bugs; some were created and fixed, and others are there because they have a low impact or affect very few customers. Rest assured no refactorings was performed.

In the previous picture, features are in green, bugs are in red, and refactorings are written in purple. Do you see any purple in this picture? No - nada! Because there were no refactorings. Pretty standard and expected. A senior manager could argue that this team is well-oiled, optimized, and result-oriented and is killing it. 

Now, if you talk to some engineers, they want refactoring pretty badly! Some engineers decided to push for refactorings, either in planings or during regular meetings; of course, the answer is NO. Such No comes with a variety of "reasons", which, if we are 100% honest, we know are just excuses like: 
  • We already committed to the business in this sprint and have no capacity.
  • We can't stop shipping features; we have no time to improve (classic: we are too busy to improve).
  • We are not confident the team can perform this refactoring. Skill issues?
  • We would need to spend multiple years rewriting the whole system, and we can't do that.
  • We are complicating things; this is rocket science. We don't need all that...
  • Maybe technology Z will fade in X years, but we better stay with what we supported.
  • There is no benefit to this refactoring just risks
To be absolutely fair, some of these concerns are far because:
  • Every time we code something, there is the risk we won't finish or it won't work out.
  • Plan and reality are different things; we might think we would have improvements, but they might lead to nothing.
  • Especially in difficult times, but really, at any time, you can't stop a company from producing; it's a very bad idea.
Now, let's forget about refactorings for a moment... (No, it's not a nothing burger post).

How the bugs are created

Remember, we are not doing refactoring; just discussing the possibility of doing refactoring puts people on the fence, and they are terrified of performing refactoring. It would be hard to justify, especially if the refactoring does not payoff. The thing that I fear is that it often is not logical or grounded on data. Let's recap one more time.

The team is not doing refactorings! Okay, some clever developer sneaky one refactoring without others realizing, okay. For sure, we are doing features, and we have bugs. If we are just doing features and bugs and still have bugs, we need to ask the question: "Where are the bugs coming from?" surprisingly, the answer is they are coming from features. All the bugs are coming from features and sometimes from other bugs. Now let me ask 2 questions:
(A) Do you see people afraid of doing features? Do you see management afraid of delivering features? 
(B) Do you see people afraid of fixing bugs? Do you see management afraid of bugs creating bugs? 

There is no such thing. There is only fear of refactoring (which is not being done) - So is this rational? Does this make sense? No engineer says, "Hold on your horses, we have way too many features I'm afraid of doing more".  Now, you see engineers with a fear of refactoring. 

The same code for everything! Whether it is to add features, bugs, or factoring, the fear should be the same. OH, but what we have must be working; we refuse to believe it does not work. Therefore, refactoring is dangerous because it puts at risk what is already working - is it? Really? 

I know it works - are you sure? Based on what?

Before anything else, in case you missed it, there was this Mozilla bug that existed for 20+ years and recently got fixed. So, bugs are in the code more often than you think. Just because you deployed in prod and a couple of sprints passed, this does not necessarily guarantee stability or even 100% bug-free software. In fact, over time, I saw many old systems have bugs that were also old. It's way more common than you think.

Now, let's be real about data! Let's say you are introducing 50 features per quarter. Again, zero refactorings. You do have tests, and you run your tests continuously in Jenkins. Here is where you can be easily fooled. Just because you keep running regression tests does not mean you are more or even safer over time. Significantly, if the number of tests does not grow, even if they grow, that does not prove you are safe. 

How could you sleep better at night, then? Make sure that:
  • The number of tests is growing every sprint.
  • Testing diversity keeps increasing, especially fuzzy, chaos, and stress testing.
  • Evaluate the quality of your tests, not only coverage numbers.
  • Make sure your code review is always looking for corner cases and reflecting that in tests.
OK, so features and bugs can create bugs, and even the software in prod could have bugs we have just not activated yet, yet to be discovered. Still not easy to sell refactoring; let's dive into that now.

Seeing the invisible

Refactoring is not one problem; it is multiple problems.IF you see refactoring a monolithc problem you are trouble and will be reduced to all or nothing. Now, if we apply architecture thinking and decouple the refactoring problem into many problems, we have way more opportunities and ways to deal with it. 
We always want to fix the past, but what if we dont need to. Sometimes, it is better just to kill the old system. Other times, you need a modernization project. Modernization projects need to be sold and are not an easy sell, but eventually, they can be landed. Refactorings can sound like anti-economical choices; for instance, consider we won't fix all issues from the past. 

Let's you estimate (as we all know, estimations are always wrong) that we need 5 people over 1 year. That's easily 1M USD. Now the business asks, what will I get from that 1M after 1 year? As an engineer, you say "Clean Code". Not gonna happen, my friend! 

Clearly, there must be a better way to sell modernization projects (and yes, there is). Now let's forget the past and focus on the future, which is something adequately exploited. The issue with low-refactoring cultures is that the "good stable code" is full of:
  • Unnecessary complexity
  • Anti-Patterns
  • Security Vulnerabilities
  • Unoptimized code that will not scale
  • Nasty bugs are hard to fix
  • Bad design decisions that will make new capabilities and features much harder
All of which lead to more time to ship features and bugs.

Now, focus on the future, if you could drive the learned lessons from the past and just stop all the anti-patterns and make better decisions, for sure, this would give us lots of benefits like:
  • The errors of the past will stop being replicated (stop the bleeding).
  • The bad decisions will be replaced with better ones - making everybody else life better.
  • You would start your new oss libraries that are up to date and have fewer vulnerabilities. 
All that can only be done with some level of refactoring, we dont need to pay the past bill all at once, we can ask for a loan and do split payments, everytimes we do a new features we could improve a little, everytime we fix a bug we could improve a little. However, we would behave very differently when we do a new feature. This is how we could start seeing something that is at first glance invisible into something visible and manageable.

Winning left and right

As you might realize, we need a much more advanced strategy to deal with systems at scale effectively. Not only sell and deliver modernization projects but also see invisible opportunities and improve things left and right, past and future. IF you want to learn more about how:
  • Drive effective modernization projects
  • Take better decisions and improve the future
  • Lean about the social phenomena and anti-patterns so you can fix them
  • Be effective at scale with lots of software 
  • Make sense of complexity and understand the root causes of bad decisions
  • Learn techniques and approaches on how to deal with complicated systems like monoliths, distributed monoliths, and microservices.
Give it a shot to my second book: Principles of Software Architecture Modernization Delivering engineering excellence with the art of fixing microservices, monoliths, and distributed monoliths to learn more about this subjects and much more.

Cheers,

Diego Pacheco

Popular posts from this blog

C Unit Testing with Check

Image
When I start on IT, several years ago(+15), C was the first language that I learned. But was not the first language I mastered and deployed in production which was...surprisingly Visual Basic 6.0. However, when I learned C or any language back on that old and arcane time noddy was into testing. However now is impossible to work professionally in any language without tests. C is a low-level language but still kicks ass and has several interesting libs/frameworks. People use to fear C a lot because of lack of syntactical sugar but once you get used to it is not that hard at all. Today I want to show how you can do Unit Testing with C. I will not show how to do Unit Testing with C++ but with C ANSI actually. We will be using Check a unit test lib for C ANSI.
read more »

Having fun with Zig Language

Image
Zig is a general-purpose language created in 2016 by Andrew Kelly .  Zig aims to fix the issues C language has while being pragmatic and simple at the same time. Zig really shines as being simple by not having any hidden control flows, not having hidden memory allocations, no pre-processors, and no macros. Basically, it's one language and one language only. Since pre-processors and macros can get really complex and become their own universe. Sure it's not the Zig Philosophy.  Zig Compiler(Zig cc) also is a C compiler, and the interoperability from C to Zig is very good. Clearly, we can see influence from other languages like Go and Rust; however, Zig is pretty unique and different from Go and Rust.  Zig is fast. Zig is perfect for system programming. I bet we will see a lot of solutions being built in Zig. As a languages aficionado, I like to learn at least one language per year and sometimes more; Zig was my 2021 pick, did not regret it. Even if you dont use all language...
Keep reading

HMAC in Java

Image
In 2018 AWS CTO(Vogels) said : "Security is everyone's job". For the last 2 years, there was so many famous and big data leaks and breaks that made that statement be very true more than ever. Security often could mean worst performance and worst user experience so in order to get it right you really need to think about the designs before jumping into to code and consider performance and user experience has main requirements.  I was thinking about writing about how to do HMAC in Java for a while and recently Redis 6.0.0 come out and to my surprise, there was a refactoring on the password part in order to use HMAC.  Security easily could scare engineers often because it is not something well spread yet but I believe this will change soon. It does not matter if you have to deal with PII Data or not, security is super relevant because everybody is running their workloads at the cloud or with IoT and Edge devices which means more distribution, more code, more points of fail...
read more »