Thoughts on Backward and Forward Compatibility

Compatibility is hard. Depending on the nature of your component/service it could be very unstable by nature. Let's consider an Adapter Pattern for instance or APIs that integrate 2 other APIs, easily they will be breaking on every API change at any side. Even with mundane, regular APIs and Services, compatibility can be very hard to achieve. So, what should we do? Break all the things all the time? Be backward compatible with all versions forever(Which there are a huge price and source of complexity) or maybe there is a middle ground approach we could take? Backward compatibility is only one part of the story the whole compatibility story is much more complex involving Forward compatibility, Isolation, Interfaces, and lots of subtle details. There are the different contexts where software can live so under a microservices in one thing, shared application code is another, shared lib or company framework is completely different stories as well. IMHO at the scale, it does not matter if your consumer is internal or external consumers. Depending on the relationship of the piece of software you might be able to say yeah go there and refactor you code, but maybe not. So this blog post will expand this concept a bit more and show in a video how we can deal with some of these challenges. The last time I wrote about this subject was 6 years ago. I felt was time to talk about it again.

Backward Compatibility

Backward compatibility is easy to understand. Not so easy to maintain. The idea is your software works with the previous version, so the question is what works means? Work means: Functional with no bugs and also no breaking change, so we need to understand what is a breaking change and what is a non-breaking change. For instance:

Breaking Changes
*  Delete a public method (if it was used).
*  Delete a method parameter (if was used)
*  Delete a Class (if was used)
*  Introduce a new required parameter

Breaking changes can be manifested in several other ways like (A) Ordering of the results you return in a List. (B) Data format you were operating let's say you change from DD/MM/YYYY to YYYY/MM/DD. (c) Data format, let's say we change from XML to JSON. Also, it could be in several other ways, it's easy to break something. There are basically a couple of different possible ways we have to deal with backward compatibility:

Go Refactor you go: Sometimes that's the best thing to do, however, no always you can afford this option.  It really depends on your moment and how big or small it is this refactoring.

Conversion tool: Another approach could be to write a program where does the migrations you need from let's say version 1 to version too. Sometimes this could be extremely simple like 1 class name or important where others could be super complicated. We easily could be talking about a simple find and replace with strings to a complex grammar written in ANTLR for instance.

Provide Backward compatibility - Contract Migration:  So Let's say your code runs on version 2 now and what you could do is have the API for version v1 in place and they just redirect and migrate the request from v1 to v2 on the fly ar runtime.

Provide Backward compatibility - New Operations: So basically you would append-only and just add new operations when you need to change. This would be easy at the first look but as time pass your refactoring blast radius would be big.

Provide Backward compatibility - Multiple Deploys: You could have 2 different versions running side by side v1 and v2. IF we are talking about a service that might be doable, considering the resources duplications. However, if we are talking about a shard lib, that would still be possible but much harder. Due to JAR/Classpath conflicts.

Forward Compatibility

Now. What is forward compatibility? Well, this means we will be compatible with future changes. This sounds hard and actually is. However, there are things we can do to make forward compatibility more doable like:

Forward Compatibility
* Extension points / Interfaces: Will allow us to extend behavior without breaking the contract.
* Hide Implementation details: Configuration is both a blessing and a curse at the same time. You might want to expose as little as possible. For sure there is a tradeoff here. Favoring smart default would always be better for the design and usage perspective but also for future changes.
* Cohesion: As you do not build a monolith, future changes often could be expressed as different components and that is fine as long as that is coherent otherwise you are really just leaking the design and making it worst.

There are other things you could to increase forward compatibility. However, the big thing is, it's hard to get all use cases, corner cases, and business understanding in the beginning. Looking to the past and say that it was a wrong decision is easy, make decisions today that you won't regret tomorrow is much harder. Principles are important for that very reason. Otherwise, we might be caught on 2 very bad extremes: Analysis paralysis or poor design delivered.

So should we just ignore forward compatibility? No, not at all. Just keep in mind if you dont have proper isolation protection i.e well-defined interfaces or boundary contexts for a microservice it might be super hard to maintain it. However, having the right boundary context and protecting the internal datastores in a properly isolated microservices might mean that:

Internally: You don't need backward and forward compatibility at all - since microservice provides you greatly isolation and therefore freedom. Talking about your internal modules and database.

Externally: Meaning your public contract(i.g rest API) you will really need to be backward compatible and forward as much as possible.  Talking about the Input/Output objects.

However, if you are in a monolith or shared-application world them you really need backward/forward compatibility internally and external because the tables are really your APIs. So depending on where you are the backward/compatibility trade-off might have a completely different weight, priority, and impact.

What is an Interface?

This is really a key concept. So several languages have Interfaces or Traits. However, thats are syntactical constructs. What people do not realize is even if you dont have an interface/trait you might have a public interface. Public interfaces can manifest themselves in almost all forms of software like and not limited by:
* Classes: Public methods that are being used outside of your module or service.
* Packages: Package code(scala) which might be invoked as you use a call.
* Modules Pretty much all languages have some abstraction for that.
* XML: Consider configuration, Mapping, Logging, Security, auditing, you name it.
* Json: Consider configuration, Mapping, Logging, Security, auditing, you name it.
* Yaml: Consider configuration, Mapping, Logging, Security, auditing, you name it. I.g Kubernetes.
* Tables: Yeah, Tables will be your interfaces if you have 2 apps accessing the same DB.
* IDs:  Yeah, Let's say you have 2 apps accessing redis/Memcached. Them IDs will be your public interface.

Public Interfaces == Contracts. When we consider SOA / Microservice contract we often just consider our beautiful contract project or our REST endpoints, however, if users config things in Tables or XML thats part of the contract since it is public. The public does not mean(0.0.0.0) it means not abstracted by the software. So you really need to pay attention to your interfaces. Eclipse had worked for decades on the OSGi model, Oracle followed a different path, but here we are in 2020 and we still did not fix this problem.

Video

So I record this video in order to show some of the concepts I was exploring in this blog post.



Cheers,
Diego Pacheco