Central and Unique Platform teams

DevOps it's a mindset, Skillset and it's not a Department. It's very common to see in the technology industry Centralized and Unique DevOps/Platform Teams. IMHO we are still not there. Most companies are still catching up on AWS and DevOps Engineering skills such as (Networking, Terraform, Observability, OS/Linux, etc...).  So DevOps centralized teams tend to be "Gate Keepers" for good practices and hygiene and several times cost zealots. That's is fine and has value, dont get me wrong. However several times unique and centrlized teams are just a pure form of a bottleneck. AWS has great infrastructure and tons of purpose-built services, I would argue most of the time you dont need to build anything in front of it. Using AWS directly could be coupling but as long as you use open APIs you should be fine. Several years ago I use to be bottered by coupling solution with ORACLE and with AWS the coupling is much bigger, not sure if any company would ever leave AWS, most of the companies I know never left ORACLE :-). Having said that, In order to have benefits and maximize value, you need to use the services you have available and don't build poor abstractions in front of existing prime solutions. Kubernetes is promising but is not quite there when we consider all the infrastructure components out there. Plus if you do Kubernetes all the things, I'm totally sure bad things will happen. 

Multiple Teams to Rescue

Single Platform / Devops team could easily be a bottleneck, who can we fix that? Multiple teams are the answer. If you have multiple teams, there are several advantages such as:

 * Reduce Communications Blast Radius (Team Topologies) you partition your communications.

 * Purpose Built Solutions and better services via specialized teams 

 * More parallelism but with a hidden orchestration cost 

 * Easier to experiment and move fast 

Comunication is complicated with the Single Centralized DevOps/Platform team since all tickets literally go to the same queue. When you have limited resources and a huge queue you end up doing the following:

  *   Standardize in general solutions (i.g EC2 for all the things). 

  *   You say NO a lot since you dont have the skills neither the time to handle the demand. 

  *   You might be blocking Innovation and being a bottleneck for other teams. 

Purpose Built solutions can be archived but simply having more teams, which will allow you to rapidly experiment with new use cases and debottleneck your system. Purpose-built solutions can be better handle by different and smaller platform teams since you will partition your backlog and also could rely on contractors and have a bigger pool temporarily. 

More Parallelism: As you have more teams, you are capable to execute more projects at the same time. However, this creates a prioritization / Orchestration issue. Since what teams you should have? What happens if one team depends on another team? The solution could lie in other principles such as:

 * Self-Service Model: IF you have services(UIs, Jenkins Jobs, Inner Sourcing model) where anyone can open PRs or use a Jenkins job to create what they need this debottleneck lots of the operations and reduce tickets(manual work). 

 * Isolation and Autonomy: Another thing you can do is to allow teams to perform their own decisions on the fashion of You Build it You Run it (Amazon/Netflix philosophy). Meaning you dont depend on other teams because you take care of all aspects of the solution given a particular domain problem. 

Experimentation it's a nice outcome of all this approach. Most companies don't understand the cloud and don't realize cloud == software and we have a huge engineering pool. The issue is Engineers and sometimes even Engineering managers are sensible with their roadmaps and deadlines and end up being afraid of taking the step forward but it's not as hard as it looks like. Experimentation means different things such as:

  * Experiment embracing the DevOps Model and have engineers coding on Terraform. 

  * Experiment with new delivery models such as More teams, Self-Service, and Isolation.

  * Experiment with new technologies, new components and make things move faster. 

Experimentation has limits

You can experiment and easily introduce solutions to new Services or new applications. However often the heavy lifting is tackling technical debt and migration of old applications. The inertia in old systems at scale is a hard problem to fix. Experimentation would not save you from that Unless you find ways to have backward compatible APIs/data solutions. 

Unfortunately, not all solutions should be Backward Compatible since you could be Backward Compatible with Technical Debts and Design Limitations. Sometimes what you need a big breaking change. The hard part is, depending on how big your solutions are, you might not be able to apply that logic for all your software so prioritization will be needed.

Engineering it's hard and it will always be hard since you have new systems, old systems, technology changes, people coming and going, deadlines, security, and many many other challenges. Team Structure and layout are critical for better or worse evolution.  

Cheers,

Diego Pacheco

Popular posts from this blog

C Unit Testing with Check

Image
When I start on IT, several years ago(+15), C was the first language that I learned. But was not the first language I mastered and deployed in production which was...surprisingly Visual Basic 6.0. However, when I learned C or any language back on that old and arcane time noddy was into testing. However now is impossible to work professionally in any language without tests. C is a low-level language but still kicks ass and has several interesting libs/frameworks. People use to fear C a lot because of lack of syntactical sugar but once you get used to it is not that hard at all. Today I want to show how you can do Unit Testing with C. I will not show how to do Unit Testing with C++ but with C ANSI actually. We will be using Check a unit test lib for C ANSI.
read more »

Having fun with Zig Language

Image
Zig is a general-purpose language created in 2016 by Andrew Kelly .  Zig aims to fix the issues C language has while being pragmatic and simple at the same time. Zig really shines as being simple by not having any hidden control flows, not having hidden memory allocations, no pre-processors, and no macros. Basically, it's one language and one language only. Since pre-processors and macros can get really complex and become their own universe. Sure it's not the Zig Philosophy.  Zig Compiler(Zig cc) also is a C compiler, and the interoperability from C to Zig is very good. Clearly, we can see influence from other languages like Go and Rust; however, Zig is pretty unique and different from Go and Rust.  Zig is fast. Zig is perfect for system programming. I bet we will see a lot of solutions being built in Zig. As a languages aficionado, I like to learn at least one language per year and sometimes more; Zig was my 2021 pick, did not regret it. Even if you dont use all language...
Keep reading

HMAC in Java

Image
In 2018 AWS CTO(Vogels) said : "Security is everyone's job". For the last 2 years, there was so many famous and big data leaks and breaks that made that statement be very true more than ever. Security often could mean worst performance and worst user experience so in order to get it right you really need to think about the designs before jumping into to code and consider performance and user experience has main requirements.  I was thinking about writing about how to do HMAC in Java for a while and recently Redis 6.0.0 come out and to my surprise, there was a refactoring on the password part in order to use HMAC.  Security easily could scare engineers often because it is not something well spread yet but I believe this will change soon. It does not matter if you have to deal with PII Data or not, security is super relevant because everybody is running their workloads at the cloud or with IoT and Edge devices which means more distribution, more code, more points of fail...
read more »