DevOps it's about Design

571px-Popular_Science_Dec_1918_p23_-_Ship_Emergency_Steam_Cutoff_Valves.JPG
DevOps it's about Design!

As Steve Jobs once said: "Design is not how it looks but how it works". DevOps implies operation work through automation. People who do DevOps could come from 2 different kinds of backgrounds, you can come from Dev or from Ops. A good Developer cares about and really knows how to design applications. We do have several principles to design software. One of the basic and most important principles is Abstraction.




Abstractions
Abstraction means doing more with less and hiding complexity. It's easy to find automation solution with does not provide any abstraction at all. There is an easy way to get this smell which is the number of parameters you get from a Jenkins job for instance. More parameters means throwing complexity in your face it does not imply more flexibility.


Having more options also can create more trouble for you because. Let's say you are using AWS and you receive the AMI_ID as parameter. Sounds like a good idea but then you realize you have microservices and need to do this for all your 1000 microservices and them there is a new version. Well you need to update all your jobs and make sure to re-deploy each one.


Code generation is not evil but it's very easy to become evil. Better than generate parameters is abstract parameters and as less parameters you ask the developer / engineering to pass the better. So this imply we should have static configs? Not really. We should have dynamic configs and there are several ways to do it so, for instance you could be using: etcd, redis or any other key/value database. It's better to have replicated multi-region, multi-datacenter solution like etcd or even Netflix/Dynomite is you have such requirement.


Telemetry of the Telemetry
Doing so you still have do dynamic configuration but you won't expose complexity to the cloud deployment jobs. All automation work can fail. So you need to make sure you have proper error handler and telemetry in place. Few days ago my car turned on a LED for ABS brake system. After few days I went to the car dealer in order to perform a review. I was told that the LED was for a sensor wih tells you how bad the break pads are. In my case that sensor stop working so we got an LED  notification. This does not means I don't have break or that my brake system crash it only tells me that the brake sensor stop working. I found this very interesting because in the end of the day this is the Telemetry of the Telemetry. So I have a monitor that monitors the monitor. It's kind of meta and inception-ish but IS your car have such system why your complex cloud solution should not have this?
The correlation between design and telemetry is this. You need to think about error cases while you do design. You can't consider only happy scenarios. Failure happens on DevOps Engineering and you need to code solution that address this concerns.
Cloud Deployments is hard
Microservices deploys are kind of easy however when you consider a cloud environment it gets more complicated. Since you might have specialized data stores for each microservice. Like one microservice can use Cassandra and other might use Elasticsearch. Provisioning data stores are often way harder them just the application(microservice). Because you will might need to feed / migrate data and schemas.
Data stores also have their own concerns like backups, restores, warm-ups, cluster management. Automation for databases in cloud it's more complex. In the end of the day it's all about requirements and the use cases you might have. Having more availability makes everything more complex and requires more work.


This kind of automation is very hard to with bash or python scripts for instance. People tend to you solutions in Go or Java for such tasks. Since you have a procedural nature in this kind of work. This is also design.


Trade-Offs come to play
Today we have more than one way to do things. For instance you can use Kubernetes. Which will give you most of this things for free but them you will be coupled to Kubernetes. You also can create your engineering solution and use open source tools which will require more automation work but you will definitely have more flexibility and you will be able to run microservices without docker. Having the possibility to run kubernetes everywhere is sexy and very attractive to me since you could run multi-cloud solution. In order to run your code in other cloud and make cloud deployments easier you also could use Netflix/Spinnaker. Spinnaker it's a multi-cloud orchestration solution.


This is design as well. What you will use it and why? What requirements do you have and what solution address this concerns. Spinnaker can call as many Jenkins jobs as you have so you can see it as one thing which makes your life better but this does not fix your job complexity problem in Jenkins for instance.
Design is a critical discipline in DevOps. It's important in all levels from a python script to a spinnaker pipeline. Having more parameters does not necessary gives you more flexibility. Hiding details is always a good idea. Using CONVENTIONS and Dynamic configuration it's also the way to go.


Cheers,
Diego Pacheco

Popular posts from this blog

Kafka Streams with Java 15

Image
Kafka Streams is a Streaming library similar to Spark and Flink which works with Apache Kafka. Kafka stream can be useful to process historical and near-real-time big data workloads but also for non-realtime analytical computations at scale for the online world as well. Kafka-Streams is elastic, highly scalable, fault-tolerant, and fully integrated with Kafka. You can use Kafka Stream with Java, Scala, Kotlin JVM applications and also have exactly-once processing semantics. Kafka-Streams is being used by the New York Times, Pinterest, Line, Trivago, Zalando, and many other companies. Today I want to share a video of Kafka-Streams running with Kafka 2.6 and Java JDK 15. So Let's get started. 
read more »

Rust and Java Interoperability

Image
Rust does not stop for one second to amaze me. It looks like Rust was designed to talk to every other language so easily. No kidding, it's clear way Web Assembly and rust have lots in common. Java is never an easy lang to interop with. Today I will show how easily we can interop Java & Rust. In order to do so, we will need to use the Java and Rust compilers. For Java, we will not require any other lib than the standard JDK. For Rust, we will need to create a lib project in cargo and we will need to use the crate JNI. I love this feature because java does not allow low-level code on the main language. So if we need to do something low level, efficiently Rust is my first choice, not only for safety reasons but because is really fast.  Let's get started.
read more »

HMAC in Java

Image
In 2018 AWS CTO(Vogels) said : "Security is everyone's job". For the last 2 years, there was so many famous and big data leaks and breaks that made that statement be very true more than ever. Security often could mean worst performance and worst user experience so in order to get it right you really need to think about the designs before jumping into to code and consider performance and user experience has main requirements.  I was thinking about writing about how to do HMAC in Java for a while and recently Redis 6.0.0 come out and to my surprise, there was a refactoring on the password part in order to use HMAC.  Security easily could scare engineers often because it is not something well spread yet but I believe this will change soon. It does not matter if you have to deal with PII Data or not, security is super relevant because everybody is running their workloads at the cloud or with IoT and Edge devices which means more distribution, more code, more points of fail
read more »