Generic and Script Remediation

Some time ago I was sharing my experiences with a Dynomite Remediation Process I wrote. Iḿ using remediation systems for a while and IMHO the add lots of value since they automate manual Cloud Operation work and save time for people. Currently, I refactored my Remediation code and now the same code can support Dynomite / Dynomite Manager but also Apache Cassandra. There are very similar concepts between Dynomite and Cassandra Remediation such as discovering AWS EC2 Ips to be remediated, AWs Resourcing(Creating SGs, Deleting LCs, Updating ASGs), Health checking(Is the node up and running? Could I remediate right now or you are in the middle of a backup or just booting up?). Once I identified that core concept I was able to create a high level and generic design for Cassandra and Dynomite. This is great for cases of Patch / Fix(apply a new AMI) or Scale Up(Increase the memory or CPU for instance). Although that the main use cases and most important ones there were some corner cases and always it will have URGENT fixes that don't and can't wait for a new AMI. Remediation allows AMI rollback and therefore is great for immutable infrastructure. I respect and agree with Immutable Infrastructure principle however there are use cases where we need to do things differently. Don't get wrong, if you don't craft your use case with care is easy to reach a scenario where you end up harassing immutable infrastructure and therefore creating lots of side effects. That's not my goal here. However, the Hell is full of good intentions some a tool like this need to be used with lots of care otherwise bad things will happen. Right now you must be wondering why do Script remediations? Why don't use something like Terraform or Ansible and that's it? Ansible is a great tool and Terraform too, however, there are several kinds of problems and solutions. My problem is quite unique(considering my customer, my current problems, and current stack). I do believe most of the things I'm sharing here can be used by a broader audience, however, IF you don't have this problems or value DevOps Automation somethings here might sound too much.

Terraform, why not? 

First of all, Terraform is awesome. However, I have a very unique kind of problem here. I use NetflixOSS Dynomite and Dynomite Manager.Dynomite is written in C and Dynomite Manager in Java. DM(Dynomite manager) does all kind of automation work like security groups, backups and restores to S3 using Java. So I don't have an Ansible or Terraform Script to change, instead, I do have Java code which is better IMHO. If you are curious about this DevOps style of work please read this. Secondly, the kind of problem we have here(Data Layer - not microservices) required very dynamic work and this kind of work could be done by Terraform however it would require write a plugin in Go or generate Terraform code. Since My Stack(based on NetflixOSS Stack) is Java and I work on a Java shop it does make sense to keep it in Java.

The Issue with Ansible

Ansible is cool. However, it falls on the same issues like Terraform. I would need to write a plugin in Python or generate ansible code which would not be ideal. So Again being in Java would be better, since DM is in Java already and I do deliver(A guy with Engineering background) DevOps with Code(For data problems) is soo much better. In regards to ansible I have a second issue - In my current project, we don't use Ansible Inventory. We just use Ansible for provisioning, so this also makes discovery hard. Even If I had inventory it would have others problems like Microservices developers often don't like do anything related to DevOps work so they might not have even access to AWS console so asking they to use Ansible might be not ideal. IMHO developers should be responsible to DevOps as well however I think core and platform teams need to provide better tools to make that work easy.

The case for Script Remediation

The Script generation is generic. It's possible to apply bash scripts in Cassandra and Dynomite clusters using a generic Jenkins job. The Script remediation will receive a bash script which will be encoded using base64 in order to be passed as java properties from Jenkins to gradle and also a timeout. Timeout is required because the generic script remediation will connect on each cluster node and apply the script. Some scripts might take more time them other so that's why we need to pass the timeout.


The Generic Script Remediation is just a simple java code that connects on each node and applies the script. There is some information that is captured to generate a final report such as host, AZ, time to execute the script, it was successful or if there were errors. That'st it.

This code just works fine for Dynomite and Cassandra. There are some use cases for this approach in my project - like:

  • Data Mugling: Sometimes developers need to clean up data(Stress test or Investigation).
  • Ops Emergency: Security Patch, Telemetry or quick rollbacks.
Most of times Patch / Fix remediation will be used and therefore force immutable infrastructure but for that 2 use cases I described Script remediation is better and also faster. Regular(Patch/Fix) remediation takes lots of time(~16 minutes for 3 node cluster). Script remediation can be done in less than a minute. 




Since you can pass a bash script that could be a backdoor to hurt immutable infrastructure or even destroy the cluster, loose data or create availability issues or even downtimes. So this need to be used with care and lots of caution. Using with wisdom this is a great tool to save people time and reduce manual work and avoid repetitive tasks such as cleaning up data before the stress test.

Whats Next

Currently, I'm wondering If I should have a 3rd kind of remediation based on security groups because time to time you need to open ports. Since DM generate all security groups(there is no script to change it) and create java code for that is too much. However I'm not sure if that would be a real abstraction and real value or would be just another interface since I don't see developers opening ports, this is a task for a cloud ops team and they don't have issues with ansible nor terraform. So they could go to aws console or run ansible from a bastion node. Thinking this way the 3rd kind of remediation based on SG might be just another UI(Jenkins) and that where I'm. Also could not find a Java API for Terraform. I also don't want to create a new DSL  for remdiation would be better leverage Terraform syntax but them this would require to colde a plugin or generate Terraform code. So for now I will wait and see if this business case become more important, right now that did not happen a lot of times and we need be lean, automation is great but there is a cost associated with and sometyhimes is better leverage new features.

cheers,
Diego Pacheco

Popular posts from this blog

C Unit Testing with Check

Image
When I start on IT, several years ago(+15), C was the first language that I learned. But was not the first language I mastered and deployed in production which was...surprisingly Visual Basic 6.0. However, when I learned C or any language back on that old and arcane time noddy was into testing. However now is impossible to work professionally in any language without tests. C is a low-level language but still kicks ass and has several interesting libs/frameworks. People use to fear C a lot because of lack of syntactical sugar but once you get used to it is not that hard at all. Today I want to show how you can do Unit Testing with C. I will not show how to do Unit Testing with C++ but with C ANSI actually. We will be using Check a unit test lib for C ANSI.
read more »

Having fun with Zig Language

Image
Zig is a general-purpose language created in 2016 by Andrew Kelly .  Zig aims to fix the issues C language has while being pragmatic and simple at the same time. Zig really shines as being simple by not having any hidden control flows, not having hidden memory allocations, no pre-processors, and no macros. Basically, it's one language and one language only. Since pre-processors and macros can get really complex and become their own universe. Sure it's not the Zig Philosophy.  Zig Compiler(Zig cc) also is a C compiler, and the interoperability from C to Zig is very good. Clearly, we can see influence from other languages like Go and Rust; however, Zig is pretty unique and different from Go and Rust.  Zig is fast. Zig is perfect for system programming. I bet we will see a lot of solutions being built in Zig. As a languages aficionado, I like to learn at least one language per year and sometimes more; Zig was my 2021 pick, did not regret it. Even if you dont use all language...
Keep reading

HMAC in Java

Image
In 2018 AWS CTO(Vogels) said : "Security is everyone's job". For the last 2 years, there was so many famous and big data leaks and breaks that made that statement be very true more than ever. Security often could mean worst performance and worst user experience so in order to get it right you really need to think about the designs before jumping into to code and consider performance and user experience has main requirements.  I was thinking about writing about how to do HMAC in Java for a while and recently Redis 6.0.0 come out and to my surprise, there was a refactoring on the password part in order to use HMAC.  Security easily could scare engineers often because it is not something well spread yet but I believe this will change soon. It does not matter if you have to deal with PII Data or not, security is super relevant because everybody is running their workloads at the cloud or with IoT and Edge devices which means more distribution, more code, more points of fail...
read more »