Diego Pacheco Tech blog

The cloud is not the end, but the beginning. Starting a quick journey to the cloud brings benefits to your business. However, to get more benefits and reduce costs it is necessary to adopt a cloud-native mindset, which means going beyond, using the maximum value of cloud computing. Kubernetes , a system designed by Google and maintained by the Cloud Native Computing Foundation , for example, is much more than a container orchestration solution. Many experts consider Kubernetes the new Linux since much evolution is to happen in this system. Nowadays all organizations use Linux, but not all organizations use Kubernetes. Possibly we will reach the point where all companies will start using kubernetes, just as they use Linux today. You must be imagining technical reasons for using Kubernetes. They exist and there are many, however, there are also very relevant business reasons regarding the use of this system. Keeping Innovation Flowing It is increasingly important to innovate wi...

The work environment can tell a lot about a company culture and values.  Companies care about productivity but in reality, several times work environments are optimized for noise, distraction, and anti-productivity.  I used to love open-spaces, currently, I'm not in favor of open-scapes any more. People often argue that open spaces improve collaboration however people are always using headsets(noise-cancelation for sure) plus moving to "meeting rooms" in order to get some work done. When I need get support important work done I work from home and I know so many people that do the same. There is a huge difference if you are a pure manager/coach or an Engineer/Architect because noise makes different effects on you. In general, pretty much everybody agrees too much noise is a problem but in practice keep the noise down is almost impossible in open space structure. Open spaces maximize the value from renting physical structure but they don't maximize value fr...

Cassandra is a kick-ass NoSQL database. Battle tested by Netflix, Apple, Uber and so many great cloud scale companies. If you want to run Cassandra in production you need to buy DataStax Enterprise or engineer your own solution - since Cassandra community is not enough. My company loves open source and my customers love open as well so we decided to build our own Cassandra orchestrator for a lack of better name we called it CM(Cassandra Manager).  CM runs on AWS(Ec2) and managed single and multi-region clusters for Cassandra. CM has java interfaces so even today it just runs on AWS it could easily be ported to in containers or other fabric runtimes, which may happen on the long-term future. So it might sound crazy when you think about to build engineering around Cassandra but is actually not several companies in Silicon Valey like Netflix and outside of the valley do similar things or Databases being NoSQL or Relational. Automation is must if you want to scale, a...

Go is a great language for writing tooling. It's very easy to produce a self-contained binary in go which makes very easy to distributed tools written in go. Today I want to show how easy is to write a simple REPL in Go. REPLs are quite popular in Functional Programming community like Scala, Clojure, and Haskell. Today even Java has a REPL :-). Cassandra has interesting REPL called CQLSH  which allow local and remote connections. Currently, I'm thinking to write REPLs for some things I´m working with. REPLs are great for productivity and context. No to mention the could reduce and simplify discovery and troubleshooting. You might be thinking Gosh to write a REPL is tough we need to have parsers, print things nicely with colors, use tab completion, remember all previous commands. and one and on... I'm a full fledge REPL might require more work however a basic and simple version is something we can do in less them 5 minutes so I won't be doing any complicated parser or...

Today I want to share some experiences and findings in regards to Code Review. Code Review is a practice that everybody does it today. I was doing code review way before Github existed. I will share experiences before Github and after Github. Difference companies and different teams have different realities and preferences. I don't think there is universal or wrong but this is my experiences. I never was and still not a big fan of tools for code review. This subject is very related to Technical Debt. I think there are 2 kinds of technical debt the one you pay often and the ones you need to schedule because they are big and might need lots of time to fix it. However, if you just managed tech debts often means you never pay it. Also, you should be paying even if no one outside of the team is asking for it. When you are doing a PR is a good time to pay some debts related to that PR, also long as is not too much. I found Github view for code review very limiting. ...

Cats is a library for Functional Programming in Scala. The name come from the Inspiration by the Category Theory  so Cats is short for Category Theory. Cats are about types, Type System and having abstractions to make it easy to work with them. If you want to go to the next level of function programming Cats is the way to go, however, I need to say that it's hard, get the ideas to take time, understand the terms, principles, and applications takes times as well. At the end of the day, this is about style so there is no right or wrong. Recently I made a post about Monads so it might be useful for you to read it if you did not read it yet. This post won't be monad heavy but reading about monads will make it easy to understand some things here. It's fine if you don't understand everything, I'm 100% sure I don't understand everything and there are lots of ideas and concepts that I still learning. So this takes time. I don't think this is something...

What is a monad ? It's easy to read lots of sites and still don't get the idea. Many people already used monads but few really understand them truly. Monads are a kind of a Myth in Scala for lots of developers. Itś possible to code without creating your own monads? Yes for sure it is. It's very likely you already create abstractions which were similar to monads but with different names. Here are some samples of Monads in Scala: Try , Either and Option . We have some of this monads in Java 8 as well like Optional . Monads come from the Category Theory which is a branch from Math. Monads are very strong concepts in Haskell Language for instance. IMHO one of the reasons why use Monads rather than your uncle abstraction is that they are universal and in theory, they are easier to understand, in practice if you don't study math, Category Theory and Functional Programming principles actually monads will sound much harder than you user uncle abstractions. So, in other word...

Stress Test is something that everyone should be doing if you care about performance. I run software at AWS so is easy to spin up more boxes but is also more expensive and you might be in a trap because you might have ineffective software with Memory Leaks, Hidden Bugs, and untuned servers which will all show up with scale or stress tests - whatever happens first. Unfortunately, this still not quite popular among most of the microservices developers. Create and run stress tests is something hard and tedious because it involves lots of moving parts. A single microservice could call several other microservices some of the downstream dependency graphs could get complex easily, this is just one of the many challenges involved with stress tests. Today I want to share some experiences in building and running a stress test and chaos platform. Do you know how many concurrent users or requests per second(RPS) your microservices can handle? Do you know what it will be the COST to scal...

Go lang is a very interesting language. Github already is the central hub for the open source on the Internet. Companys have lots of repositories and it's very hard to catch up nowadays. So I build a simple silly tool in Go in order to fetch all repositories a company might have and they compare with the previous fetch and see if there are new repositories.  Thanks to GO elegance I was able to do that with less than 200 lines of code. In order to do what we need to do, we will use some basic functionality in Go such as IO, Http Calls, JSON support for Structs, Slices, Maps, Error Handling(Which is the sucking part) and a bit of logic. This program can fail since I'm not using a proper TOKEN for github so you will HIT the github Throttling limits pretty quickly if you keep running the program. You can create a proper DEV account and fix the code to use your token or you can schedule the program to run in different times in your crontab. This is not production ready, bu...

Go is a very simple and powerful language. I like Go.Go performance is amazing. I don't like that Go don't have proper Generics and Exceptions but besides that is all great. Go is great for DevOps because go generate a simple binary that is executable and this is really awesome. Today I want to show to build a simple counter service using Go, Gorilla/Mux and Docker. Gorilla/Mux is a simple HTTP Router for Go. Using native JSON support from Go + Mux you can build some simple and yet powerful services. We will use sync/atomic in order to make sure if we have a code that is threadsafe.  You will need to have Go and Docker installed in your machine.  We will use Go language version 1.10 and latest Gorilla/Mux. Let's get started! The Dockerfile Here we are using Go 1.10 docker image as the image. We are using to get in order to install gorilla/mux. Now we can build the docker image. We just need to run the command: $  docker run diegopacheco/go-cou...

Ansible is a great provisioning tool. However, it can be painful to get some ansible scripts right. Especially if you need some stuff with bash and Ansible. Often baking time in AWS can be pretty high. So It's better you can run ansible locally. However, running ansible local could mess up with your OS. So the best thing is run ansible in Docker. Since the docker container will be ephemeral, once you finish running the container all changes will be lost. You also will benefit from running locally and being able to figure it out quickly whats wrong.  So today I want to share about some simple project I create in order to help to do that. This is called Ansible-Docker this is an ansible sandbox using Amazon Linux.

Terraform is a good tool for infrastructure provisioning.  However to test terraform it could be pretty difficult. So you will create some terraform scripts and upload to the cloud a run some slow Jenkins job? and if your syntax is wrong? Well, this process can be very painful. So I want to share some simple sandbox I built in order to speed up terraform + aws development in your local machine. I might be wondering how is that possible─? Well, my secret sauce is Localstack . So we are limited to all endpoints that localstack mocks. As Localsttack adds more endpoints we benefit from that. The main idea behind this simple project is to show how easy is to docker-ize somDevOpsps tools and make engineering easy.  Currently is very often to spend 40mim or more doing baking and that's is wrong. So that's kind of mainframe era so the idea is to save time and run things local - as much as possible. Docker helps a lot with that. I run software in production using...

Ethereum is the most interesting blockchain platform right now. I have lots of interest in IOTA and HashGraph  because they claim to solve scalability problems that ethereum and pretty much most of platforms/token have, however they don't have as big adoption and rich ecosystem as ethereum has. Blockchain got popular in 2009 thanks to Bitcoin paper but the market and development industry still pretty new. Ethereum allows us to create Smart Contracts which are far more powerful then what you got when you work with bitcoins. Most of the ideas in blockchain and ethereum platform are not new, they are pretty old concepts like Cryptography , Immutability , Distributed Computing , and Consensus Algorithms . Clearly, the Financial sector was the first application field for blockchain, however, we are not limited to financial applications and we could use for many other industries like HealthCare, Agriculture, Energy, Real estate, Legal, Insurance, Education, Gov an...

Several years ago it was impossible to think about software development without having at least a business specification(usually in a word document) and a bunch of UML diagrams like package, class and sequence diagram. That was a pre-agile era where process and formality ruled the IT software development industry(yes folks, yes I'm old). Back on that time software engineering was pretty much conceived as "process".  Now we live in a more civilized era, however, back on that time, there were some interesting and valuable things(tools, techniques, principles and even practices).  Like the English expression, you need throw bath water but keeping the baby. I was wondering why these phenomena happened and still happen today. First of all, we are SOCIAL animals we tend to have some kind of "fashion" behavior where we follow the "flow". Needless to say that UML and Software internal design is not quite popular today with the new kids. I don't miss the...

It's normal to talk about failure when we talk about cloud-native architectures. Pretty much anything that runs on the cloud(data center) means more distribution(networking). Distributed Systems tend to fail all time. Chaos Engineering is great, however, it's for the infrastructure, therefore focused on the big picture. When I say big picture I mean outside of a microservice. Reliability is not just for the outside boundary or infrastructure but also for Inside a service or inside the system. Reliability Inside the system might have many names such as(very popular in Brazil during 2k years) Defensive Programming .  There are lots of synergy between reliability , defensive programming, anti-fragility and Efficient Internal System design. Efficient design it's not only about making your system efficient in sense of economics(code, readability, reduce maintained cost) but also reliability. Today I want to share some internal system design way of think in order to ach...

When I start on IT, several years ago(+15), C was the first language that I learned. But was not the first language I mastered and deployed in production which was...surprisingly Visual Basic 6.0. However, when I learned C or any language back on that old and arcane time noddy was into testing. However now is impossible to work professionally in any language without tests. C is a low-level language but still kicks ass and has several interesting libs/frameworks. People use to fear C a lot because of lack of syntactical sugar but once you get used to it is not that hard at all. Today I want to show how you can do Unit Testing with C. I will not show how to do Unit Testing with C++ but with C ANSI actually. We will be using Check a unit test lib for C ANSI.

Some time ago I was sharing my experiences with a Dynomite Remediation Process I wrote. Iḿ using remediation systems for a while and IMHO the add lots of value since they automate manual Cloud Operation work and save time for people. Currently, I refactored my Remediation code and now the same code can support Dynomite / Dynomite Manager but also Apache Cassandra. There are very similar concepts between Dynomite and Cassandra Remediation such as discovering AWS EC2 Ips to be remediated, AWs Resourcing(Creating SGs, Deleting LCs, Updating ASGs), Health checking(Is the node up and running? Could I remediate right now or you are in the middle of a backup or just booting up?). Once I identified that core concept I was able to create a high level and generic design for Cassandra and Dynomite. This is great for cases of Patch / Fix(apply a new AMI) or Scale Up(Increase the memory or CPU for instance). Although that the main use cases and most important ones there were some c...

Cloud Computing is the default today. I do believe the future is containers and multi-cloud solutions. However today I work a lot with AWS . There are specific endpoints such as S3, AutoScaling , Route53 and other that are among the ones I use more in my day to day work. AWS API is easy to use however not no easy to test. Distributed systems tend to be hard to test. Having quick feedback is very important for engineers. There is some kind of tasks that need to interact with AWS APIs like S3 for backups for instance. However, if you need to wait to deploy in AWS to test it because is basically impossible to test it locally then we have a problem. We need to be able to do end-2-end testing however while you are coding or doing some troubleshooting is important to do things faster. There are 2 specific projects that can help us with this task. TestContainers and LocalStack . Today I will show to use LocalStack and TestContainers together with JUnit in order to do unit tests m...

DevOps Engineering is often "perceived" as a operations matter. In fact, it's true but is not 100% JUST ops. There are some blurred lines between engineering and operations. When we take a look at the DevOps ecosystem we can highlight some great tools like Ansible, Terraform, Jenkins, Packer and so many others. Provisioning, Release, Observability, and SRE are different aspects of a broader DevOps adoption. The higher benefit comes when we start applying more engineering to DevOps. It's very important to have good tooling around. When we think about tooling engineering makes a lot of sense, however, even outside of the tooling space there are areas where we definitely need an engineering injection like Cloud Production Operation Automation. Releasing is the easy part, things get complicated when you automate the operational aspects of your systems. It sounds so meta or Inception(If you watch the movie ) but Systems managing Systems looks the way to go for me. This ...

Working with the cloud is great but it's not easy. The Cloud allows us spins machines pretty easily, however, update these machines are not that an easy task, especially if we talking about databases. Today I want to share some experience with remediation process I've built for Dynomite in my current project.  I'm using NetflixOSS Dynomite in production(AWS) for more than 2 years now both as cache but also as Source of Truth. I need to say that is rock solid and just works. However often we need to increase some machine capacity, i.e: Add more memory or increase the disk size. There are times where we need to apply patches on the OS - I use Amazon Linux on Production on EC2 and recently there was the Meltdown and Spectre situation. Other times there are telemetry or even provisioning and configuration bugs that need to be fixed.  No matter the reason you will always need, time to time, change something on the underlying AMI or in the Java or C application which is ru...

Today I want to share some experiences I had in the last 2 years with AWS Lambda as a Serverless solution for Microservices Remediation. This was not a single effort from me only but a team effort. The lessons learned I want to share with you today was not only in sense of Serverless but also in sense of DevOps Engineering. DevOps Engineering is the norm today. I really can't see a world without it. Code it is the lingua franca of everything around IT. Currently, there are more and more abstractions and solutions rising for DevOps Engineering, chances are we will need code fewer things as the time pass. However, as we push the boundaries of innovation we face new problems and new problems will always require better solutions. DevOps Engineering is great however it's not a FREE LUNCH at all, like microservices, which are great too, there are lots of COST that are introduced. There is a requirement do structure and lay down teams in a different way and also there is a need ...

Kanban is a Lean tool. It's focused on the psychological part of the work. In a nutshell, you manage the system via constraints and you don't manage people. I used kanban a lot in the last 8 years. When I started I was a Software Architect | Coach in not-so-traditional SOA projects.  Today I work a lot with DevOps Engineering and Cloud Computing and I don't work with 2 weeks sprints but with quarters which are 3-months length buffers of work. You might think as a quarter as a super big sprint. There are several different kinds of classes of work like (A) Support for Developers, (B) Tuning, (C) Stress Testing, (D) NoSQL Development, (E) Telemetry and Observability, (F) Distributed Tracking and so on and on. I'm working on CORE team with other architect and DevOps Engineers. For a long time, I was using kanban metrics and sheets pretty religiously. Because I was working with Big Teams(up to 33 people) and multiple teams who had to deliver on time. As more close y...

C language is old(since 1972 to be precise) but still kicks ass. Some folks consider C too low level. IMHO is not that low level, actually, there are some abstractions and syntactical sugars I miss in C but IMHO is something you can leave without since you are befiniting from better performance. Many developers care too much about syntax today and not many care about performance. It really depends on what you are doing and what kind of company you are. There are languages that are more productive than C for sure, however, once you get how to do things properly is not bad at all actually is quite good.  Most of the big languages like Java, C#, python, ruby are actually based on C. C often is a language the most of people fear it. Maybe because we got used to much with languages that do a lot of things for us. IMHO is not that hard, it's just a matter of taste, after some time I learn to enjoy working with C professionally and today I like it a lot. But this really depends...

Lately, I'm working with custom redis-modules and I need to say: redis-modules kick ass. I started working with Cthulu and coding modules with javascript which was nice and very productive but given some of my current use cases I had to go to C. I recommend you take a look at my previous posts here:   -  Building Redis-Modules with JavaScript using Cthulhu   -  Dispatching Custom Commands with Lettuce and Redis-Modules Today I want to you how to do proper debug with Redis modules written in C language. I will be using eclipse Oxygen CPP in order to debug redis module and redis 4. Here there is a video which I recorded with a simple live demo about how to debug a redis-module project and redis. The source code is available on my GitHub . I hope this is useful for you.

Last year I started to blog about something I'm calling "Stability Mindset". I'm still working on the ideas but I think I had enough time to time and experiences to evolve the concept to some basic guiding principles. You might be wondering if I'm just trying to do some kind of ugly re-branding of Reliability like Software Reliability Engineering(SRE) but I don't think is the case - I see is different things or at least a specific subset of SRE. If you are interested in a broader spectrum you can check out google insights on the matter. Google has this amazing book you can check it out about SRE. I thought a lot if this was just SRE for mere mortals but I don't think it is. I really think is a different kind of problem or at least different subset for sure. Today I'm writing this post because I want to capture some of my recent findings and also because I did a retrospective exercise with my team today I promise I you share some of my thoughts...