Showing posts from 2018

github-fecther: Checking new Repos from Github with Go

Go lang is a very interesting language. Github already is the central hub for the open source on the Internet. Companys have lots of repositories and it's very hard to catch up nowadays. So I build a simple silly tool in Go in order to fetch all repositories a company might have and they compare with the previous fetch and see if there are new repositories.  Thanks to GO elegance I was able to do that with less than 200 lines of code. In order to do what we need to do, we will use some basic functionality in Go such as IO, Http Calls, JSON support for Structs, Slices, Maps, Error Handling(Which is the sucking part) and a bit of logic. This program can fail since I'm not using a proper TOKEN for github so you will HIT the github Throttling limits pretty quickly if you keep running the program. You can create a proper DEV account and fix the code to use your token or you can schedule the program to run in different times in your crontab. This is not production ready, but it'…

Running Gorilla/Mux in Docker

Go is a very simple and powerful language. I like Go.Go performance is amazing. I don't like that Go don't have proper Generics and Exceptions but besides that is all great. Go is great for DevOps because go generate a simple binary that is executable and this is really awesome.
Today I want to show to build a simple counter service using Go, Gorilla/Mux and Docker. Gorilla/Mux is a simple HTTP Router for Go. Using native JSON support from Go + Mux you can build some simple and yet powerful services.
We will use sync/atomic in order to make sure if we have a code that is threadsafe.  You will need to have Go and Docker installed in your machine.  We will use Go language version 1.10 and latest Gorilla/Mux. Let's get started!

The Dockerfile

Here we are using Go 1.10 docker image as the image. We are using to get in order to install gorilla/mux. Now we can build the docker image. We just need to run the command: $ docker run diegopacheco/go-counter.

Go Code

Go code is so simpl…

Running Ansible with Docker

Ansible is a great provisioning tool. However, it can be painful to get some ansible scripts right. Especially if you need some stuff with bash and Ansible. Often baking time in AWS can be pretty high. So It's better you can run ansible locally. However, running ansible local could mess up with your OS. So the best thing is run ansible in Docker. Since the docker container will be ephemeral, once you finish running the container all changes will be lost. You also will benefit from running locally and being able to figure it out quickly whats wrong.  So today I want to share about some simple project I create in order to help to do that. This is called Ansible-Docker this is an ansible sandbox using Amazon Linux.

Mocking Terraform AWS using Docker

Terraform is a good tool for infrastructure provisioning.  However to test terraform it could be pretty difficult. So you will create some terraform scripts and upload to the cloud a run some slow Jenkins job? and if your syntax is wrong? Well, this process can be very painful. So I want to share some simple sandbox I built in order to speed up terraform + aws development in your local machine. I might be wondering how is that possible─? Well, my secret sauce is Localstack. So we are limited to all endpoints that localstack mocks. As Localsttack adds more endpoints we benefit from that. The main idea behind this simple project is to show how easy is to docker-ize somDevOpsps tools and make engineering easy.  Currently is very often to spend 40mim or more doing baking and that's is wrong. So that's kind of mainframe era so the idea is to save time and run things local - as much as possible. Docker helps a lot with that. I run software in production using AWS Amazon Linux. Now t…

Ethereum - Smart Contracts and Blockchain Principles

Ethereum is the most interesting blockchain platform right now. I have lots of interest in IOTA and HashGraph because they claim to solve scalability problems that ethereum and pretty much most of platforms/token have, however they don't have as big adoption and rich ecosystem as ethereum has. Blockchain got popular in 2009 thanks to Bitcoin paper but the market and development industry still pretty new. Ethereum allows us to create Smart Contracts which are far more powerful then what you got when you work with bitcoins. Most of the ideas in blockchain and ethereum platform are not new, they are pretty old concepts like Cryptography, Immutability, Distributed Computing, and Consensus Algorithms. Clearly, the Financial sector was the first application field for blockchain, however, we are not limited to financial applications and we could use for many other industries like HealthCare, Agriculture, Energy, Real estate, Legal, Insurance, Education, Gov and many others. Here you can …

Internal System Design: The Forgotten Discipline

Several years ago it was impossible to think about software development without having at least a business specification(usually in a word document) and a bunch of UML diagrams like package, class and sequence diagram. That was a pre-agile era where process and formality ruled the IT software development industry(yes folks, yes I'm old). Back on that time software engineering was pretty much conceived as "process".  Now we live in a more civilized era, however, back on that time, there were some interesting and valuable things(tools, techniques, principles and even practices).  Like the English expression, you need throw bath water but keeping the baby. I was wondering why these phenomena happened and still happen today. First of all, we are SOCIAL animals we tend to have some kind of "fashion" behavior where we follow the "flow". Needless to say that UML and Software internal design is not quite popular today with the new kids. I don't miss the …

Code for Reliability

It's normal to talk about failure when we talk about cloud-native architectures. Pretty much anything that runs on the cloud(data center) means more distribution(networking). Distributed Systems tend to fail all time. Chaos Engineering is great, however, it's for the infrastructure, therefore focused on the big picture. When I say big picture I mean outside of a microservice. Reliability is not just for the outside boundary or infrastructure but also for Inside a service or inside the system. Reliability Inside the system might have many names such as(very popular in Brazil during 2k years) Defensive Programming.  There are lots of synergy between reliability, defensive programming, anti-fragility and Efficient Internal System design. Efficient design it's not only about making your system efficient in sense of economics(code, readability, reduce maintained cost) but also reliability. Today I want to share some internal system design way of think in order to achieve bette…

C Unit Testing with Check

When I start on IT, several years ago(+15), C was the first language that I learned. But was not the first language I mastered and deployed in production which was...surprisingly Visual Basic 6.0. However, when I learned C or any language back on that old and arcane time noddy was into testing. However now is impossible to work professionally in any language without tests. C is a low-level language but still kicks ass and has several interesting libs/frameworks. People use to fear C a lot because of lack of syntactical sugar but once you get used to it is not that hard at all. Today I want to show how you can do Unit Testing with C. I will not show how to do Unit Testing with C++ but with C ANSI actually. We will be using Check a unit test lib for C ANSI.

Generic and Script Remediation

Some time ago I was sharing my experiences with a Dynomite Remediation Process I wrote. Iḿ using remediation systems for a while and IMHO the add lots of value since they automate manual Cloud Operation work and save time for people. Currently, I refactored my Remediation code and now the same code can support Dynomite / Dynomite Manager but also Apache Cassandra. There are very similar concepts between Dynomite and Cassandra Remediation such as discovering AWS EC2 Ips to be remediated, AWs Resourcing(Creating SGs, Deleting LCs, Updating ASGs), Health checking(Is the node up and running? Could I remediate right now or you are in the middle of a backup or just booting up?). Once I identified that core concept I was able to create a high level and generic design for Cassandra and Dynomite. This is great for cases of Patch / Fix(apply a new AMI) or Scale Up(Increase the memory or CPU for instance). Although that the main use cases and most important ones there were some corner cases and…

Mocking and Testing AWS APIs with TestContainers and LocalStack

Cloud Computing is the default today. I do believe the future is containers and multi-cloud solutions. However today I work a lot with AWS. There are specific endpoints such as S3, AutoScaling, Route53 and other that are among the ones I use more in my day to day work. AWS API is easy to use however not no easy to test. Distributed systems tend to be hard to test. Having quick feedback is very important for engineers. There is some kind of tasks that need to interact with AWS APIs like S3 for backups for instance. However, if you need to wait to deploy in AWS to test it because is basically impossible to test it locally then we have a problem. We need to be able to do end-2-end testing however while you are coding or doing some troubleshooting is important to do things faster. There are 2 specific projects that can help us with this task. TestContainers and LocalStack. Today I will show to use LocalStack and TestContainers together with JUnit in order to do unit tests mocking S3 API.…

DevOps Engineering with Service Orientation

DevOps Engineering is often "perceived" as a operations matter. In fact, it's true but is not 100% JUST ops. There are some blurred lines between engineering and operations. When we take a look at the DevOps ecosystem we can highlight some great tools like Ansible, Terraform, Jenkins, Packer and so many others. Provisioning, Release, Observability, and SRE are different aspects of a broader DevOps adoption. The higher benefit comes when we start applying more engineering to DevOps. It's very important to have good tooling around. When we think about tooling engineering makes a lot of sense, however, even outside of the tooling space there are areas where we definitely need an engineering injection like Cloud Production Operation Automation. Releasing is the easy part, things get complicated when you automate the operational aspects of your systems. It sounds so meta or Inception(If you watch the movie) but Systems managing Systems looks the way to go for me. This pa…

Dynomite Remediation

Working with the cloud is great but it's not easy. The Cloud allows us spins machines pretty easily, however, update these machines are not that an easy task, especially if we talking about databases. Today I want to share some experience with remediation process I've built for Dynomite in my current project.  I'm using NetflixOSS Dynomite in production(AWS) for more than 2 years now both as cache but also as Source of Truth. I need to say that is rock solid and just works. However often we need to increase some machine capacity, i.e: Add more memory or increase the disk size. There are times where we need to apply patches on the OS - I use Amazon Linux on Production on EC2 and recently there was the Meltdown and Spectre situation. Other times there are telemetry or even provisioning and configuration bugs that need to be fixed.  No matter the reason you will always need, time to time, change something on the underlying AMI or in the Java or C application which is running…

Lessons Learned using AWS Lambda as Remediation System

Today I want to share some experiences I had in the last 2 years with AWS Lambda as a Serverless solution for Microservices Remediation. This was not a single effort from me only but a team effort. The lessons learned I want to share with you today was not only in sense of Serverless but also in sense of DevOps Engineering.

DevOps Engineering is the norm today. I really can't see a world without it. Code it is the lingua franca of everything around IT. Currently, there are more and more abstractions and solutions rising for DevOps Engineering, chances are we will need code fewer things as the time pass. However, as we push the boundaries of innovation we face new problems and new problems will always require better solutions.

DevOps Engineering is great however it's not a FREE LUNCH at all, like microservices, which are great too, there are lots of COST that are introduced. There is a requirement do structure and lay down teams in a different way and also there is a need for …

Kanban for DevOps Engineering: From Sense to Predictability

Kanban is a Lean tool. It's focused on the psychological part of the work. In a nutshell, you manage the system via constraints and you don't manage people. I used kanban a lot in the last 8 years. When I started I was a Software Architect | Coach in not-so-traditional SOA projects. 

Today I work a lot with DevOps Engineering and Cloud Computing and I don't work with 2 weeks sprints but with quarters which are 3-months length buffers of work. You might think as a quarter as a super big sprint. There are several different kinds of classes of work like (A) Support for Developers, (B) Tuning, (C) Stress Testing, (D) NoSQL Development, (E) Telemetry and Observability, (F) Distributed Tracking and so on and on. I'm working on CORE team with other architect and DevOps Engineers.

For a long time, I was using kanban metrics and sheets pretty religiously. Because I was working with Big Teams(up to 33 people) and multiple teams who had to deliver on time. As more close you are …

Feeling OOP in C

C language is old(since 1972 to be precise) but still kicks ass. Some folks consider C too low level. IMHO is not that low level, actually, there are some abstractions and syntactical sugars I miss in C but IMHO is something you can leave without since you are befiniting from better performance.

Many developers care too much about syntax today and not many care about performance. It really depends on what you are doing and what kind of company you are. There are languages that are more productive than C for sure, however, once you get how to do things properly is not bad at all actually is quite good.  Most of the big languages like Java, C#, python, ruby are actually based on C.

C often is a language the most of people fear it. Maybe because we got used to much with languages that do a lot of things for us. IMHO is not that hard, it's just a matter of taste, after some time I learn to enjoy working with C professionally and today I like it a lot. But this really depends on your …

Debugging Redis Module

Lately, I'm working with custom redis-modules and I need to say: redis-modules kick ass. I started working with Cthulu and coding modules with javascript which was nice and very productive but given some of my current use cases I had to go to C.

I recommend you take a look at my previous posts here:
  - Building Redis-Modules with JavaScript using Cthulhu
  - Dispatching Custom Commands with Lettuce and Redis-Modules

Today I want to you how to do proper debug with Redis modules written in C language. I will be using eclipse Oxygen CPP in order to debug redis module and redis 4.

Here there is a video which I recorded with a simple live demo about how to debug a redis-module project and redis. The source code is available on my GitHub. I hope this is useful for you.

Stability Principles

Last year I started to blog about something I'm calling "Stability Mindset". I'm still working on the ideas but I think I had enough time to time and experiences to evolve the concept to some basic guiding principles.

You might be wondering if I'm just trying to do some kind of ugly re-branding of Reliability like Software Reliability Engineering(SRE) but I don't think is the case - I see is different things or at least a specific subset of SRE. If you are interested in a broader spectrum you can check out google insights on the matter. Google has this amazing book you can check it out about SRE.

I thought a lot if this was just SRE for mere mortals but I don't think it is. I really think is a different kind of problem or at least different subset for sure. Today I'm writing this post because I want to capture some of my recent findings and also because I did a retrospective exercise with my team today I promise I you share some of my thoughts.