Posts

Showing posts from 2019

Rust - The Language of the future: Rich Type System, Correct and Fast

Image
Learning is a chaotic process sometimes. It's like trying to fix a puzzle with 1k pieces, sometimes you don't know where you are going but you need to feel the vide and let your gut feeling drive you. Being a software engineer means that I need to learn every single day, otherwise I'm devaluing myself and also devaluing the solutions I built and therefore the customer is getting less value at the end of the day. IMHO a software engineer is like a startup that went IPO if it's sexy, healthy, awesome and doing bleeding edge stuff the stocks go up otherwise they might go down. However, for many engineers, the only way to get better in change jobs which I think is ultimately true in the IT industry, however, I would like to ask a question, What's the point to change job if you will be the same? So in order to change you need to change first. Rust language was the last language I learned so far(2019). Every time I learn a language I learn something new and that's i…

Reflections on Serverless:From SOA to Serverless part 3

Image
This is the 3rd part and last one on the series: Reflections on Serverless from SOA to Serverless. On the previous parts 1 and 2 I was explaining how software architecture evolved in several movements and how much we need to change our mindset in the sense of Culture, Movements, and Experiments. Our tools are changing pretty quickly and multi/poly cloud will be soon a reality for most companies. Even if so few companies have this problem in sense of availability and active-active global biz, most of the companies can benefit from multi-poly cloud in sense of getting lock in free from cloud vendors, cost reduction for better and multi offerings, better solutions for specific problems and much much more.

Microk8s

Image
Microk8s is another lightweight k8s distribution - perfect for local tests, an interesting alternative to Minikube, k3s, and Kind.  Microk8s works smoothly in 42 flavors of Linux(Geek Moment: You know is the right answer when you see the number 42. LOL. ).  Microk8s has easy install and several interesting features like Local Storage, Dashboards, Metrics, DNS, Ingress, Istio and much more.

Zero Ops needed with a single k8s cluster done right. That's the marketing of microk8s. In practices, the product works very well but I found a bit slow(start and stop). However installing istio, is 1 command line away from happiness.  Microk8s supports several versions of Kubernetes from 1.10+ to 1.13+ right now(20/04/2019) - k8s is 1.14+.  If you are a hardcore Linux user you must try mcirok8s.




Running Microk8s on Ubuntu



Cheers,
Diego Pacheco

Kind - Kubernetes in Docker

Image
Kind - Allow us to run Kubernetes in Docker. This is not a production-ready solution - however, has a lot of potential in order to make a safe and lightweight alternative option to Minikube.  Kind is built with Go and uses Docker API in order to get things spinning.
I got impressed with the speed thing runs with kind, this is an interesting alternative together with k3s and Minikube.




Using Multiple profiles with Minikube

Image
Kubernetes is the new Linux. K8s is the spec for the multi-poly cloud world. Running k8s could be very resource intensive, so is always a good idea being able to run things locally. For several reasons like Engineering Productivity, Tests and Experiments and so on and on. If you are working with Istio like I'm, you might realize it's a bit heavy to run local, especially if you do have other things running on k8s.  Minikube is the goto solution for local kubernetes clusters. However, as I said before, it can get pretty heavy when Istio gets involved. So the solution is pretty simple, it not much advertised. Minikube has a profile feature which allows you to create multiple profiles. Each profile will be a 2GB DISK VM created in Virtualbox. This is great because now you can run multiple kubernetes versions and multiple clusters doing multi experiments. IMHO is always great to have a k8s cluster ready to test things so I have multiple profiles like istio, lightweight, tests, etc…

GIT based wiki with Gollum

Image
We all need information. There are some many things we need to remember on a standard DevOps project. There are multiple things that we can forget pretty easy. I know so many teams that really only on Slack to track information and that is a big problem. IMHO no one wants complex and heavyweight tools to track information. Developers want simple, easy and pain-free tools like wikis. The problem with most of the wiki tools is that they are heavy and complex to manage. IMHO for small and medium projects we a better solution. I love Git and I love GitHub. GIT has some many nice properties like Easy to audit, easy to merge, we never lot updates, we know everything that happened, we can get back in time pretty easy. For those reasons I really like GIT based Wikies like Gollum. Gollum is built with Ruby and has a nice and clean UI. Using Gollum UI - Gollum translates all the changes into git commands and make commits for us. When we run Gollum, it takes care of everything - the only thing …

Kaf: useful kafka cli

Image
Kafka is the leading distributed Log OSS solution. It is great to inject large amounts of data and is used by many companies nowadays. Kafka is used in many use cases like Data Injection, Stream/Batch processing, Microservices CQRS, and ES and many others. Some years ago a showed how to get started with Kafka. Now I want to share a pretty useful and simple tool called Kaf. Kaf is written in Go language and was inspired by Docker and Kubectl. Today I will how to download and use Kaf. Have fun.


JQ the best Devops Json Processor tool

Image
JQ is a very powerful json processing tool. JQ is great because is written in C and is super fast. JQ also make json appear with a color highlight when you use it in Linux terminal. JQ is useful because you can combine with other DevOps tools like kubectl, docker, ectd, vault, and so on one. Basically, anything that returns a json output you can use JQ in order to process it. JQ has lots of interesting features and is possible to do pretty simple and yet powerful json processing. JQ allow us to subtract parts of the json document and also provide some powerful processing function like map, add, flatten and much more. It's possible to do some very high level and simple functional programming with it.


Now let's take a look on Sheet Cheat I made for JQ.

Running Multithreaded Redis using KeyDB and Dynomite

Image
Keys is the default oss standard for K/V store for many use cases and solutions. When you need run low latency, high throughput IN Memory Persistence - Dynomite is your solution. I always wanted to have multi-thread support in redis like Memcached has. Now someone made it and call it KeyDB. The best thing is the fact that this is done supporting the redis RESP protocol, so all your tools, scripts and code using redis protocol directly still works. Another killer feature in KeyDB is the support for FLASH storage, only available on Redis Enterprise. KeyDB also supports backup to S3. For this post, I will show how to build and run KeyDB which is dead simple and also how to use it with Dynomite like it was Redis.





KeyDB Benchmarks

As you can see on the KeyDB benchmarks, KeyDB does 1.5x less latency and 50% more throughput. These numbers are pretty impressive. The best part is the fact that is all transparent.

Running KeyDB


Running KeyDB with Dynomite



See the Results

KeyDB running


Dynomite Run…

Running AWS ES Open Distro locally

Image
AWS Open Distro for ES is an open source distribution from Amazon for ElasticSearch cluster and Kibana. Today I will show a simple script I made in order to run ES/Kibana locally without pain. You will be able to run the solution without worrying about any configuration. You need to keep in mind this is a developer sandbox script, this is not production ready ES cluster config. AWS ES Open Distro is enterprise-grade because it has advanced capabilities like SQL, Alerting and cluster diagnostics, I like the cluster tools a lot.

The Script


The script uses docker and link 2 containers(es and kibana). The script also creates an index in ES called Twitter having 3 documents getting indexed with the following props: user, post_data and message.  The script copy a custom kibana.yml file into kibana container in order to pass proper docker DNS / Container link to ES container - by default config looks up to localhost and I have to change to es DNS(same name used on docker link).

Checking ES and…

Polyglot and Native runtime with GraalVM

Image
GraalVM is a universal VM mainly for Java and JVM supported languages. With GraalVM you can run native programs written in Python, Ruy, R and all JVM languages like Scala, Clojure, Groovy and Kotlin also LLVM based like C and C++. Graal VM removes isolation between languages and enables interoperability with natively shared runtime. You might be wondering why botter use other VM? Well, there are a couple of good reasons. Java is a pretty old language and does not have the best startup time. Today we are all running services in containers on top of kubernetes and we are always looking for ways to improve the customer experience. Startup time is not only a matter of better customer experience but also availability. For the developers, this means better development experience and idiomatic experience with the best solution for each specific problem.



GrralVM Supported Runtimes
GraalVM can be used standalone runtime or embedded in the context of OpenJDK, Node.JS, Oracle Database or MySQL.…

Reflections on Serverless:From SOA to Serverless part 2

Image
In the previous blog post, I describe the architecture evolution from SOA to Serverless. For this blog post, I will continue to talk about the same subject in the sense of strategy evolution and architecture strategy. Business value does not come only by adopting new technology but also from a new way of thinking or the right mindset shift and the right understanding about tech culture and vision. Most companies have the right Tech view and often don't see themselves as tech companies because they think this business is logistics, sales, health care, insurance, finance anything but tech. Nowadays several companies depend so much on Technology in order to service, prosper and deliver better customer experience(CX). However, tech is often seen is a COST CENTER rather than a VALUE center. Because often IT fail to DELIVER and SHOW value to the business. That's why some many companies are looking into the DevOps movement and other movements like Digital Transformation and Bi-Model…

Reflections on Serverless:From SOA to Serverless part 1

Image
New technology is great. It's very important to open space for new ideas, better solutions, cheaper, faster and exciting new things. However, even with technology, there are principles and lessons learned with previous technologies we need to carry on otherwise we will keep making the same mistakes and we might not take full advantage of new tech. Why? Because often new tech requires a new way of things and often times also require to keep some things from the previous movements. Companies want to do Serverless now a day, often because cloud vendors are pushing that hard and because of the short term cost benefit. However, in the long run, Serverless solutions might be way more expensive than in-house solutions using OSS. There is no silver bullet, you need to understand what you are doing in order to take full benefit from new approaches, techs, and ideas rather than just moving everything from on-trend to another.  I want to share some thoughts from several architecture models …

Kubernetes Everywhere with k3s

Image
Kubernetes is becoming the truly cloud-native specification for the cloud. However not all software runs on the cloud, there are other uses cases like embedded software(IoT and microcontrollers) besides IoT there is another kind of computation that is growing more a more nowadays called Edge, thanks to the low latency and facility capabilities. Would be great run the same infrastructure for the edge/IoT as we do for the cloud? Well, rancher folks addressed that with k3s. A super lightweight kubernetes distribution where they replace things and remove features instead of adding, This is great not only for the IoT/Edge uses cases but also for folks who study the cluster in the academia and for the developers in 2 fronts. First on the CI/CD front, so we can spin up a lightweight cluster to run all kinds of tests, this also has a perfect fit with GitOPS model.

Istio & Kubernetes: Developer Productivity and freedom to deliver your OKRs

Image
Innovation is a must have for companies today and work with the product mindset. Given the digital transformation, we live in, having a product mindset is more than delivering software but it is empowering teams and working with OKR-based management models that focus on Business Objectives rather than what teams have to do.

Development teams are motivated by challenges and they should have the freedom to make choices. But these choices can often have a very high cost for the business and low return. Cloud usage is certainly a great disruptive force for all digital businesses. 
There are many ways to architect Cloud-Native solutions (get the most out of cloud) often for a short-term view many companies are opting for Managed Services solution. These solutions, which are often database solutions, but not limited to the database, give a lot of speed for software development but bring two major problems in the long run.




Kubernetes 101

Image
Kubernetes has lots of concepts. It's easy to get lost in the middle of all core concept the project has. However, all these abstractions provide a great benefit which is the standard spec for different kinds of services and workloads. IF cloud providers were born with a common spec/API, we would not be talking about this subject now a day, unfortunately, this is not the reality. Today I want to share a simple presentation I made trying to explain the many concepts present in kubernetes and how they differentiate between each other like what's the difference of a Secret vs ConfigMap or how ReplicateSet is different compared with a Deployment. Also, cover what kinds of service are available and when to use ClusterIP vs Loadbalancer for instance. You will also see GitOps model, which the standard k8s way to work in production.



Running Istio on EKS

Image
Besides all network overhead, Istio offers very interesting trade-off for sacrificing latency and network overhead for developer productivity and stack independence. In previous posts, I blogged a lot about kubernetes, Istio, Aws, Kops, Eksctl and EKS. Today I will show how to run Istio in AWS using EKS. Keep in mind EKS don't support Alpha* Specs right now(v1,v2 or v3) so some demos from the istio best selection of slideware won't work. But is possible to have istio installed and booking app running.




Running k8s on EKS

Image
EKS is the new AWS managed Service for Kubernetes launched at last Re-Invent 2018.  EKS is not available in all regions right now. EKS an option for those who don't want to use KOPS.  For this blog post, I will show how to easily set up a kubernetes cluster in AWS using EKS.  EKS has some benefits, first of all, is a managed service that you are not locked in since the API is kubernetes based so you can easily migrate to other kubernetes installation or even other kubernetes installation in other cloud vendor or on-premises.




Running Istio on AWS with Kops

Image
In previous posts, I show how to run Istio in Minikube and with Docker-Compose/Consul in local env, today I will show how to run on AWS using KOPS.

This installation is Linux based(Ubuntu), I'm running all commands from my local-desktop, if you don't use Linux(shame on you) you can create a virtual-machine on AWS with ubuntu and run this commands there, also is possible to run Vagrant with Linux and them run this commands on Vagrant box as well. Istio runs smoothly in AWS with Kops. You don't need much, pretty much 3 machines(1 master node, 2 minions).  Keep in mind this is not a production-grade setup, for production, you should be running with 3 masters at least for High Availability.




Running Kubernetes on AWS with KOPS

Image
Kops is the best way to have Kubernetes running in AWS. Kops allow us to install kubernetes in EC2. Kops is written in Go. Kops helps us to create, update, maintain and destroy kubernetes clusters on aws. Kops also supports GCP(Google Could Platform). Kops has some interesting ability to generate terraform files if that's your you thing :-).  For this blog post, we will be using AWS ELB as DNS so we won't be using public DNS records which are done by setting carefully the name of the cluster - which need to end with .k8s.local. Right now is way faster to spin up a kubernetes cluster with Kops rather than EKS.



Running istio with docker-compose and consul

Image
In my previous post, I showed how o install and run istio locally with minikube. However, if you don't have 8GB of ram FREE it might not be a good FIT for you. Today I want to show a lightweight approach for a local environment where we can run Istio with Docker, Docker-Compose, and Consul.

I will be doing more posts about istio, this week, talking about how to run Istio on AWS for instance. But going back to this post.

In order to have this solution working in your machine you have some pre-requirements such as: have docker installed, docker-compose installed and kubectl installed.  Running with consult and docker-compose is way easier than running with minikube/kubernetes however you are not as close as the production topology. For istio, we will be using istio version 1.0.5 Let's get started!



Getting Started with Istio and Minikube

Image
Istio is the new standard for microservices in Kubernetes. Around 2014 Netflix defined before everybody else how to do proper microservices using they brand new stack called NetflixOSS. Introducing game-changing concepts for the Cloud Native microservice components such as Mid-tier load balancing, fault tolerance, circuit breaking, retry/timeouts, service registry and discoverability and much more. NetflixOSS was super important and still is in the cloud-native microservices world. Today the great majority of companies who do Java development use