Alerting and Monitoring with Sensu part 1

Sensu is an Infrastructure and monitoring / telemetry solution. There are big sites using it such as Yahoo, Yelp, Cisco, and Tesla.

Sensu provides standard solutions for monitoring and alerting.Which I can say is okay.  However If you have been following my Telemetry and Microservices blog posts series(part1 and part2) you will see sensu is not ideal for other use cases like: 

* Dynamic Thresholds / ML
* Infrastructure Tracking
* Canary

Because Sensu it doesn't have a Time Series database. You can use Sensu with Graphite or InfluxDB to work around it. However IMHO the solution it would be: have collectors(collectd) to get basic HOST metrics plus app instrumentation and them make checkers that do queries over the TSDB. 
Sensu Features

  • Sensu has a centralized dashboard thought Uchiwa. 
  • Alerting via checkers: You call a script or API which returns 0, 1 or 2. 
  • Notifications: Email, Slack, Hipchat, Pager Dutty. 
  • Nagios and Zabbix Script reuse. Since they all use the same format: 0,1 and 2.
  • Dynamic Client registration and de-registration
  • Lots of plugins most in ruby and python: https://sensuapp.org/plugins
Sensu can be deployed in Cloud-Native environments or traditional bare metal data centers if you wish. Sensu has a client / server architecture based on a Pub/Sub model delivered by RabbitMQ.  Sensu server works with Debian and CentOS OS and sensu client can work with Debian, CentOS, Mac, Windows, AIX, Solaris, and FreeBSD. 

Sensu is heavily dependent on the OS. Sensu client must be installed on each HOST and the configs are stored in JSON files. Sensu client scripts often are running on the HOST side which means you need to copy the scripts to the HOST and have a provisioning solution like Ansible, Puppet or Chef.

I have lots of mixed feelings with this. In one side of the coin, this is great for operations because they know Ansible, however.  It makes all about provisioning and backing. What I don't like about it that you will need to generate a new version of the AMI and reploy yout app again. That is specially bad for me because letś say you want to change a Threshold and them you will need to reploy everything because a single number? 

Sensu Architecture


Sensu works in a client-server architecture where the server schdulle checkers to run into the clients which are deployed into HOSTS. RabbitMQ is used a transport mecanism beetween sensu server and the clients. Results are stores into the memory using REDIS at the Sensu Server.  All this information can be queried by the APi which will use the information store in REDIS.

Sensu CORE Concepts

Sensu has same core concepts like: Checkers, Event Processors, Handlers, Mutators. Let's see they one by one with more details. 

Service Checkers: You have to use Checkers in order to do HOST monioring. For instance your web server disk is full or the CPU is high or the memory is growing up a lot are all thrsolds you might want be alerted on. So you have to write a script or call and API which will return 0 if its all good, 1 meaning something is in warning and 2 mening critical. 

Event Processing; This is how sensu allow you to so some simple trivial Event Processing. SO you can have another script which can run based on other handlers. You can so somethings, for instance you can take actions based on alerts. If the Disk is full you can use boto cli in AWS and get more disk, this can be executed based on a alert. You can use this feature to do custon alerting like sensing a email or a custon notification channel or even use this to store metrics into a diffrent place like a TSDB. You can also use other event processors such as Filters to get just the events you want or even use Mutators which can change data. 

Sensu Handlers: There are some types of Handlers in sensu. The default is the Pipe handler where you can use standard linux commands via STDIN. There are TCP and UDP handlers, Transport Handlers like RabbitMQ and a special kind of handler called SET handler which you can use to GROUP event handlers.

RESTful API: Sensu has a RESTful API that expose inofrmation about the clients, checks, events, results, aggregates ans stashes. 

Sensu is lightweight and easy to use. There are some fundamental issue like the backing issues and some bad choises like use a TEMP dir to store information which can gave you lots of pain :-)  Next blog post I will show how to work with Sensu in practice and provide a Vagrantfile so you can play with it easily. 

Cheers,
Diego Pacheco




Popular posts from this blog

C Unit Testing with Check

Image
When I start on IT, several years ago(+15), C was the first language that I learned. But was not the first language I mastered and deployed in production which was...surprisingly Visual Basic 6.0. However, when I learned C or any language back on that old and arcane time noddy was into testing. However now is impossible to work professionally in any language without tests. C is a low-level language but still kicks ass and has several interesting libs/frameworks. People use to fear C a lot because of lack of syntactical sugar but once you get used to it is not that hard at all. Today I want to show how you can do Unit Testing with C. I will not show how to do Unit Testing with C++ but with C ANSI actually. We will be using Check a unit test lib for C ANSI.
read more »

Having fun with Zig Language

Image
Zig is a general-purpose language created in 2016 by Andrew Kelly .  Zig aims to fix the issues C language has while being pragmatic and simple at the same time. Zig really shines as being simple by not having any hidden control flows, not having hidden memory allocations, no pre-processors, and no macros. Basically, it's one language and one language only. Since pre-processors and macros can get really complex and become their own universe. Sure it's not the Zig Philosophy.  Zig Compiler(Zig cc) also is a C compiler, and the interoperability from C to Zig is very good. Clearly, we can see influence from other languages like Go and Rust; however, Zig is pretty unique and different from Go and Rust.  Zig is fast. Zig is perfect for system programming. I bet we will see a lot of solutions being built in Zig. As a languages aficionado, I like to learn at least one language per year and sometimes more; Zig was my 2021 pick, did not regret it. Even if you dont use all languages in p
Keep reading

HMAC in Java

Image
In 2018 AWS CTO(Vogels) said : "Security is everyone's job". For the last 2 years, there was so many famous and big data leaks and breaks that made that statement be very true more than ever. Security often could mean worst performance and worst user experience so in order to get it right you really need to think about the designs before jumping into to code and consider performance and user experience has main requirements.  I was thinking about writing about how to do HMAC in Java for a while and recently Redis 6.0.0 come out and to my surprise, there was a refactoring on the password part in order to use HMAC.  Security easily could scare engineers often because it is not something well spread yet but I believe this will change soon. It does not matter if you have to deal with PII Data or not, security is super relevant because everybody is running their workloads at the cloud or with IoT and Edge devices which means more distribution, more code, more points of fail
read more »