Running AWS ES Open Distro locally

AWS Open Distro for ES is an open source distribution from Amazon for ElasticSearch cluster and Kibana. Today I will show a simple script I made in order to run ES/Kibana locally without pain. You will be able to run the solution without worrying about any configuration. You need to keep in mind this is a developer sandbox script, this is not production ready ES cluster config. AWS ES Open Distro is enterprise-grade because it has advanced capabilities like SQL, Alerting and cluster diagnostics, I like the cluster tools a lot.

The Script

	#!/bin/bash
	docker run -d -P --rm --name es -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" amazon/opendistro-for-elasticsearch
	docker run -d -P --rm --name kibana --link es:es -p 5601:5601 amazon/opendistro-for-elasticsearch-kibana:0.7.0
	docker cp kibana.yml kibana:/usr/share/kibana/config/kibana.yml
	docker exec -t kibana /bin/sh -c "cat /usr/share/kibana/config/kibana.yml"
	sleep 15
	curl -XGET https://localhost:9200 -u admin:admin --insecure | jq .
	curl -XGET https://localhost:9200/_cat/nodes?v -u admin:admin --insecure | jq .
	curl -XGET https://localhost:9200/_cat/plugins?v -u admin:admin --insecure | jq .
	docker exec -it es /bin/sh -c "cat /usr/share/elasticsearch/config/root-ca.pem" > es.pem
	curl -u admin:admin --cacert es.pem -X PUT "https://localhost:9200/twitter/_doc/1" -H 'Content-Type: application/json' -d'
	{
	"user" : "kimchy",
	"post_date" : "2009-11-15T14:12:12",
	"message" : "trying out Elasticsearch"
	}' | jq .
	curl -u admin:admin --cacert es.pem -X PUT "https://localhost:9200/twitter/_doc/2" -H 'Content-Type: application/json' -d'
	{
	"user" : "Diego Pacheco",
	"post_date" : "2009-11-15T14:12:12",
	"message" : "having fun with Elasticsearch"
	}' | jq .
	curl -u admin:admin --cacert es.pem -X PUT "https://localhost:9200/twitter/_doc/3" -H 'Content-Type: application/json' -d'
	{
	"user" : "Gandalfy",
	"post_date" : "2009-11-15T14:12:12",
	"message" : "nian cat on Elasticsearch"
	}' | jq .
	curl --silent -u admin:admin --cacert es.pem -X GET "https://localhost:9200/twitter/_search?q=*" | jq .
	echo "open kibana: http://localhost:5601/"
	docker ps

view raw docker-run-es-aws.sh hosted with ❤ by GitHub

	---
	# Default Kibana configuration from kibana-docker.
	server.name: kibana
	server.host: "0"
	elasticsearch.url: https://es:9200
	elasticsearch.ssl.verificationMode: none
	elasticsearch.username: kibanaserver
	elasticsearch.password: kibanaserver
	elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]
	opendistro_security.multitenancy.enabled: true
	opendistro_security.multitenancy.tenants.preferred: ["Private", "Global"]
	opendistro_security.readonly_mode.roles: ["kibana_read_only"]

view raw kibana.yml hosted with ❤ by GitHub

Download PERF TOP CLI

wget https://d3g5vo6xdbdb9a.cloudfront.net/downloads/perftop/perf-top-0.7.0.0-LINUX.zip
unzip perf-top-0.7.0.0-LINUX.zip

./perf-top-linux --dashboard dashboards/ClusterNetworkMemoryAnalysis.json --endpoint localhost:9600
./perf-top-linux --dashboard dashboards/ClusterOverview.json --endpoint localhost:9600
./perf-top-linux --dashboard dashboards/ClusterThreadAnalysis.json --endpoint localhost:9600
./perf-top-linux --dashboard dashboards/NodeAnalysis.json --endpoint localhost:9600

view raw perf-top-linux.md hosted with ❤ by GitHub

The script uses docker and link 2 containers(es and kibana). The script also creates an index in ES called Twitter having 3 documents getting indexed with the following props: user, post_data and message.  The script copy a custom kibana.yml file into kibana container in order to pass proper docker DNS / Container link to ES container - by default config looks up to localhost and I have to change to es DNS(same name used on docker link).

Checking ES and Kibana

ES: https://localhost:9200/_all 
Kibana: http://localhost:5601/app/kibana

ES (_all)

Kibana

Perf Tool

Node Analysis

Cluster Thread Analysis

Cluster Network Analysis

Cluster Overview

When you are done you can simply do:

docker kill es kibana

Cheers,
Diego Pacheco