17 Temmuz 2017 Pazartesi

Creating a Beat : Membeat

Memcached is a high performance multithreaded event-based key/value cache store intended to be used in a distributed system.

We create membeat that collects to memcached statistic and sends to Elasticsearch. Statistics can be inspected by telnet.
Membeat is written in Golang.

Also, you can check what the beats are and how to create a beat posts.

15 Temmuz 2017 Cumartesi

How to create a Beat? (Elasticsearch)

All Beats are written in Golang. If you want to create your own beat, you must install the latest version of golang.
Provided that there is a need to collect other custom data, your own beat based on the libbeat framework can be easily built. libbeat contains packages for sending data to Elasticsearch and logstash, for signal handling, for logging, for configuration file handling, and more.

A simple Beat has two main components; a publisher that sends the data to the specified output, and a component that collects the actual data. The publisher is implemented in libbeat. Generally, you just need to create the logical part of collecting data.

The event that you create is a JSON-like object that contains the collected data to send to the publisher.  What you need in your beat is sending data that you want to take to elasticsearch with the help of this subject.

Beat generator package helps you create your own Beat. Beat generator generates the Beat skeleton.
To generate your own Beat you must install cookiecutter. Now you must decide on a name for the Beat that must be one word all lowercase. cookiecutter will ask you information about your beat.
Beat generator should have created a directory name for your Beat inside out folder with several files. In this way, we have a raw template of the Beat.

To fetch dependencies and set up the Beat, you need to install python and virtualenv. To bring back the dependencies $ make setup command must be used. Now that the simplest beat application is ready for use.

The Beat should implement the new function, run method and stop method.
The New function creates the beat and returns the beat object of type Beats. The Run method implements the Beat its run loop that is to say contains main application loop. Stop method contains logic that is called when the Beat is signaled to stop.

All basic configurations are written in membeat.yml and membeat.template.json files.
When you run the $make setup command, config folder having basic configuration options consists.
This folder reflects the options of configurations in the membeat.yml file. If you want to add a new option, you need to update structure part in config.go and etc/beat.yml.
In order to be valid your updates, you must run the $make update command.

All beats should have a main loop that is very simple. you implement run method.

When creating events, you must use some conventions.

14 Temmuz 2017 Cuma

What is the Elasticsearch Beats?

The Beats are lightweight data shippers that you install your server to send different types of operational data to Elasticsearch.
Filebeat, Metricbeat and Packetbeat are a few examples of Beats.
In more detail, filebeat is used for retrieving logs from your servers.
Where as Packetbeat is for monitoring the network traffic exchanged between your servers.
Metricbeat periodically collects metrics from the operating system and from services running on the server.

We can look metricbeat more closely.
Metricbeat could help monitor servers by collecting metrics from systems and services.
Metricbeat consists of modules and metricsets. A Metricbeat module defines the basic logic for collecting data from a specific service, such as Redis, MySQL, and so on.
The module specifies details about the service, including how to connect, how often to collect metrics, and which metrics to collect.
Also, metricbeat can send data directly to Elasticsearch or send it to Elasticsearch via Logstash, which you can use to parse and transform the data.

There are already 25+ Community Beats made by the community.  Community Beats keeps growing for every day.