17 Ağustos 2017 Perşembe

Etcdbeat : Elastic Beat for Etcd Statistics

What is the Etcdbeat

Etcdbeat is an Elastic Beat that reads stats from the Etcd v2 API and indexes them into Elasticsearch or Logstash.

Configuration Options

We have some settings. You need to adjust the etcdbeat.yml configuration file to your needs. Here is a sample configuration:
input:
  # Defines how often an event is sent to the output
  period: 30

  # Defines the http port serviced
  # Default to :2379
  port: :2379

  # Defines the http host serviced
  # Default to localhost
  host: localhost

  # Statistics to collect
  statistics:
    leader: false
    self: true
    store: true

  # Authentication
  authentication:
    #Default to enable: false
    enable: false
    username: test
    password: test1234

Options

period
How often events are sent to the elasticsearch.
period: 30
port and host
etcdbeat collect data from the etcd v2 API that predefined to localhost: 2379. You should be careful when you change these settings.
 port: :2379 ` 
 host: localhost
statistics
We have three different stats types of documents exported. They are leader stats, self stats and store stats. You can decide which statistics to collect.
statistics: 
  leader: false
  self: true 
  store: true
Leader option is false by default. If you have followers you need to change this true. if all settings are true, three different events are sent to the output.
authentication
Authentication setting is disabled by default. If you want to authenticate, you need to enable authentication in the etcd API and you have to set the username and password.
authentication:  
  enable: false  
  username: test  
  password: test1234 

Run EtcdBeat

First, setup Golang environment (if you don't have it already)
cd $GOPATH
mkdir -p src/github.com/gamegos
cd src/github.com/gamegos
git clone https://github.com/gamegos/etcdbeat.git
To run Etcdbeat with debugging output enabled, run:
./etcdbeat -c etcdbeat.yml -e -d "*"

Exported Fields

We have three different outputs.

Sample of Etcd leader statistics document:

{
  "_index": "etcdbeat-2017.08.17",
  "_type": "etcdbeat",
  "_id": "AV3wL_FkWRDJnTYSks1I",
  "_score": null,
  "_source": {
    "@timestamp": "2017-08-17T12:34:19.088Z",
    "beat": {
      "hostname": "kripton",
      "name": "kripton",
      "version": "5.1.3"
    },
    "leader": {
      "followers": {
        "6e3bd23ae5f1eae0": {
          "counts": {
            "fail": 0,
            "success": 745
          },
          "latency": {
            "average": 0.017039507382550306,
            "current": 0.000138,
            "maximum": 1.007649,
            "minimum": 0,
            "standardDeviation": 0.05289178277920594
          }
        },
        "a8266ecf031671f3": {
          "counts": {
            "fail": 0,
            "success": 735
          },
          "latency": {
            "average": 0.012124141496598642,
            "current": 0.000559,
            "maximum": 0.791547,
            "minimum": 0,
            "standardDeviation": 0.04187900156583733
          }
        }
      },
      "leader": "924e2e83e93f2560"
    },
    "type": "etcdbeat"
  },
  "fields": {
    "@timestamp": [
      1502973259088
    ]
  },
  "sort": [
    1502973259088
  ]
}

Sample of Etcd self statistics document:

{
  "_index": "etcdbeat-2017.08.16",
  "_type": "doc",
  "_id": "AV3rbDQyWRDJnTYSks0_",
  "_score": null,
  "_source": {
    "@timestamp": "2017-08-16T14:22:02.256Z",
    "beat": {
      "hostname": "kripton",
      "name": "kripton",
      "version": "5.5.1"
    },
    "self": {
      "id": "8e9e05c52164694d",
      "leaderInfo": {
        "leader": "8e9e05c52164694d",
        "startTime": "2017-08-16T17:16:00.239282723+03:00",
        "uptime": "6m2.017314336s"
      },
      "name": "default",
      "recvAppendRequestCnt": 0,
      "recvBandwidthRate": 0,
      "recvPkgRate": 0,
      "sendAppendRequestCnt": 0,
      "startTime": "2017-08-16T17:15:59.437631995+03:00",
      "state": "StateLeader"
    },
    "type": "etcdbeat"
  },
  "fields": {
    "@timestamp": [
      1502893322256
    ],
    "self.leaderInfo.startTime": [
      1502892960239
    ],
    "self.startTime": [
      1502892959437
    ]
  },
  "sort": [
    1502893322256
  ]
}

Sample of Etcd store statistics document:

{
  "_index": "etcdbeat-2017.08.16",
  "_type": "doc",
  "_id": "AV3rbDQyWRDJnTYSks1A",
  "_score": null,
  "_source": {
    "@timestamp": "2017-08-16T14:22:02.257Z",
    "beat": {
      "hostname": "kripton",
      "name": "kripton",
      "version": "5.5.1"
    },
    "store": {
      "compareAndDeleteFail": 0,
      "compareAndDeleteSuccess": 0,
      "compareAndSwapFail": 0,
      "compareAndSwapSuccess": 0,
      "createFail": 0,
      "createSuccess": 1,
      "deleteFail": 0,
      "deleteSuccess": 0,
      "expireCount": 0,
      "getsFail": 3,
      "getsSuccess": 0,
      "setsFail": 0,
      "setsSuccess": 6,
      "updateFail": 0,
      "updateSuccess": 0,
      "watchers": 0
    },
    "type": "etcdbeat"
  },
  "fields": {
    "@timestamp": [
      1502893322257
    ]
  },
  "sort": [
    1502893322257
  ]
}

I am working on it to make an upstream module.

17 Temmuz 2017 Pazartesi

Creating a Beat : Membeat

Memcached is a high performance multithreaded event-based key/value cache store intended to be used in a distributed system.

We create membeat that collects to memcached statistic and sends to Elasticsearch. Statistics can be inspected by telnet.
Membeat is written in Golang.

Also, you can check what the beats are and how to create a beat posts.

15 Temmuz 2017 Cumartesi

How to create a Beat? (Elasticsearch)

All Beats are written in Golang. If you want to create your own beat, you must install the latest version of golang.
Provided that there is a need to collect other custom data, your own beat based on the libbeat framework can be easily built. libbeat contains packages for sending data to Elasticsearch and logstash, for signal handling, for logging, for configuration file handling, and more.

A simple Beat has two main components; a publisher that sends the data to the specified output, and a component that collects the actual data. The publisher is implemented in libbeat. Generally, you just need to create the logical part of collecting data.

The event that you create is a JSON-like object that contains the collected data to send to the publisher.  What you need in your beat is sending data that you want to take to elasticsearch with the help of this subject.

Beat generator package helps you create your own Beat. Beat generator generates the Beat skeleton.
To generate your own Beat you must install cookiecutter. Now you must decide on a name for the Beat that must be one word all lowercase. cookiecutter will ask you information about your beat.
Beat generator should have created a directory name for your Beat inside out folder with several files. In this way, we have a raw template of the Beat.

To fetch dependencies and set up the Beat, you need to install python and virtualenv. To bring back the dependencies $ make setup command must be used. Now that the simplest beat application is ready for use.

The Beat should implement the new function, run method and stop method.
The New function creates the beat and returns the beat object of type Beats. The Run method implements the Beat its run loop that is to say contains main application loop. Stop method contains logic that is called when the Beat is signaled to stop.

All basic configurations are written in membeat.yml and membeat.template.json files.
When you run the $make setup command, config folder having basic configuration options consists.
This folder reflects the options of configurations in the membeat.yml file. If you want to add a new option, you need to update structure part in config.go and etc/beat.yml.
In order to be valid your updates, you must run the $make update command.

All beats should have a main loop that is very simple. you implement run method.

When creating events, you must use some conventions.

14 Temmuz 2017 Cuma

What is the Elasticsearch Beats?

The Beats are lightweight data shippers that you install your server to send different types of operational data to Elasticsearch.
Filebeat, Metricbeat and Packetbeat are a few examples of Beats.
In more detail, filebeat is used for retrieving logs from your servers.
Where as Packetbeat is for monitoring the network traffic exchanged between your servers.
Metricbeat periodically collects metrics from the operating system and from services running on the server.

We can look metricbeat more closely.
Metricbeat could help monitor servers by collecting metrics from systems and services.
Metricbeat consists of modules and metricsets. A Metricbeat module defines the basic logic for collecting data from a specific service, such as Redis, MySQL, and so on.
The module specifies details about the service, including how to connect, how often to collect metrics, and which metrics to collect.
Also, metricbeat can send data directly to Elasticsearch or send it to Elasticsearch via Logstash, which you can use to parse and transform the data.

There are already 25+ Community Beats made by the community.  Community Beats keeps growing for every day.