21 Mart 2018 Çarşamba

Metricbeat Module: Etcd Module

Etcd Module is a Metricbeat Module that collects metrics from Etcd v2 API and indexes them into Elasticsearch or Logstash.
Etcd Module is written in Golang like all Metricbeat Modules and Beats.


What is the Etcd?


Etcd is an open-source distributed key value store that provides a reliable way to store data across a cluster of machines.

An etcd cluster keeps track of a number of statistics including latency, bandwidth and uptime. Etcd Module collects this stats with Etcd v2 API.



Etcd Module


We have some files in metricbeat/module/etcd.
Let's list up files in metricbeat/module/etcd:
$ tree etcd 

etcd
├── doc.go
├── leader
│   ├── data.go
│   ├── leader.go
│   ├── leader_integration_test.go
│   ├── leader_test.go
│   └── _meta
│       ├── data.json
│       ├── docs.asciidoc
│       └── fields.yml
├── _meta
│   ├── config.yml
│   ├── Dockerfile
│   ├── docs.asciidoc
│   ├── env
│   ├── fields.yml
│   └── test
│       ├── leaderstats.json
│       ├── selfstats.json
│       └── storestats.json
├── self
│   ├── data.go
│   ├── _meta
│   │   ├── data.json
│   │   ├── docs.asciidoc
│   │   └── fields.yml
│   ├── self.go
│   ├── self_integration_test.go
│   └── self_test.go
└── store
    ├── data.go
    ├── _meta
    │   ├── data.json
    │   ├── docs.asciidoc
    │   └── fields.yml
    ├── store.go
    ├── store_integration_test.go
    └── store_test.go
8 directories, 30 files

Getting Started with Etcd Module


Initially, you need to make Etcd Module enabled:

./metricbeat modules enable etcd 
You can see a list of enabled and disabled modules by running following command:

./metricbeat modules list

Then when you run Metricbeat, it loads the corresponding module configurations specified in the modules.d directory.


  • The modules.d directory contains default configurations for all modules available in Metricbeat. For more information, you can check [1]
Now Etcd Module is enabled and ready to collect metrics.

Configuration


Etcd Module has some settings. If you want to change some config options adjust the modules.d/etcd.yml file to your needs. 

Here is a sample configuration:


- module: etcd
  metricsets: ["leader", "self", "store"]
  period: 10s
  hosts: ["localhost:2379"]

module: The name of the module to run.
metricsets: A list of metricsets to execute.
period: How often the metricsets are executed.
hosts: A list of hosts of fetch information from.

Metricsets


Etcd Module has tree metricsets; leader, self and store.
You need to remove metricset that you don't want to be taken in etcd.yml located in modules.d.

For example; if you want to take only store metricset:

- module: etcd  metricsets: ["store"]  period: 10s  hosts: ["localhost:2379"]

Leader Metricset


The leader has a view of the entire cluster and keeps track of two interesting statistics: latency to each peer in the cluster and the number of failed and successful Raft RPC requests.

Here is an example document generated by this metricset:


{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "beat": {
        "hostname": "host.example.com",
        "name": "host.example.com"
    },
    "etcd": {
        "leader": {
            "followers": {},
            "leader": "8e9e05c52164694d"
        }
    },
    "metricset": {
        "host": "etcd:2379",
        "module": "etcd",
        "name": "leader",
        "rtt": 115
    }
}

Self Metricset


Each node keeps a number of internal statistics.


  • id: the unique identifier for the member
  • leaderInfo.leader: id of the current leader member
  • leaderInfo.uptime: amount of time the leader has been leader
  • name: this member's name
  • recvAppendRequestCnt: number of bytes per second this node is receiving (follower only)
  • recvBandwidthRate: number of bytes per second this node is receiving (follower only)
  • recvPkgRate: number of requests per second this node is receiving (follower only)
  • sendAppendRequestCnt: number of requests that this node has sent
  • sendBandWithRate: number of bytes per second this node is sendling (leader only). This value is undefined on single member clusters.
  • sendPkgRate: number of requests per second this node is sending (leader only).  This value is undefined on single member clusters.
  • state: either leader of follower
  • startTime: the time when this node was started

Here is an example document generated by this metricset:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "beat": {
        "hostname": "host.example.com",
        "name": "host.example.com"
    },
    "etcd": {
        "self": {
            "id": "8e9e05c52164694d",
            "leaderinfo": {
                "leader": "8e9e05c52164694d",
                "starttime": "2017-12-07T07:20:33.241031712Z",
                "uptime": "9m4.919923089s"
            },
            "name": "default",
            "recv": {
                "appendrequest": {
                    "count": 0
                },
                "bandwithrate": 0,
                "pkgrate": 0
            },
            "send": {
                "appendrequest": {
                    "count": 0
                },
                "bandwithrate": 0,
                "pkgrate": 0
            },
            "starttime": "2017-12-07T07:20:32.340598119Z",
            "state": "StateLeader"
        }
    },
    "metricset": {
        "host": "etcd:2379",
        "module": "etcd",
        "name": "self",
        "rtt": 115
    }
}


Store Metricset


The store statistics include information about the operations that this node has handled. Operations that modify the store's state like create, delete, set and update are seen by the entire cluster and the number will increase on all nodes.

Here is an example document generated by this metricset:


{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "beat": {
        "hostname": "host.example.com",
        "name": "host.example.com"
    },
    "etcd": {
        "store": {
            "compareanddelete": {
                "fail": 0,
                "success": 0
            },
            "compareandswap": {
                "fail": 0,
                "success": 0
            },
            "create": {
                "fail": 0,
                "success": 1
            },
            "delete": {
                "fail": 0,
                "success": 0
            },
            "expire": {
                "count": 0
            },
            "gets": {
                "fail": 1,
                "success": 35
            },
            "sets": {
                "fail": 0,
                "success": 2
            },
            "update": {
                "fail": 0,
                "success": 0
            },
            "watchers": 0
        }
    },
    "metricset": {
        "host": "etcd:2379",
        "module": "etcd",
        "name": "store",
        "rtt": 115
    }
}





Etcd Module available for Beats 6.1 version.

My Metricbeat Process

Now I am going to clarify how this process by my myself goes on apart from Etcd Module.

Etctbeat and Etcd Module were my 3-month internship project. In this way, my process of starting the contributing the Beats has begun. 

Very long before starting the internship, I had begun to the Golang and Beats under the guidance of Kaan and my teacher Necdet Yücel. And our meeting with Kaan happened thanks to Necdet Hoca.

I want to be a part of free software community and we were all agree about Beats. Everything started with my internship that I did it with Kaan for a Gamegos firm. 

We were not sure that we should make a Community Beats or Metricbeat Module with Etcd. What I mentioned here is that you can ask that on the page of discuss of elastic

In order to get used to this process, I have started to work for contributing to Etcdbeat.
The group in front of you is so much eager to work with you that they can be one of your motivation sources.

From the beginning, our aim was Metricbeat Module. Because it is an upstream project. So I have started to work for Metricbeat Module.

At that part, there is so much to learn and Module has its own necessities. On the contrary to Etcdbeat, test files were needed to be added as well. The creation process of Module was both teaching and joyful. 
I send the first pull request for Etcd Module at the last week of the internship.
After correcting a few mistakes, Etcd Module was also accepted.
And you can find in Beats 6.1 version.

Etcdbeat was my first project. And also Metricbeat Module was my first upstream project.
I learned that if you work enough, the results are very satisfying. During this period, I really appreciate both people on the side of Elastic, my teacher Necdet Yücel and Kaan.


    14 Mart 2018 Çarşamba

    How to Create a Metricbeat Module?


    Metricbeat is written in Golang like all Beats. If you want to create to Metricbeat Module, you must install the latest version of Golang.
    I recommend you to install vim-go.

    What is the Metricbeat?


    Metricbeat is a lightweight shipper that you can install on your servers to periodically collect metrics from the operating system and from services running on the server. Metricbeat takes the metrics and statistics that it collects and ships them to the output that you specify, such as Elasticsearch or Logstash.

    Metricbeat consists of modules and metricsets.  A Metricbeat module defines a basic logic for collecting data from a specific service. You can configure the frequency at which Metricbeat collects the metrics and what specific metric metrics to collect using these modules and metricsets.

    Contributing a Metricbeat Module


    At first, you should get enough information about the service whose statistics you would like to collect such as how you can have the statistics or what kind of outputs it produces. For a module to be accepted, it should focus on fetching service metrics directly from the service itself.

    Metricbeat Module and Community Beats are different, but it's a good idea to create your own beat before you start working on the Metricbeat Module. 
    For more information about Community Beats check [1] and [2].


    First Part of Modules: Metricsets


    A metricset is the part of a Metricbeat module that fetches and structures the data from the remote service.

    To create a new metricsets you can run the following command inside Metricbeat directory:
    make create-metricset
    When you run this command, it creates all the basic files for your metricset, along with the required module files if the module doesn't already exist. The metricset that you created is already a functioning metricset and can be compiled.

    And then compile your new metricset by running the following command:
    make collect
    make
    The first command updates all generated files with the most recent files, data, and meta information from the metricset. The second command compiles your source code.

    For example:
    $ make create-metricset 
    Module name: kripton                   
    Metricset name: metricwork           
    Module kripton created.                 
    Metricset metricwork created.         
    $ make update
    $ make
    This should now have created a directory kripton inside metricbeat/module with several files. Let's change to this directory and list up files automatically created.
    $ cd kripton
    $ tree
    .
    ├── doc.go
    ├── _meta
    │   ├── config.yml
    │   ├── docs.asciidoc
    │   └── fields.yml
    └── metricwork
        ├── _meta
        │   ├── data.json
        │   ├── docs.asciidoc
        │   └── fields.yml
        └── metricwork.go
    3 directories, 8 files

    The {metricset}.go file contains the logic on how to fetch data from the service and convert it for sending to the output.

    The generated file looks like [3].
    For more details, you can check Beats Developer Guide and Metricset Details.

    Second Part of Modules: Testing


    It's very important to add tests for your metricset. 
    There are three different types of tests:
    • unit tests
    • integration tests
    • system tests
    You should use a combination of the three types to test your metricsets.
    You have to ensure the 80% test coverage. 
    For more details about testing you can check [4].

    Last Part of Modules: Module Files


    Metricbeat modules are used to group multiple metricsets together and to implement shared functionality of the metricsets.
    It's very important to complete documentation and configuration files for a module. When you create new metricset, default versions these files are generated in the _meta directory.

    For example: 

    We already have kripton module and skipping creating module kripton when we running make create-metricset command. Just new metricset created.

    $ make create-metricset
    Module name: kripton
    Metricset name: metrickripton
    Module already exists.
    Skipping creating module kripton.
    Metricset metrickripton created. 
    $ tree _meta 
    _meta/
    ├── config.yml
    ├── docs.asciidoc
    └── fields.yml

    The config.yml file contains the basic configuration options. Looks like this:

    - module: kripton   metricsets: ["metricwork","metrickripton"]   enabled: false   period: 10s   hosts: ["localhost"]

    We have multiple metricsets in kripton module, we have to make sure that extend metricset array.


    The docs.asciidoc file contains the documentation about your module. Looks like this:


    == kripton module
    This is the kripton module.

    Use this file to describe your module in more detail and to document specific configuration options.


    The fields.yml file contains the top level structure for the fields in your metricset. Looks like this:
    - key: kripton
      title: "kripton"
      description: >
        experimental[]
        kripton module
      fields:
        - name: kripton
          type: group
          description: >
          fields:

    This blog post is included basic information about Metricbeat Modules. For more information, you can look into Metricbeat Modules.