21 Mart 2018 Çarşamba

Metricbeat Module: Etcd Module

Etcd Module is a Metricbeat Module that collects metrics from Etcd v2 API and indexes them into Elasticsearch or Logstash.
Etcd Module is written in Golang like all Metricbeat Modules and Beats.


What is the Etcd?


Etcd is an open-source distributed key value store that provides a reliable way to store data across a cluster of machines.

An etcd cluster keeps track of a number of statistics including latency, bandwidth and uptime. Etcd Module collects this stats with Etcd v2 API.



Etcd Module


We have some files in metricbeat/module/etcd.
Let's list up files in metricbeat/module/etcd:
$ tree etcd 

etcd
├── doc.go
├── leader
│   ├── data.go
│   ├── leader.go
│   ├── leader_integration_test.go
│   ├── leader_test.go
│   └── _meta
│       ├── data.json
│       ├── docs.asciidoc
│       └── fields.yml
├── _meta
│   ├── config.yml
│   ├── Dockerfile
│   ├── docs.asciidoc
│   ├── env
│   ├── fields.yml
│   └── test
│       ├── leaderstats.json
│       ├── selfstats.json
│       └── storestats.json
├── self
│   ├── data.go
│   ├── _meta
│   │   ├── data.json
│   │   ├── docs.asciidoc
│   │   └── fields.yml
│   ├── self.go
│   ├── self_integration_test.go
│   └── self_test.go
└── store
    ├── data.go
    ├── _meta
    │   ├── data.json
    │   ├── docs.asciidoc
    │   └── fields.yml
    ├── store.go
    ├── store_integration_test.go
    └── store_test.go
8 directories, 30 files

Getting Started with Etcd Module


Initially, you need to make Etcd Module enabled:

./metricbeat modules enable etcd 
You can see a list of enabled and disabled modules by running following command:

./metricbeat modules list

Then when you run Metricbeat, it loads the corresponding module configurations specified in the modules.d directory.


  • The modules.d directory contains default configurations for all modules available in Metricbeat. For more information, you can check [1]
Now Etcd Module is enabled and ready to collect metrics.

Configuration


Etcd Module has some settings. If you want to change some config options adjust the modules.d/etcd.yml file to your needs. 

Here is a sample configuration:


- module: etcd
  metricsets: ["leader", "self", "store"]
  period: 10s
  hosts: ["localhost:2379"]

module: The name of the module to run.
metricsets: A list of metricsets to execute.
period: How often the metricsets are executed.
hosts: A list of hosts of fetch information from.

Metricsets


Etcd Module has tree metricsets; leader, self and store.
You need to remove metricset that you don't want to be taken in etcd.yml located in modules.d.

For example; if you want to take only store metricset:

- module: etcd  metricsets: ["store"]  period: 10s  hosts: ["localhost:2379"]

Leader Metricset


The leader has a view of the entire cluster and keeps track of two interesting statistics: latency to each peer in the cluster and the number of failed and successful Raft RPC requests.

Here is an example document generated by this metricset:


{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "beat": {
        "hostname": "host.example.com",
        "name": "host.example.com"
    },
    "etcd": {
        "leader": {
            "followers": {},
            "leader": "8e9e05c52164694d"
        }
    },
    "metricset": {
        "host": "etcd:2379",
        "module": "etcd",
        "name": "leader",
        "rtt": 115
    }
}

Self Metricset


Each node keeps a number of internal statistics.


  • id: the unique identifier for the member
  • leaderInfo.leader: id of the current leader member
  • leaderInfo.uptime: amount of time the leader has been leader
  • name: this member's name
  • recvAppendRequestCnt: number of bytes per second this node is receiving (follower only)
  • recvBandwidthRate: number of bytes per second this node is receiving (follower only)
  • recvPkgRate: number of requests per second this node is receiving (follower only)
  • sendAppendRequestCnt: number of requests that this node has sent
  • sendBandWithRate: number of bytes per second this node is sendling (leader only). This value is undefined on single member clusters.
  • sendPkgRate: number of requests per second this node is sending (leader only).  This value is undefined on single member clusters.
  • state: either leader of follower
  • startTime: the time when this node was started

Here is an example document generated by this metricset:

{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "beat": {
        "hostname": "host.example.com",
        "name": "host.example.com"
    },
    "etcd": {
        "self": {
            "id": "8e9e05c52164694d",
            "leaderinfo": {
                "leader": "8e9e05c52164694d",
                "starttime": "2017-12-07T07:20:33.241031712Z",
                "uptime": "9m4.919923089s"
            },
            "name": "default",
            "recv": {
                "appendrequest": {
                    "count": 0
                },
                "bandwithrate": 0,
                "pkgrate": 0
            },
            "send": {
                "appendrequest": {
                    "count": 0
                },
                "bandwithrate": 0,
                "pkgrate": 0
            },
            "starttime": "2017-12-07T07:20:32.340598119Z",
            "state": "StateLeader"
        }
    },
    "metricset": {
        "host": "etcd:2379",
        "module": "etcd",
        "name": "self",
        "rtt": 115
    }
}


Store Metricset


The store statistics include information about the operations that this node has handled. Operations that modify the store's state like create, delete, set and update are seen by the entire cluster and the number will increase on all nodes.

Here is an example document generated by this metricset:


{
    "@timestamp": "2017-10-12T08:05:34.853Z",
    "beat": {
        "hostname": "host.example.com",
        "name": "host.example.com"
    },
    "etcd": {
        "store": {
            "compareanddelete": {
                "fail": 0,
                "success": 0
            },
            "compareandswap": {
                "fail": 0,
                "success": 0
            },
            "create": {
                "fail": 0,
                "success": 1
            },
            "delete": {
                "fail": 0,
                "success": 0
            },
            "expire": {
                "count": 0
            },
            "gets": {
                "fail": 1,
                "success": 35
            },
            "sets": {
                "fail": 0,
                "success": 2
            },
            "update": {
                "fail": 0,
                "success": 0
            },
            "watchers": 0
        }
    },
    "metricset": {
        "host": "etcd:2379",
        "module": "etcd",
        "name": "store",
        "rtt": 115
    }
}





Etcd Module available for Beats 6.1 version.

My Metricbeat Process

Now I am going to clarify how this process by my myself goes on apart from Etcd Module.

Etctbeat and Etcd Module were my 3-month internship project. In this way, my process of starting the contributing the Beats has begun. 

Very long before starting the internship, I had begun to the Golang and Beats under the guidance of Kaan and my teacher Necdet Yücel. And our meeting with Kaan happened thanks to Necdet Hoca.

I want to be a part of free software community and we were all agree about Beats. Everything started with my internship that I did it with Kaan for a Gamegos firm. 

We were not sure that we should make a Community Beats or Metricbeat Module with Etcd. What I mentioned here is that you can ask that on the page of discuss of elastic

In order to get used to this process, I have started to work for contributing to Etcdbeat.
The group in front of you is so much eager to work with you that they can be one of your motivation sources.

From the beginning, our aim was Metricbeat Module. Because it is an upstream project. So I have started to work for Metricbeat Module.

At that part, there is so much to learn and Module has its own necessities. On the contrary to Etcdbeat, test files were needed to be added as well. The creation process of Module was both teaching and joyful. 
I send the first pull request for Etcd Module at the last week of the internship.
After correcting a few mistakes, Etcd Module was also accepted.
And you can find in Beats 6.1 version.

Etcdbeat was my first project. And also Metricbeat Module was my first upstream project.
I learned that if you work enough, the results are very satisfying. During this period, I really appreciate both people on the side of Elastic, my teacher Necdet Yücel and Kaan.


    Hiç yorum yok:

    Yorum Gönder