ELK基础

ELK Stack介绍

ELK Stack是三个开放源代码产品的集合:ElasticsearchLogstashKibana

  • Elasticsearch
    是一个基于Lucene搜索引擎的NoSQL数据库,用于存储所有日志。
  • Logstash
    是一个日志管道工具,它接受来自各种源的输入,执行不同的转换,并将数据导出到各种目标。
  • Kibana
    是一个可视化层,可在Elasticsearch之上运行,用于搜索和可视化日志的Web界面。
  • Filebeat
    安装在将其日志发送到Logstash的客户端服务器,Filebeat充当日志传送代理,使用 lumberjack 网络协议与 Logstash 进行通信。

ELK Stack还包括一系列名为Beats的log shippers。

对于小型开发环境,经典架构将如下所示:
elkjing经典架构.png
为了处理为生产中处理大量数据而构建的更复杂的管道,可能会在日志记录体系结构中添加其他组件,以实现弹性(Kafka,RabbitMQ,Redis)和安全性(nginx):
elk弹性架构.png

实践之后将其进一步优化为EFK,F代表Filebeat,用以解决Logstash导致的问题。
efk架构.png

安装ELK Stack

MacOS中安装

准备

  • 安装HomeBrew
  • 安装Java8

安装Elasticsearch

1
2
$ brew install elasticsearch && brew info elasticsearch
$ brew services start elasticsearch

访问
http://localhost:9200

安装Logstash

1
2
$ brew install logstash
$ brew services start logstash

安装Kibana

1
2
3
4
$ brew install kibana
$ brew services start kibana
$ brew services list
$ sudo vi /usr/local/etc/kibana/kibana.yml

内容如下:

1
2
server.port: 5601
elasticsearch.url: "http://localhost:9200”

访问
http://localhost:5601/status

CentOS中安装

准备

关闭SELinux

1
$ vi /etc/sysconfig/selinux

设置如下:

1
SELINUX=disabled

1
2
$ reboot
$ getenforce

安装java

jdk下载地址:http://www.oracle.com/technetwork/java/javase/downloads

1
2
3
$ wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http:%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u191-b12/2787e4a523244c269598db4e85c51e0c/jdk-8u191-linux-x64.rpm"
$ rpm -ivh jdk-8u191-linux-x64.rpm
$ java -version

安装Elasticsearch

安装Elasticsearch

1
2
3
$ rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.rpm
$ rpm -ivh elasticsearch-6.2.4.rpm

配置Elasticsearch

1
$ vi /etc/elasticsearch/elasticsearch.yml

配置如下:

1
2
3
bootstrap.memory_lock: true
network.host: localhost
http.port: 9200

启动Elasticsearch

1
2
$ systemctl daemon-reload
$ systemctl enable elasticsearch

设置Elasticsearch开机启动

1
$ systemctl start elasticsearch

安装Kibana

安装Kibana

1
2
$ wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-x86_64.rpm
$ rpm -ivh kibana-6.2.4-x86_64.rpm

配置Kibana

1
$ vi /etc/kibana/kibana.yml

配置内容如下:

1
2
3
server.port: 5601
server.host: "localhost"
elasticsearch.url: "http://localhost:9200"

启动Kibana

1
$ systemctl enable kibana

设置Kibana开机启动

1
$ systemctl start kibana

安装Nginx

安装Nginx

1
2
$ yum install epel-release
$ yum install -y nginx httpd-tools

配置Nginx

1
$ /etc/nginx/conf.d/kibana.conf

内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
server {
listen 80;

server_name node11;

auth_basic "Restricted Access";
auth_basic_user_file /etc/nginx/htpasswd.kibana;

location / {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}

配置密码

1
2
3
4
$ htpasswd -c /etc/nginx/htpasswd.kibana admin
New password:
Re-type new password:
Adding password for user admin

启动Nginx

1
$ systemctl enable nginx

设置Nginx开机启动

1
$ systemctl start nginx

安装Logstash

1
2
3
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.rpm
$ rpm -ivh logstash-6.2.4.rpm
$ vi /etc/pki/tls/openssl.cnf

在[ v3_ca ]增加新的行:

1
2
3
4
[ v3_ca ]

# Server IP Address
subjectAltName = IP: 10.0.2.15

1
2
$ openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt
$ vi /etc/logstash/conf.d/filebeat-input.conf
1
2
3
4
5
6
7
8
input {
beats {
port => 5443
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
1
$ vi /etc/logstash/conf.d/syslog-filter.conf

内容如下:

1
2
3
4
5
6
7
8
input {
beats {
port => 5443
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

1
$ vi /etc/logstash/conf.d/syslog-filter.conf

内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

1
$ vi /etc/logstash/conf.d/output-elasticsearch.conf

内容如下:

1
2
3
4
5
6
7
8
output {
elasticsearch { hosts => ["localhost:9200"]
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

1
2
$ systemctl restart logstash
$ systemctl enable logstash

配置 Logstash 参考 Configuring Logstash

Ubuntu中安装

使用Docker安装

安装Filebeat

CentOS中安装

Ubuntu中安装

Sender

参考

坚持原创技术分享,您的支持将鼓励我继续创作!