ELK filebeat 설치(RPM)

설치환경

# cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)

# java -version
openjdk version "1.8.0_282"
OpenJDK Runtime Environment (build 1.8.0_282-08)
OpenJDK 64-Bit Server VM (build 25.282-b08, mixed mode)

호환성확인
www.elastic.co/kr/support/matrix

Elastic Support Matrix | Elasticsearch

# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

1. Elasticsearch(엘라스틱서치) 설치

# vi /etc/yum.repos.d/elasticsearch.repo 아래내용 추가
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=0
autorefresh=1
type=rpm-md

# yum -y install --enablerepo=elasticsearch elasticsearch

설정변경

# vi /etc/elasticsearch/elasticsearch.yml

cluster.name: elasticsearch

node.name: node-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 0.0.0.0

http.port: 9200

discovery.seed_hosts: ["127.0.0.1", "[::1]"]

cluster.initial_master_nodes: ["node-1"]

엘라스틱서치 서비스 시작 및 확인

# systemctl start elasticsearch

# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: active (running) since 금 2021-04-02 16:09:01 KST; 1min 39s ago
     Docs: https://www.elastic.co
 Main PID: 72982 (java)
    Tasks: 56
   Memory: 4.1G
   CGroup: /system.slice/elasticsearch.service
           ├─72982 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xs...
           └─73173 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

# curl http://localhost:9200/
{
  "name" : "darksharavim.tistory.com",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "WddOCn74RZ-dx231hH0k2w",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

2. Logstash(로그스테이시) 설치

# vi /etc/yum.repos.d/logstash.repo
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

# yum -y install logstash

설정

# vi /etc/logstash/conf.d/logstash.conf

input {
  beats {
    port => 5044
    host => "localhost"      # 모든IP일 경우 0.0.0.0
  }
}

filter {
        grok {
                match => { "message" => "%{IPORHOST:clientip} (?:-|) (?:-|) \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:httpMethod} %{URIPATH:uri}%{GREEDYDATA}(?: HTTP/%{NUMBER})?|-)\" %{NUMBER:responseCode} (?:-|%{NUMBER:responseBytes})" } # 받아온 로그를 파싱 
        }

        date {
                match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]     # fields의 시간을 로그발생 시간으로 지정
        }

        geoip {
                source => "clientip"                                                   # geoip에 사용할 정보를  clientip에서 가져옴
        }
}

output {
	elasticsearch { 
		hosts => [ "localhost:9200" ] # elasticsearch 정보
	}    

	stdout {
		codec => rubydebug	# filebeat에서 받은 정보를 json 형식으로 출력
	}                       
        
	file {
		path => "/var/log/logstash/logstash.log"
		codec => rubydebug
	}
}

www.elastic.co/guide/en/elastic-stack-get-started/7.5/get-started-elastic-stack.html

Getting started with the Elastic Stack | Getting Started [7.5] | Elastic

로그스테이시 시작 및 확인

# systemctl start logstash

# systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since 금 2021-04-02 16:48:57 KST; 1s ago
 Main PID: 73917 (java)
   CGroup: /system.slice/logstash.service
           └─73917 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djav...

 4월 02 16:48:57 darksharavim.com systemd[1]: Started logstash.
 4월 02 16:48:57 darksharavim.com logstash[73917]: Using bundled JDK: /usr/share/logstash/jdk
 4월 02 16:48:57 darksharavim.com logstash[73917]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be... release.
Hint: Some lines were ellipsized, use -l to show in full.

로그스테이시 로그 확인

[2021-04-06T17:30:13,138][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x38b72a80 run>"}

3. Kibana(키바나) 설치

# vi /etc/yum.repos.d/kibana.repo
[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

# yum -y install kibana

설정변경

# vi /etc/kibana/kibana.yml
server.port: 5601
server.host: "localhost"     # 모든ip에서 접근할려면 0.0.0.0으로 변경
elasticsearch.hosts: ["http://localhost:9200"]

키바나 서비스 시작 및 확인

# systemctl start kibana

# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled)
   Active: active (running) since 금 2021-04-02 16:30:09 KST; 2s ago
     Docs: https://www.elastic.co
 Main PID: 73427 (node)
    Tasks: 11
   Memory: 120.2M
   CGroup: /system.slice/kibana.service
           └─73427 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest="/var/log/kibana/kibana.log" --pid.file="/run/kibana/kiba...

 4월 02 16:30:09 darksharavim.com systemd[1]: Started Kibana.

키바나 웹접속
http://[키바나설치서버ip]:5601

4. filebeat(파일비트) 설치

# curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.12.0-x86_64.rpm

# rpm -vi filebeat-7.12.0-x86_64.rpm

filebeat 모듈활성화 및 확인
filebeat 설정법이 2가지 존재
1. 모듈활성화
2. yml설정

# filebeat modules enable apache
Enabled apache

# filebeat modules list | more
Enabled:
system

Disabled:
activemq
....

설정변경

# vi /etc/filebeat/filebeat.yml

주석처리
#output.elasticsearch:
  #hosts: ["localhost:9200"]

주석제거
output.logstash:
  hosts: ["localhost:5044"]

파일비트 시작 및 확인

# systemctl start filebeat

# systemctl status filebeat
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
   Active: active (running) since 토 2021-04-03 18:41:13 KST; 7min ago
     Docs: https://www.elastic.co/products/beats/filebeat
 Main PID: 3906 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─3906 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path...

키바나설정

1. Manage버튼 클릭

2. Index Patterns클릭

3. Create index pattern 클릭

4. filebeat-* 입력

5. Time field @timestamp선택 후 create index pattern 클릭

6. Discover화면결과