[elasticsearch]bootstrap check failure

엘라스틱서치 아래와 같이 설정 변경 후 구동시 에러발생

[darksharavim.tistory.com]vi /etc/elasticsearch/elasticsearch.yml
network.host: _site_

[2022-06-16T16:39:36,037][ERROR][o.e.b.Bootstrap          ] [node-1] node validation exception
[1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: memory locking requested for elasticsearch process but memory is not locked

 

network.host: <ip 주소>


Elasticsearch가 실행되는 서버의 ip 주소입니다.


디폴트는 루프백(127.0.0.1) 입니다.

주석 처리 되어 있거나 루프백인 경우에는 Elasticsearch 노드가 개발 모드로 실행이 됩니다.

만약에 이 설정을 실제 IP 주소로 변경하게 되면 그 때부터는 운영 모드로 실행이 되며 노드를 시작할 때 부트스트랩 체크를 하게 됩니다.

network.host는 서버의 내/외부 주소를 모두 지정하는데 만약 내부망에서 사용하는 주소와 외부망에서 접근하는 주소를 다르게 설정하고자 하면 아래의 값 들을 이용해서 구분이 가능합니다.

network.bind_host : 내부망

network.publish_host : 외부망

그리고 network.host 설정에 사용되는 특별한 변수값이 있는데 다음과 같습니다.

• _local_ : 루프백 주소 127.0.0.1 과 같습니다. 디폴트로 설정되어 있습니다.
• _site_ : 로컬 네트워크 주소로 설정됩니다. 실제로 클러스터링 구성 시 주로 설정하는 값입니다.
• _global_ : 네트워크 외부에서 바라보는 주소로 설정합니다.

실제로 클러스터를 구성할 때 설정을 network.host: _site_ 로 해 놓으면 서버의 네트워크 주소가 바뀌어도 설정 파일은 변경하지 않아도 되기 때문에 편리합니다.

ulimit설정 확인 및 변경

[darksharavim.tistory.com]ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30793
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30793
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[darksharavim.tistory.com]vi /etc/security/limits.conf
# 아래내용 추가
*       soft    nofile  65536
*       hard    nofile  65536
*       soft    memlock unlimited
*       hard    memlock unlimited

[darksharavim.tistory.com]ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 30793
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 30793
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[darksharavim.tistory.com]vi /etc/sysctl.conf
# 아래내용추가
fs.file-max = 6815744
vm.max_map_count = 262144

[darksharavim.tistory.com]sysctl -p
fs.file-max = 6815744
vm.max_map_count = 262144

위와 같이 설정하고 해봤는데도 계속 실패!

각각의 서비스에서 적용되는 limit설정값이 다른것으로 확인.
아래와 같이 옵션 추가해주고 구동하니 정상!

[darksharavim.tistory.com]vi /usr/lib/systemd/system/elasticsearch.service
# [service]라인 하위에 아래내용추가
LimitMEMLOCK=infinity

[darksharavim.tistory.com]systemctl daemon-reload

[darksharavim.tistory.com]systemctl start elasticsearch

[darksharavim.tistory.com]systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-06-17 11:31:59 KST; 11min ago
     Docs: https://www.elastic.co
 Main PID: 3964 (java)
    Tasks: 61 (limit: 100878)
   Memory: 8.4G
   CGroup: /system.slice/elasticsearch.service
           ├─3964 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networka>
           └─4256 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

 6월 17 11:31:50 localhost.localdomain systemd[1]: Starting Elasticsearch...
 6월 17 11:31:59 localhost.localdomain systemd[1]: Started Elasticsearch.
 

rockylinux에서 default limit값확인방법

[darksharavim.tistory.com]systemctl show | grep Limit
DefaultStartLimitIntervalUSec=10s
DefaultStartLimitBurst=5
DefaultLimitCPU=infinity
DefaultLimitCPUSoft=infinity
DefaultLimitFSIZE=infinity
DefaultLimitFSIZESoft=infinity
DefaultLimitDATA=infinity
DefaultLimitDATASoft=infinity
DefaultLimitSTACK=infinity
DefaultLimitSTACKSoft=8388608
DefaultLimitCORE=infinity
DefaultLimitCORESoft=0
DefaultLimitRSS=infinity
DefaultLimitRSSSoft=infinity
DefaultLimitNOFILE=262144
DefaultLimitNOFILESoft=1024
DefaultLimitAS=infinity
DefaultLimitASSoft=infinity
DefaultLimitNPROC=63049
DefaultLimitNPROCSoft=63049
DefaultLimitMEMLOCK=65536
DefaultLimitMEMLOCKSoft=65536
DefaultLimitLOCKS=infinity
DefaultLimitLOCKSSoft=infinity
DefaultLimitSIGPENDING=63049
DefaultLimitSIGPENDINGSoft=63049
DefaultLimitMSGQUEUE=819200
DefaultLimitMSGQUEUESoft=819200
DefaultLimitNICE=0
DefaultLimitNICESoft=0
DefaultLimitRTPRIO=0
DefaultLimitRTPRIOSoft=0
DefaultLimitRTTIME=infinity
DefaultLimitRTTIMESoft=infinity

https://www.elastic.co/guide/en/elasticsearch/reference/8.4/max-number-of-threads.html

Number of threads | Elasticsearch Guide [8.4] | Elastic

https://www.elastic.co/guide/en/elasticsearch/reference/8.4/setting-system-settings.html

Configuring system settings | Elasticsearch Guide [8.4] | Elastic

1

https://www.elastic.co/guide/en/elasticsearch/reference/master/bootstrap-checks.html

Bootstrap Checks | Elasticsearch Guide [master] | Elastic

https://stackoverflow.com/questions/51382869/unable-to-lock-jvm-memory-error-12-reason-cannot-allocate-memory-elasticsear

Unable to lock JVM Memory: error=12, reason=Cannot allocate memory - Elasticsearch 6.2.4