Elasticsearch 403 FORBIDDEN Index read-only Error 처리

Elastic Stack을 운영중 data node 용량 문제로 아래와 같은 에러 로그를 발견.

해결책에 대하여 알아보자.

 

 

org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

 

Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

 

 

 

1. Elasticsearch Error Logs

---

 

elastic master node log files

dz-secops-2021-04-01.log:org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
dz-secops-2021-04-01.log:Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];

 

 

 

 

 

2. Elasticsearch Error 원인 (Cause)

---

elastic data node 환경 설정 (elasticsearch.yml)

 

 

thread_pool.index.queue_size: 1200
thread_pool.get.queue_size: 20000
thread_pool.search.queue_size: 20000
thread_pool.bulk.queue_size: 20000
indices.memory.index_buffer_size: 30%

indices.fielddata.cache.size : 82%
indices.breaker.fielddata.limit: 84%
indices.breaker.total.limit: 85%
indices.breaker.request.limit: 70%
#http.cors.enabled: true
cluster.routing.allocation.disk.threshold_enabled : false
cluster.routing.allocation.disk.watermark.low: 94%
cluster.routing.allocation.disk.watermark.high: 95%  

 

 

data node 가 디스크 용량이 cluster.routing.allocation.disk.watermark.high: 95%  이상이면 forbidden 되며,

read only만 할수 있게됩니다.

 

 

3. 해결책 (Solution)

---

 

Kibana -> Dev Tools -> Console -> Query 

 

PUT _all/_settings
{
    "index": {
        "blocks": {
        "read_only_allow_delete": "false"
        }
    }
}

 

Kibana Dev Tools Console

 

 

 

 

 

Linux Shell 에서 curl 을 사용하여 Elasticsearch API 포트로 명령어를 직접 날려도 됩니다.

 

 

 

 

 

TIP 더보기 (..See more)

https://rootkey.tistory.com/category/SYSTEM/TIP

'SYSTEM/TIP' 카테고리의 글 목록