Table of Content
1.
2.Elasticsearch Failed to start service in Ubuntu 18.04- Could not create the Java Virtual Machine
Elasticsearch Failed to start service in Ubuntu 18.04-low disk watermark [85%] exceeded
Table of Contents
Elasticsearch 7.14.1
Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-minimal-setup.html to enable security.
Elasticsearch 7.4.2
Open the Elasticsearch configuration file for editing:
sudo vi /etc/elasticsearch/elasticsearch.yml
cat /var/log/elasticsearch/elasticsearch.log
error：
the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured20240322
at java.lang.Thread.run(Thread.java:830) [?:?]
[2024-03-23T06:42:21,205][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [PhA-71jUQdqMH7QboJjoWA][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 2.5gb[6.3%], shards will be relocated away from this node
[2024-03-23T06:42:51,210][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [PhA-71jUQdqMH7QboJjoWA][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 2.5gb[6.3%], shards will be relocated away from this node
[2024-03-23T06:42:51,241][INFO ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2024-03-23T06:43:21,242][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [PhA-71jUQdqMH7QboJjoWA][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 2.5gb[6.3%], shards will be relocated away from this node
[2024-03-23T06:47:15,821][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [PhA-71jUQdqMH7QboJjoWA][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 2.4gb[6.3%], shards will be relocated away from this node
[2024-03-23T06:47:15,955][INFO ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] rerouting shards: [high disk watermark exceeded on one or more nodes]
network.host : 0.0.0.0
将 #cluster.initial_master_nodes: ["node-1", "node-2"]
修改为 cluster.initial_master_nodes: ["node-1"]，记得保存。# ---------------------------------- Network -----------------------------------
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
+network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
@@ -70,6 +71,7 @@ path.logs: /var/log/elasticsearch
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
+cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#访问公网ip出现审计错误：
Valid NEST response built from a successful low level call on POST: /project/doc/_search
# Audit trail of this API call:
- [1] SniffOnStartup: Took: 00:00:00.0360264
- [2] SniffSuccess: Node: http://localhost:9200/ Took: 00:00:00.0310228
- [3] PingSuccess: Node: http://127.0.0.1:9200/ Took: 00:00:00.0115074
- [4] HealthyResponse: Node: http://127.0.0.1:9200/ Took: 00:00:00.1477640
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
to help with troubleshootin
https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/audit-trail.html
解决：原为Nest 6.02 版本，按Nest 7.4.2 语法重写创建索引和查询，问题得到解决。
low disk watermark [85%] exceeded on
error:low disk watermark [85%] exceeded
2021-09-20T00:00:07,336][INFO ][o.e.c.r.a.DiskThresholdMonitor] [iZuf6hn15mpmrtypv9ltogZ]
low disk watermark [85%] exceeded on
[zH0bEPJ9R1C-xKZIcb4BFg][iZuf6hn15mpmrtypv9ltogZ][/var/lib/elasticsearch/nodes/0]
free: 4.5gb[11.4%], replicas will not be assigned to this node20231105 updated
[2023-11-05T22:24:28,220][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [StCKq1TXQ7S2jAZC30DSWg][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 3gb[7.8%], shards will be relocated away from this node
[2023-11-05T22:24:52,131][INFO ][o.e.n.Node ] [iZbp18j0fr5xhywnh03waxZ] stopping ...
[2023-11-05T22:24:52,434][INFO ][o.e.x.w.WatcherService ] [iZbp18j0fr5xhywnh03waxZ] stopping watch service, reason [shutdown initiated]
[2023-11-05T22:24:52,436][INFO ][o.e.x.w.WatcherLifeCycleService] [iZbp18j0fr5xhywnh03waxZ] watcher has stopped and shutdown
[2023-11-05T22:24:52,858][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [iZbp18j0fr5xhywnh03waxZ] [controller/12124] [Main.cc@150] Ml controller exiting
[2023-11-05T22:24:52,869][INFO ][o.e.x.m.p.NativeController] [iZbp18j0fr5xhywnh03waxZ] Native controller process has stopped - no new native processes can be started
[2023-11-05T22:24:52,883][INFO ][o.e.n.Node ] [iZbp18j0fr5xhywnh03waxZ] stopped
[2023-11-05T22:24:52,883][INFO ][o.e.n.Node ] [iZbp18j0fr5xhywnh03waxZ] closing ...
[2023-11-05T22:24:52,909][INFO ][o.e.n.Node ] [iZbp18j0fr5xhywnh03waxZ] closed
1.节点iZuf6hn15mpmrtypv9ltogZ上的磁盘空间使用率超过了85%，空闲4.5gb，占总磁盘空间的11.4%。将不会在该节点上分配副本了。
This is a function of Elasticsearch. Per the Elasticsearch error, all indices on this node will marked read-only。
根据日志信息，提示是因为磁盘超过85%，磁盘空间到达85%时，es会将节点上面的索引标为只读，导致不能写入数据。
解决办法：
In short, Permanent fixes are；
1.Delete unused indices.
2.Merge segment using this SO answer.（https://stackoverflow.com/questions/60204556/optimize-api-for-reducing-the-segments-and-eliminating-es-deleted-docs-not-worki）
3.Attach external disk or increase the disk used by the problematic data node.Solution 1.释放磁盘空间
Solution 2.
# /etc/elasticsearch/elasticsearch.yml
（1） 调整磁盘空间比例(使用百分比或gb)
cluster.routing.allocation.disk.watermark.low: 30g
cluster.routing.allocation.disk.watermark.high: 20gb
（ 2） 关闭阀值设置
cluster.routing.allocation.disk.threshold_enabled: false
More information on this can be found here:https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation
update Nest 6.0.2 to Nest 7.4.2，出现编译错误
SortField not exsit
change SortField to FieldSort
Highlights not exist
change Highlights tos Highlight
foreach (var h in hl.Value.Highlights)
{
WriteLiteral(h);
}
to
foreach (var h in hl.Value)
{
WriteLiteral(h);
}
change field map Name to RelationName
/// <summary>
/// 版本 es7.4.2
/// </summary>
[ElasticsearchType(IdProperty = "Id", Name = "F10FileContentModel")]
public class F10FileContentModel
{
to
/// <summary>
/// 版本 es7.4.2
/// </summary>
[ElasticsearchType(IdProperty = "Id", RelationName = "F10FileContentModel")]
public class F10FileContentModel
{
Elasticsearch Failed to start service in Ubuntu 18.04
step 1: first check elasticsearch.service runnng or not
wget http://localhost:9200
or
systemctl status elasticsearch.service
output
--2021-03-17 01:03:37-- (try:18) http://localhost:9200/
Connecting to localhost (localhost)|127.0.0.1|:9200... failed: Connection refused.
Connecting to localhost (localhost)|::1|:9200... failed: Cannot assign requested address.
Retrying.202407
systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: http://www.elastic.co
step 2:Restart elasticsearch.service
sudo journalctl -fu elasticsearch.service
sudo systemctl restart elasticsearch.serviceoutput
# sudo journalctl -fu elasticsearch.service
-- Logs begin at Sat 2021-09-18 05:27:58 CST. --
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]: Error: Could not create the Java Virtual Machine.
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]: Error: A fatal exception has occurred. Program will exit.
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]: at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]: at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:81)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]: at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:135)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd[1]: Failed to start Elasticsearch.
step 3:check Elasticsearch log file
root@i6:/var/log/elasticsearch//elasticsearch.log
# ls -l
total 516
-rw-r--r-- 1 elasticsearch elasticsearch 0 Dec 2 2019 elasticsearch_audit.json
-rw-r--r-- 1 elasticsearch elasticsearch 782 Jan 27 2020 elasticsearch_deprecation.json
-rw-r--r-- 1 elasticsearch elasticsearch 360 Jan 27 2020 elasticsearch_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 Dec 2 2019 elasticsearch_index_indexing_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch 0 Dec 2 2019 elasticsearch_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 0 Dec 2 2019 elasticsearch_index_search_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch 0 Dec 2 2019 elasticsearch_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch 18879 Mar 17 01:38 elasticsearch.log
-rw-r--r-- 1 elasticsearch elasticsearch 33774 Mar 17 01:38 elasticsearch_server.json
-rw-r--r-- 1 elasticsearch elasticsearch 448349 Mar 17 02:08 gc.log
-rw-r--r-- 1 elasticsearch elasticsearch 1464 Feb 10 23:36 gc.log.00
-rw-r--r-- 1 elasticsearch elasticsearch 3311 Mar 15 01:27 hs_err_pid1909.log
You can also check Elasticsearch log file, /var/log/elasticsearch/elasticsearch.log, for more details.
2021-03-17T01:22:59,227][INFO ][o.e.c.r.a.DiskThresholdMonitor] [iZuf6hn15mpmrtypv9ltogZ]
low disk watermark [85%] exceeded on [d1FehlgHS-uj1BBQ09eIvQ][iZuf6hn15mpmrtypv9ltogZ][/var/lib/elasticsearch/nodes/0]
free: 4.8gb[12.4%], replicas will not be assigned to this node
3. Elasticsearch Failed to start service in Ubuntu 18.04 -Could not create the Java Virtual Machine
step 1：The errors says Could not create the Java Virtual Machine
after using the command systemctl status elasticsearch.service I get the following error log:
# systemctl status elasticsearch.service
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2021-09-19 00:28:51 CST; 5min ago
Docs: https://www.elastic.co
Process: 2078 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 2078 (code=exited, status=1/FAILURE)
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]: Error: Could not create the Java Virtual Machine.
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]: Error: A fatal exception has occurred. Program will exit.
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]: at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]: at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]: at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]: at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsPars
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]: at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.jav
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd[1]: Failed to start Elasticsearch.
Failed to start service in Ubuntu 18.04 -Invalid NEST response built from a successful (404) low level call on HEAD
error on runing：
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
System.Exception: Invalid NEST response built from a successful (404)
low level call on POST: /wwwiaspnetcoreindex/_search?typed_keys=true
# Audit trail of this API call:
- [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.0061855
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
Answers：
I think the message you see tells you everything.
System.Exception: Invalid NEST response built from a successful (404)
low level call on POST: /wwwiaspnetcoreindex/_search?typed_keys=true1.response built from a successful - elasticsearch call was successful,
2.but status 404 (not exists) was returned and mapped to existsResponse.Exists which is false in this case. There is no additional request/response information attached, this is why you see:
Request:
Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.
Response:
Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.3. the HTTP Response from ES tells me that the index does not exist (404).
解决办法：
添加判断索引文件是否存在的代码
20231012
故障：
Elasticsearch 运行中停止
排查
1.日志
root@i6:cat /var/log/elasticsearch//elasticsearch.log
[2023-10-12T02:17:48,458][DEBUG][o.e.a.s.TransportSearchAction] [iZbp18j0fr5xhywnh03waxZ] [wwwiaspnetcoreindex][0], node[StCKq1TXQ7S2jAZC30DSWg], [P], s[STARTED], a[id=838ONbmVTQyM3MZASpqT6Q]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[wwwiaspnetcoreindex], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"from":0,"size":10,"query":{"query_string":{"query":"家庭1000M网络组网方案-FTTR（光纤部署到房间）","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},"sort":[{"createdOn":{"order":"desc"}}],"highlight":{"pre_tags":["<b>"],"post_tags":["</b>"],"fields":{"Body":{"pre_tags":["<font color =\"red\">"],"post_tags":["</font>"]},"Title":{"pre_tags":["<font color =\"red\">"],"post_tags":["</font>"]}}}}}]
org.elasticsearch.transport.RemoteTransportException: [iZbp18j0fr5xhywnh03waxZ][127.0.0.1:9300][indices:data/read/search[phase/query]]
...
[2023-10-12T02:17:48,459][DEBUG][o.e.a.s.TransportSearchAction] [iZbp18j0fr5xhywnh03waxZ] All shards failed for phase: [query]
org.elasticsearch.index.query.QueryShardException: No mapping found for [createdOn] in order to sort on
...
[2023-10-12T02:29:20,461][INFO ][o.e.n.Node ] [iZbp18j0fr5xhywnh03waxZ] stopping ...
[2023-10-12T02:29:20,582][INFO ][o.e.x.w.WatcherService ] [iZbp18j0fr5xhywnh03waxZ] stopping watch service, reason [shutdown initiated]
[2023-10-12T02:29:20,583][INFO ][o.e.x.w.WatcherLifeCycleService] [iZbp18j0fr5xhywnh03waxZ] watcher has stopped and shutdown
[2023-10-12T02:29:20,781][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [iZbp18j0fr5xhywnh03waxZ] [controller/1846] [Main.cc@150] Ml controller exiting
[2023-10-12T02:29:20,784][INFO ][o.e.x.m.p.NativeController] [iZbp18j0fr5xhywnh03waxZ] Native controller process has stopped - no new native processes can be started
[2023-10-12T02:29:20,839][INFO ][o.e.n.Node ] [iZbp18j0fr5xhywnh03waxZ] stopped
[2023-10-12T02:29:20,841][INFO ][o.e.n.Node ] [iZbp18j0fr5xhywnh03waxZ] closing ...
[2023-10-12T02:29:20,862][INFO ][o.e.n.Node ] [iZbp18j0fr5xhywnh03waxZ] closed
程序代码运行出错造成
解决办法 重启 Elasticsearch解决
20231120
[2023-11-21T02:58:12,667][ERROR][o.e.b.Bootstrap ] [iZbp18j0fr5xhywnh03waxZ] Exception
java.lang.IllegalArgumentException: Plugin [analysis-ik] was built for Elasticsearch version 7.4.2 but version 7.17.15 is running
Useful links
1.low disk watermark [??%] exceeded on
https://stackoverflow.com/questions/33369955/low-disk-watermark-exceeded-on