受欢迎的博客标签

Elasticsearch | Error:Elasticsearch on Ubuntu(3) -Error and Configure Elasticsearch 7.x on Ubuntu

Published

Table of Content

1.

2.Elasticsearch Failed to start service in Ubuntu 18.04- Could not create the Java Virtual Machine

 

Elasticsearch Failed to start service in Ubuntu 18.04-low disk watermark [85%] exceeded

 

3. Elasticsearch 7.4.2 Failed to start service in Ubuntu 18.04 -Invalid NEST response built from a successful (404) low level call on HEAD

 

 

Elasticsearch 7.14.1

 

Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-minimal-setup.html to enable security.

 

 

 

Elasticsearch 7.4.2

 

Open the Elasticsearch configuration file for editing:

sudo vi /etc/elasticsearch/elasticsearch.yml

 

cat /var/log/elasticsearch/elasticsearch.log

 

error:

the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

20240322

	at java.lang.Thread.run(Thread.java:830) [?:?]
[2024-03-23T06:42:21,205][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [PhA-71jUQdqMH7QboJjoWA][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 2.5gb[6.3%], shards will be relocated away from this node
[2024-03-23T06:42:51,210][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [PhA-71jUQdqMH7QboJjoWA][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 2.5gb[6.3%], shards will be relocated away from this node
[2024-03-23T06:42:51,241][INFO ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] rerouting shards: [high disk watermark exceeded on one or more nodes]
[2024-03-23T06:43:21,242][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [PhA-71jUQdqMH7QboJjoWA][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 2.5gb[6.3%], shards will be relocated away from this node
[2024-03-23T06:47:15,821][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [PhA-71jUQdqMH7QboJjoWA][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 2.4gb[6.3%], shards will be relocated away from this node
[2024-03-23T06:47:15,955][INFO ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] rerouting shards: [high disk watermark exceeded on one or more nodes]

 

network.host : 0.0.0.0
将 #cluster.initial_master_nodes: ["node-1", "node-2"] 
修改为 cluster.initial_master_nodes: ["node-1"],记得保存。
# ---------------------------------- Network -----------------------------------
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
+network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
	@@ -70,6 +71,7 @@ path.logs: /var/log/elasticsearch
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
+cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#

访问公网ip出现审计错误:

Valid NEST response built from a successful low level call on POST: /project/doc/_search
# Audit trail of this API call:
 - [1] SniffOnStartup: Took: 00:00:00.0360264
 - [2] SniffSuccess: Node: http://localhost:9200/ Took: 00:00:00.0310228
 - [3] PingSuccess: Node: http://127.0.0.1:9200/ Took: 00:00:00.0115074
 - [4] HealthyResponse: Node: http://127.0.0.1:9200/ Took: 00:00:00.1477640
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
to help with troubleshootin


https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/audit-trail.html

解决:原为Nest 6.02 版本,按Nest 7.4.2 语法重写创建索引和查询,问题得到解决。

low disk watermark [85%] exceeded on

error:low disk watermark [85%] exceeded

2021-09-20T00:00:07,336][INFO ][o.e.c.r.a.DiskThresholdMonitor] [iZuf6hn15mpmrtypv9ltogZ]
low disk watermark [85%] exceeded on
 [zH0bEPJ9R1C-xKZIcb4BFg][iZuf6hn15mpmrtypv9ltogZ][/var/lib/elasticsearch/nodes/0] 
free: 4.5gb[11.4%], replicas will not be assigned to this node

20231105 updated

[2023-11-05T22:24:28,220][WARN ][o.e.c.r.a.DiskThresholdMonitor] [iZbp18j0fr5xhywnh03waxZ] high disk watermark [90%] exceeded on [StCKq1TXQ7S2jAZC30DSWg][iZbp18j0fr5xhywnh03waxZ][/var/lib/elasticsearch/nodes/0] free: 3gb[7.8%], shards will be relocated away from this node
[2023-11-05T22:24:52,131][INFO ][o.e.n.Node               ] [iZbp18j0fr5xhywnh03waxZ] stopping ...
[2023-11-05T22:24:52,434][INFO ][o.e.x.w.WatcherService   ] [iZbp18j0fr5xhywnh03waxZ] stopping watch service, reason [shutdown initiated]
[2023-11-05T22:24:52,436][INFO ][o.e.x.w.WatcherLifeCycleService] [iZbp18j0fr5xhywnh03waxZ] watcher has stopped and shutdown
[2023-11-05T22:24:52,858][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [iZbp18j0fr5xhywnh03waxZ] [controller/12124] [Main.cc@150] Ml controller exiting
[2023-11-05T22:24:52,869][INFO ][o.e.x.m.p.NativeController] [iZbp18j0fr5xhywnh03waxZ] Native controller process has stopped - no new native processes can be started
[2023-11-05T22:24:52,883][INFO ][o.e.n.Node               ] [iZbp18j0fr5xhywnh03waxZ] stopped
[2023-11-05T22:24:52,883][INFO ][o.e.n.Node               ] [iZbp18j0fr5xhywnh03waxZ] closing ...
[2023-11-05T22:24:52,909][INFO ][o.e.n.Node               ] [iZbp18j0fr5xhywnh03waxZ] closed

 

1.节点iZuf6hn15mpmrtypv9ltogZ上的磁盘空间使用率超过了85%,空闲4.5gb,占总磁盘空间的11.4%。将不会在该节点上分配副本了。

This is a function of Elasticsearch. Per the Elasticsearch error, all indices on this node will marked read-only。

根据日志信息,提示是因为磁盘超过85%,磁盘空间到达85%时,es会将节点上面的索引标为只读,导致不能写入数据。

解决办法:

In short, Permanent fixes are;

1.Delete unused indices.
2.Merge segment using this SO answer.(https://stackoverflow.com/questions/60204556/optimize-api-for-reducing-the-segments-and-eliminating-es-deleted-docs-not-worki)
3.Attach external disk or increase the disk used by the problematic data node.
Solution 1.释放磁盘空间

Solution 2.

# /etc/elasticsearch/elasticsearch.yml

 (1) 调整磁盘空间比例(使用百分比或gb)
        cluster.routing.allocation.disk.watermark.low: 30g
        cluster.routing.allocation.disk.watermark.high: 20gb
  ( 2) 关闭阀值设置
        cluster.routing.allocation.disk.threshold_enabled: false

More information on this can be found here:https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation

 

 

 

update Nest 6.0.2 to Nest 7.4.2,出现编译错误

SortField  not exsit

change SortField to FieldSort

Highlights not exist

change Highlights tos Highlight

 foreach (var h in hl.Value.Highlights)
                {
                    WriteLiteral(h);

                }

to
foreach (var h in hl.Value)
                {
                    WriteLiteral(h);

                }

 

change field map Name to RelationName

    /// <summary>
    /// 版本 es7.4.2
    /// </summary>
    [ElasticsearchType(IdProperty = "Id", Name = "F10FileContentModel")]
    public class F10FileContentModel
    {


to


    /// <summary>
    /// 版本 es7.4.2
    /// </summary>
    [ElasticsearchType(IdProperty = "Id", RelationName = "F10FileContentModel")]
    public class F10FileContentModel
    {

 

Elasticsearch Failed to start service in Ubuntu 18.04

step 1: first check   elasticsearch.service runnng or not

 wget http://localhost:9200
or

systemctl status elasticsearch.service

output

--2021-03-17 01:03:37--  (try:18)  http://localhost:9200/
Connecting to localhost (localhost)|127.0.0.1|:9200... failed: Connection refused.
Connecting to localhost (localhost)|::1|:9200... failed: Cannot assign requested address.
Retrying.

step 2:Restart elasticsearch.service

sudo journalctl -fu elasticsearch.service
sudo systemctl restart elasticsearch.service

output

# sudo journalctl -fu elasticsearch.service
-- Logs begin at Sat 2021-09-18 05:27:58 CST. --
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]: Error: Could not create the Java Virtual Machine.
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]: Error: A fatal exception has occurred. Program will exit.
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]:         at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]:         at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:81)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]:         at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:135)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[1671]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:86)
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 19 00:01:12 iZuf6hn15mpmrtypv9ltogZ systemd[1]: Failed to start Elasticsearch.

 

step 3:check Elasticsearch log file

 

root@i6:/var/log/elasticsearch//elasticsearch.log
# ls -l
total 516
-rw-r--r-- 1 elasticsearch elasticsearch      0 Dec  2  2019 elasticsearch_audit.json
-rw-r--r-- 1 elasticsearch elasticsearch    782 Jan 27  2020 elasticsearch_deprecation.json
-rw-r--r-- 1 elasticsearch elasticsearch    360 Jan 27  2020 elasticsearch_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch      0 Dec  2  2019 elasticsearch_index_indexing_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch      0 Dec  2  2019 elasticsearch_index_indexing_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch      0 Dec  2  2019 elasticsearch_index_search_slowlog.json
-rw-r--r-- 1 elasticsearch elasticsearch      0 Dec  2  2019 elasticsearch_index_search_slowlog.log
-rw-r--r-- 1 elasticsearch elasticsearch  18879 Mar 17 01:38 elasticsearch.log
-rw-r--r-- 1 elasticsearch elasticsearch  33774 Mar 17 01:38 elasticsearch_server.json
-rw-r--r-- 1 elasticsearch elasticsearch 448349 Mar 17 02:08 gc.log
-rw-r--r-- 1 elasticsearch elasticsearch   1464 Feb 10 23:36 gc.log.00
-rw-r--r-- 1 elasticsearch elasticsearch   3311 Mar 15 01:27 hs_err_pid1909.log

You can also check Elasticsearch log file, /var/log/elasticsearch/elasticsearch.log, for more details.

2021-03-17T01:22:59,227][INFO ][o.e.c.r.a.DiskThresholdMonitor] [iZuf6hn15mpmrtypv9ltogZ]
 low disk watermark [85%] exceeded on [d1FehlgHS-uj1BBQ09eIvQ][iZuf6hn15mpmrtypv9ltogZ][/var/lib/elasticsearch/nodes/0]
 free: 4.8gb[12.4%], replicas will not be assigned to this node

 

3. Elasticsearch Failed to start service in Ubuntu 18.04 -Could not create the Java Virtual Machine

 

step 1:The errors says Could not create the Java Virtual Machine

after using the command systemctl status elasticsearch.service I get the following error log:

# systemctl status elasticsearch.service

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2021-09-19 00:28:51 CST; 5min ago
     Docs: https://www.elastic.co
  Process: 2078 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 2078 (code=exited, status=1/FAILURE)

Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]: Error: Could not create the Java Virtual Machine.
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]: Error: A fatal exception has occurred. Program will exit.
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]:         at org.elasticsearch.tools.launchers.JvmOption.flagsFinal(JvmOption.java:119)
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]:         at org.elasticsearch.tools.launchers.JvmOption.findFinalOptions(JvmOption.java:
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]:         at org.elasticsearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:38
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsPars
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd-entrypoint[2078]:         at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.jav
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Sep 19 00:28:51 iZuf6hn15mpmrtypv9ltogZ systemd[1]: Failed to start Elasticsearch.

 

Failed to start service in Ubuntu 18.04 -Invalid NEST response built from a successful (404) low level call on HEAD

error on runing:

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
      An unhandled exception has occurred while executing the request.
      System.Exception: Invalid NEST response built from a successful (404)
 low level call on POST: /wwwiaspnetcoreindex/_search?typed_keys=true
      # Audit trail of this API call:
       - [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.0061855
      # Request:
      <Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
      # Response:
      <Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

Answers:

 

I think the message you see tells you everything.

System.Exception: Invalid NEST response built from a successful (404)
 low level call on POST: /wwwiaspnetcoreindex/_search?typed_keys=true

1.response built from a successful - elasticsearch call was successful,

2.but status 404 (not exists) was returned and mapped to existsResponse.Exists which is false in this case. There is no additional request/response information attached, this is why you see:

Request:
Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.
Response:
Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.

3. the HTTP Response from ES tells me that the index does not exist (404).

 

解决办法:

添加判断索引文件是否存在的代码

 

20231012

故障:

Elasticsearch  运行中停止

排查

1.日志

root@i6:cat /var/log/elasticsearch//elasticsearch.log

[2023-10-12T02:17:48,458][DEBUG][o.e.a.s.TransportSearchAction] [iZbp18j0fr5xhywnh03waxZ] [wwwiaspnetcoreindex][0], node[StCKq1TXQ7S2jAZC30DSWg], [P], s[STARTED], a[id=838ONbmVTQyM3MZASpqT6Q]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[wwwiaspnetcoreindex], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"from":0,"size":10,"query":{"query_string":{"query":"家庭1000M网络组网方案-FTTR(光纤部署到房间)","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}},"sort":[{"createdOn":{"order":"desc"}}],"highlight":{"pre_tags":["<b>"],"post_tags":["</b>"],"fields":{"Body":{"pre_tags":["<font color =\"red\">"],"post_tags":["</font>"]},"Title":{"pre_tags":["<font color =\"red\">"],"post_tags":["</font>"]}}}}}]
org.elasticsearch.transport.RemoteTransportException: [iZbp18j0fr5xhywnh03waxZ][127.0.0.1:9300][indices:data/read/search[phase/query]]
...
[2023-10-12T02:17:48,459][DEBUG][o.e.a.s.TransportSearchAction] [iZbp18j0fr5xhywnh03waxZ] All shards failed for phase: [query]
org.elasticsearch.index.query.QueryShardException: No mapping found for [createdOn] in order to sort on
...
[2023-10-12T02:29:20,461][INFO ][o.e.n.Node               ] [iZbp18j0fr5xhywnh03waxZ] stopping ...
[2023-10-12T02:29:20,582][INFO ][o.e.x.w.WatcherService   ] [iZbp18j0fr5xhywnh03waxZ] stopping watch service, reason [shutdown initiated]
[2023-10-12T02:29:20,583][INFO ][o.e.x.w.WatcherLifeCycleService] [iZbp18j0fr5xhywnh03waxZ] watcher has stopped and shutdown
[2023-10-12T02:29:20,781][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [iZbp18j0fr5xhywnh03waxZ] [controller/1846] [Main.cc@150] Ml controller exiting
[2023-10-12T02:29:20,784][INFO ][o.e.x.m.p.NativeController] [iZbp18j0fr5xhywnh03waxZ] Native controller process has stopped - no new native processes can be started
[2023-10-12T02:29:20,839][INFO ][o.e.n.Node               ] [iZbp18j0fr5xhywnh03waxZ] stopped
[2023-10-12T02:29:20,841][INFO ][o.e.n.Node               ] [iZbp18j0fr5xhywnh03waxZ] closing ...
[2023-10-12T02:29:20,862][INFO ][o.e.n.Node               ] [iZbp18j0fr5xhywnh03waxZ] closed

程序代码运行出错造成

解决办法 重启 Elasticsearch解决

 

20231120

[2023-11-21T02:58:12,667][ERROR][o.e.b.Bootstrap          ] [iZbp18j0fr5xhywnh03waxZ] Exception
java.lang.IllegalArgumentException: Plugin [analysis-ik] was built for Elasticsearch version 7.4.2 but version 7.17.15 is running

 

 

Useful links

1.low disk watermark [??%] exceeded on

https://stackoverflow.com/questions/33369955/low-disk-watermark-exceeded-on