OB断电后服务器重启没有自动启动,手工启动后报错

【 使用环境 】测试环境
【 OB or 其他组件 】全部
【 使用版本 】
【问题描述】清晰明确描述问题
【复现路径】问题出现前后相关操作
【附件及日志】
服务器断电后重启,oceanbase server不会自动重启, obd cluster list显示是running,但是obd cluster display xxx显示oceanbase-ce, obporxy-ce等都是是stopped。
通过手工obd cluster restart xx重启后报错:
Get local repositories and plugins ok
Load cluster param plugin ok
Open ssh connection ok
Cluster status check ok
Search plugins ok
Load cluster param plugin ok
Cluster status check ok
Check before start observer ok
Check before start obproxy ok
Check before start obagent ok
Check before start ocp-express ok
Start observer ok
observer program health check ok
Connect to observer 172.16.0.58:2881 ok
Start obproxy ok
obproxy program health check ok
Connect to obproxy ok
Start obagent ok
[ERROR] failed to start 172.16.0.58 obagent.
[ERROR] obagent start failed

[ERROR] OBD-1005: Some of the servers in the cluster have been stopped

到服务器上看一下是不是observer和obproxy都已经启动了,我看这一块的检查已经过了

然后obagent 启动报错的原因,查一下 obd display-trace xx的内容

OceanBase 生态组件重启方式
可以参考看下

经重新查询 obd cluster display xxx,发现obagent是正常的,估计上面的错误是判断超时导致判断不准确,然后中断后,ocp-express就没有继续执行启动

可以看下ocp-express进程是否正常。 ps -ef|grep ocp
应该是ocpexpress没起来 可以obd cluster restart name -c ocp-express 启动下看看

也不行,运行不起来,数据库连接不上

obd display-trace 可以看下有什么信息。

[root@node58 ~]# obd display-trace
Usage: obd display-trace [options]

Options:
-h, --help Show help and exit.
-v, --verbose Activate verbose output.
参数是什么

image
obd display-trace ip 这个命令

[root@node58 ~]# obd display-trace 2b90751a-b8e8-11ee-8eb5-b499bab66c38
[2024-01-22 13:36:19.788] [DEBUG] - cmd: [‘wisdombase’]
[2024-01-22 13:36:19.788] [DEBUG] - opts: {‘servers’: None, ‘components’: ‘ocp-express’, ‘with_parameter’: None}
[2024-01-22 13:36:19.788] [DEBUG] - mkdir /root/.obd/lock/
[2024-01-22 13:36:19.789] [DEBUG] - unknown lock mode
[2024-01-22 13:36:19.789] [DEBUG] - try to get share lock /root/.obd/lock/global
[2024-01-22 13:36:19.789] [DEBUG] - share lock /root/.obd/lock/global, count 1
[2024-01-22 13:36:19.790] [DEBUG] - Get Deploy by name
[2024-01-22 13:36:19.790] [DEBUG] - mkdir /root/.obd/cluster/
[2024-01-22 13:36:19.790] [DEBUG] - mkdir /root/.obd/config_parser/
[2024-01-22 13:36:19.791] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_wisdombase
[2024-01-22 13:36:19.791] [DEBUG] - exclusive lock /root/.obd/lock/deploy_wisdombase, count 1
[2024-01-22 13:36:19.805] [DEBUG] - Deploy status judge
[2024-01-22 13:36:19.806] [INFO] Get local repositories and plugins
[2024-01-22 13:36:19.863] [DEBUG] - mkdir /root/.obd/repository
[2024-01-22 13:36:19.863] [DEBUG] - Get local repository oceanbase-ce-4.2.0.0-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:19.864] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2024-01-22 13:36:19.864] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 1
[2024-01-22 13:36:19.869] [DEBUG] - Get local repository obproxy-ce-4.1.0.0-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:19.869] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 2
[2024-01-22 13:36:19.873] [DEBUG] - Get local repository obagent-1.3.1-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:19.874] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 3
[2024-01-22 13:36:19.878] [DEBUG] - Get local repository ocp-express-1.0.1-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:19.878] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 4
[2024-01-22 13:36:19.883] [DEBUG] - Searching restart plugin for components …
[2024-01-22 13:36:19.883] [DEBUG] - Searching restart plugin for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:19.884] [DEBUG] - mkdir /root/.obd/plugins
[2024-01-22 13:36:19.885] [DEBUG] - Found for oceanbase-ce-py_script_restart-4.0.0.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:19.886] [DEBUG] - Searching restart plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:19.886] [DEBUG] - Found for obproxy-ce-py_script_restart-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:19.886] [DEBUG] - Searching restart plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:19.887] [DEBUG] - Found for obagent-py_script_restart-1.3.0 for obagent-1.3.1
[2024-01-22 13:36:19.887] [DEBUG] - Searching restart plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:19.887] [DEBUG] - Found for ocp-express-py_script_restart-1.0 for ocp-express-1.0.1
[2024-01-22 13:36:19.887] [DEBUG] - Searching reload plugin for components …
[2024-01-22 13:36:19.888] [DEBUG] - Searching reload plugin for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:19.888] [DEBUG] - Found for oceanbase-ce-py_script_reload-3.1.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:19.889] [DEBUG] - Searching reload plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:19.889] [DEBUG] - Found for obproxy-ce-py_script_reload-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:19.889] [DEBUG] - Searching reload plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:19.890] [DEBUG] - Found for obagent-py_script_reload-1.3.0 for obagent-1.3.1
[2024-01-22 13:36:19.890] [DEBUG] - Searching reload plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:19.890] [DEBUG] - Found for ocp-express-py_script_reload-1.0 for ocp-express-1.0.1
[2024-01-22 13:36:19.890] [DEBUG] - Searching start_check plugin for components …
[2024-01-22 13:36:19.891] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:19.892] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.0.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:19.892] [DEBUG] - Searching start_check plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:19.892] [DEBUG] - Found for obproxy-ce-py_script_start_check-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:19.892] [DEBUG] - Searching start_check plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:19.893] [DEBUG] - Found for obagent-py_script_start_check-1.3.0 for obagent-1.3.1
[2024-01-22 13:36:19.893] [DEBUG] - Searching start_check plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:19.894] [DEBUG] - Found for ocp-express-py_script_start_check-1.0.1 for ocp-express-1.0.1
[2024-01-22 13:36:19.894] [DEBUG] - Searching start plugin for components …
[2024-01-22 13:36:19.894] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:19.895] [DEBUG] - Found for oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:19.895] [DEBUG] - Searching start plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:19.896] [DEBUG] - Found for obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:19.896] [DEBUG] - Searching start plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:19.896] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-1.3.1
[2024-01-22 13:36:19.896] [DEBUG] - Searching start plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:19.897] [DEBUG] - Found for ocp-express-py_script_start-1.0.1 for ocp-express-1.0.1
[2024-01-22 13:36:19.897] [DEBUG] - Searching stop plugin for components …
[2024-01-22 13:36:19.897] [DEBUG] - Searching stop plugin for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:19.898] [DEBUG] - Found for oceanbase-ce-py_script_stop-3.1.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:19.898] [DEBUG] - Searching stop plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:19.898] [DEBUG] - Found for obproxy-ce-py_script_stop-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:19.898] [DEBUG] - Searching stop plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:19.899] [DEBUG] - Found for obagent-py_script_stop-1.3.0 for obagent-1.3.1
[2024-01-22 13:36:19.899] [DEBUG] - Searching stop plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:19.899] [DEBUG] - Found for ocp-express-py_script_stop-1.0 for ocp-express-1.0.1
[2024-01-22 13:36:19.900] [DEBUG] - Searching connect plugin for components …
[2024-01-22 13:36:19.900] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:19.900] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:19.901] [DEBUG] - Searching connect plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:19.901] [DEBUG] - Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:19.901] [DEBUG] - Searching connect plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:19.902] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-1.3.1
[2024-01-22 13:36:19.902] [DEBUG] - Searching connect plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:19.902] [DEBUG] - Found for ocp-express-py_script_connect-1.0.1 for ocp-express-1.0.1
[2024-01-22 13:36:19.902] [DEBUG] - Searching display plugin for components …
[2024-01-22 13:36:19.902] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:19.903] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:19.903] [DEBUG] - Searching display plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:19.904] [DEBUG] - Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:19.904] [DEBUG] - Searching display plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:19.904] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-1.3.1
[2024-01-22 13:36:19.904] [DEBUG] - Searching display plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:19.905] [DEBUG] - Found for ocp-express-py_script_display-1.0.1 for ocp-express-1.0.1
[2024-01-22 13:36:19.905] [DEBUG] - Searching bootstrap plugin for components …
[2024-01-22 13:36:19.905] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:19.906] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:19.906] [DEBUG] - Searching bootstrap plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:19.906] [DEBUG] - Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:19.907] [DEBUG] - Searching bootstrap plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:19.907] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-1.3.1
[2024-01-22 13:36:19.907] [DEBUG] - Searching bootstrap plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:19.908] [DEBUG] - Found for ocp-express-py_script_bootstrap-1.0.1 for ocp-express-1.0.1
[2024-01-22 13:36:19.949] [INFO] Load cluster param plugin
[2024-01-22 13:36:19.950] [DEBUG] - Searching param plugin for components …
[2024-01-22 13:36:19.950] [DEBUG] - Search param plugin for oceanbase-ce
[2024-01-22 13:36:19.951] [DEBUG] - Found for oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:19.951] [DEBUG] - Applying oceanbase-ce-param-4.2.0.0 for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:20.933] [DEBUG] - Search param plugin for obproxy-ce
[2024-01-22 13:36:20.933] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:20.933] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:21.165] [DEBUG] - Search param plugin for obagent
[2024-01-22 13:36:21.166] [DEBUG] - Found for obagent-param-1.3.0 for obagent-1.3.1
[2024-01-22 13:36:21.166] [DEBUG] - Applying obagent-param-1.3.0 for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:21.272] [DEBUG] - Search param plugin for ocp-express
[2024-01-22 13:36:21.272] [DEBUG] - Found for ocp-express-param-1.0.1 for ocp-express-1.0.1
[2024-01-22 13:36:21.272] [DEBUG] - Applying ocp-express-param-1.0.1 for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:21.525] [INFO] Open ssh connection
[2024-01-22 13:36:21.788] [INFO] Cluster status check
[2024-01-22 13:36:21.789] [DEBUG] - Searching status plugin for components …
[2024-01-22 13:36:21.789] [DEBUG] - Searching status plugin for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:21.791] [DEBUG] - Found for oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.0.0
[2024-01-22 13:36:21.791] [DEBUG] - Searching status plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:21.792] [DEBUG] - Found for obproxy-ce-py_script_status-3.1.0 for obproxy-ce-4.1.0.0
[2024-01-22 13:36:21.792] [DEBUG] - Searching status plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:21.792] [DEBUG] - Found for obagent-py_script_status-1.3.0 for obagent-1.3.1
[2024-01-22 13:36:21.792] [DEBUG] - Searching status plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:21.793] [DEBUG] - Found for ocp-express-py_script_status-1.0 for ocp-express-1.0.1
[2024-01-22 13:36:21.793] [DEBUG] - Call oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.0.0-101000032023091319.el7-176ae96ab6ea606c860e0a9db09f8046eec0ceba
[2024-01-22 13:36:21.793] [DEBUG] - import status
[2024-01-22 13:36:21.795] [DEBUG] - add status ref count to 1
[2024-01-22 13:36:21.796] [DEBUG] – root@172.16.0.58 execute: cat /root/wisdombase/oceanbase/run/observer.pid
[2024-01-22 13:36:21.946] [DEBUG] – exited code 0
[2024-01-22 13:36:21.947] [DEBUG] – root@172.16.0.58 execute: ls /proc/10843
[2024-01-22 13:36:22.010] [DEBUG] – exited code 0
[2024-01-22 13:36:22.011] [DEBUG] - sub status ref count to 0
[2024-01-22 13:36:22.011] [DEBUG] - export status
[2024-01-22 13:36:22.012] [DEBUG] - Call obproxy-ce-py_script_status-3.1.0 for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2024-01-22 13:36:22.012] [DEBUG] - import status
[2024-01-22 13:36:22.014] [DEBUG] - add status ref count to 1
[2024-01-22 13:36:22.014] [DEBUG] – root@172.16.0.58 execute: cat /root/wisdombase/obproxy/run/obproxy-172.16.0.58-2883.pid
[2024-01-22 13:36:22.074] [DEBUG] – exited code 0
[2024-01-22 13:36:22.075] [DEBUG] – root@172.16.0.58 execute: ls /proc/13271
[2024-01-22 13:36:22.139] [DEBUG] – exited code 0
[2024-01-22 13:36:22.140] [DEBUG] - sub status ref count to 0
[2024-01-22 13:36:22.140] [DEBUG] - export status
[2024-01-22 13:36:22.141] [DEBUG] - Call obagent-py_script_status-1.3.0 for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2024-01-22 13:36:22.141] [DEBUG] - import status
[2024-01-22 13:36:22.142] [DEBUG] - add status ref count to 1
[2024-01-22 13:36:22.142] [DEBUG] – root@172.16.0.58 execute: cat /root/wisdombase/obagent/run/ob_agentd.pid
[2024-01-22 13:36:22.203] [DEBUG] – exited code 0
[2024-01-22 13:36:22.205] [DEBUG] – root@172.16.0.58 execute: ls /proc/16893
[2024-01-22 13:36:22.268] [DEBUG] – exited code 0
[2024-01-22 13:36:22.269] [DEBUG] - sub status ref count to 0
[2024-01-22 13:36:22.270] [DEBUG] - export status
[2024-01-22 13:36:22.270] [DEBUG] - Call ocp-express-py_script_status-1.0 for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:22.270] [DEBUG] - import status
[2024-01-22 13:36:22.271] [DEBUG] - add status ref count to 1
[2024-01-22 13:36:22.272] [DEBUG] – root@172.16.0.58 execute: cat /root/wisdombase/ocpexpress/run/ocp-express.pid
[2024-01-22 13:36:22.332] [DEBUG] – exited code 0
[2024-01-22 13:36:22.333] [DEBUG] – root@172.16.0.58 execute: ls /proc/12179
[2024-01-22 13:36:22.398] [DEBUG] – exited code 0
[2024-01-22 13:36:22.399] [DEBUG] - sub status ref count to 0
[2024-01-22 13:36:22.399] [DEBUG] - export status
[2024-01-22 13:36:22.444] [DEBUG] - Call ocp-express-py_script_restart-1.0 for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:22.445] [DEBUG] - import restart
[2024-01-22 13:36:22.457] [DEBUG] - add restart ref count to 1
[2024-01-22 13:36:22.457] [DEBUG] – Call ocp-express-py_script_stop-1.0 for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:22.458] [DEBUG] — import stop
[2024-01-22 13:36:22.466] [DEBUG] — add stop ref count to 1
[2024-01-22 13:36:22.468] [INFO] Stop ocp-express
[2024-01-22 13:36:22.469] [DEBUG] ---- root@172.16.0.58 execute: cat /root/wisdombase/ocpexpress/run/ocp-express.pid
[2024-01-22 13:36:22.492] [DEBUG] ---- exited code 0
[2024-01-22 13:36:22.493] [DEBUG] ---- root@172.16.0.58 execute: ls /proc/12179
[2024-01-22 13:36:22.558] [DEBUG] ---- exited code 0
[2024-01-22 13:36:22.558] [DEBUG] ---- root@172.16.0.58 execute: ls /proc/12179/fd
[2024-01-22 13:36:22.624] [DEBUG] ---- exited code 0
[2024-01-22 13:36:22.625] [DEBUG] ---- 172.16.0.58 ocp-express[pid: 12179] stopping…
[2024-01-22 13:36:22.625] [DEBUG] ---- root@172.16.0.58 execute: kill -9 12179
[2024-01-22 13:36:22.694] [DEBUG] ---- exited code 0
[2024-01-22 13:36:23.696] [DEBUG] ---- 172.16.0.58 check whether the port is released
[2024-01-22 13:36:23.696] [DEBUG] ---- root@172.16.0.58 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:1FF4’ | awk -F’ ’ ‘{print $2}’ | uniq
[2024-01-22 13:36:23.734] [DEBUG] ---- exited code 0
[2024-01-22 13:36:23.734] [DEBUG] ---- root@172.16.0.58 execute: rm -rf /root/wisdombase/ocpexpress/run/ocp-express.pid
[2024-01-22 13:36:23.796] [DEBUG] ---- exited code 0
[2024-01-22 13:36:23.797] [DEBUG] ---- 172.16.0.58 ocp-express is stopped
[2024-01-22 13:36:23.904] [DEBUG] — sub stop ref count to 0
[2024-01-22 13:36:23.904] [DEBUG] — export stop
[2024-01-22 13:36:23.905] [DEBUG] – Call ocp-express-py_script_start_check-1.0.1 for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:23.905] [DEBUG] — import start_check
[2024-01-22 13:36:23.916] [DEBUG] — add start_check ref count to 1
[2024-01-22 13:36:23.917] [INFO] Check before start ocp-express
[2024-01-22 13:36:23.921] [DEBUG] ---- root@172.16.0.58 execute: cat /root/wisdombase/ocpexpress/run/ocp-express.pid
[2024-01-22 13:36:23.944] [DEBUG] ---- exited code 1, error output:
[2024-01-22 13:36:23.944] [DEBUG] cat: /root/wisdombase/ocpexpress/run/ocp-express.pid: 没有那个文件或目录
[2024-01-22 13:36:23.944] [DEBUG]
[2024-01-22 13:36:23.945] [DEBUG] ---- root@172.16.0.58 execute: bash -c ‘cat /proc/net/{udp*,tcp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:1FF4’ | awk -F’ ’ ‘{print $2}’ | uniq
[2024-01-22 13:36:24.018] [DEBUG] ---- exited code 0
[2024-01-22 13:36:24.019] [DEBUG] ---- root@172.16.0.58 execute: java -version
[2024-01-22 13:36:24.192] [DEBUG] ---- exited code 0
[2024-01-22 13:36:24.194] [DEBUG] ---- root@172.16.0.58 execute: cat /proc/meminfo
[2024-01-22 13:36:24.254] [DEBUG] ---- exited code 0
[2024-01-22 13:36:24.255] [DEBUG] ---- root@172.16.0.58 execute: df --block-size=1024
[2024-01-22 13:36:24.319] [DEBUG] ---- exited code 0
[2024-01-22 13:36:24.320] [DEBUG] ---- get disk info for path /dev, total: 20990853120 avail: 20990853120
[2024-01-22 13:36:24.321] [DEBUG] ---- get disk info for path /dev/shm, total: 21003476992 avail: 21003476992
[2024-01-22 13:36:24.321] [DEBUG] ---- get disk info for path /run, total: 21003476992 avail: 20993929216
[2024-01-22 13:36:24.321] [DEBUG] ---- get disk info for path /sys/fs/cgroup, total: 21003476992 avail: 21003476992
[2024-01-22 13:36:24.321] [DEBUG] ---- get disk info for path /, total: 139739422720 avail: 15992414208
[2024-01-22 13:36:24.321] [DEBUG] ---- get disk info for path /home, total: 2046640128 avail: 1873690624
[2024-01-22 13:36:24.321] [DEBUG] ---- get disk info for path /boot, total: 499337216 avail: 246888448
[2024-01-22 13:36:24.321] [DEBUG] ---- get disk info for path /run/user/0, total: 4200697856 avail: 4200697856
[2024-01-22 13:36:24.322] [DEBUG] ---- root@172.16.0.58 execute: df --block-size=1024 /root/wisdombase/ocpexpress/log
[2024-01-22 13:36:24.382] [DEBUG] ---- exited code 0
[2024-01-22 13:36:24.382] [DEBUG] ---- get disk info for path /, total: 139739422720 avail: 15992414208
[2024-01-22 13:36:24.443] [DEBUG] — sub start_check ref count to 0
[2024-01-22 13:36:24.443] [DEBUG] — export start_check
[2024-01-22 13:36:24.443] [DEBUG] – Call ocp-express-py_script_start-1.0.1 for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2024-01-22 13:36:24.443] [DEBUG] — import start
[2024-01-22 13:36:25.185] [DEBUG] — add start ref count to 1
[2024-01-22 13:36:25.221] [INFO] Start ocp-express
[2024-01-22 13:36:25.222] [DEBUG] ---- root@172.16.0.58 execute: cat /root/wisdombase/ocpexpress/run/ocp-express.pid
[2024-01-22 13:36:25.245] [DEBUG] ---- exited code 1, error output:
[2024-01-22 13:36:25.245] [DEBUG] cat: /root/wisdombase/ocpexpress/run/ocp-express.pid: 没有那个文件或目录
[2024-01-22 13:36:25.246] [DEBUG]
[2024-01-22 13:36:25.246] [DEBUG] ---- root@172.16.0.58 execute: ls /root/wisdombase/ocpexpress/.bootstrapped
[2024-01-22 13:36:25.308] [DEBUG] ---- exited code 2, error output:
[2024-01-22 13:36:25.308] [DEBUG] ls: 无法访问/root/wisdombase/ocpexpress/.bootstrapped: 没有那个文件或目录
[2024-01-22 13:36:25.308] [DEBUG]
[2024-01-22 13:36:25.309] [DEBUG] ---- connect 172.16.0.58 -P2883 -umeta@ocp -poA3Pu9oKyZ
省略一大段相同的
[2024-01-22 13:46:23.384] [DEBUG] ---- connect 172.16.0.58 -P2883 -umeta@ocp -poA3Pu9oKyZ
[2024-01-22 13:46:25.384] [ERROR] 172.16.0.58: failed to connect meta db
[2024-01-22 13:46:25.474] [INFO] [ERROR] 172.16.0.58: failed to connect meta db
[2024-01-22 13:46:25.474] [INFO]
[2024-01-22 13:46:25.474] [DEBUG] — sub start ref count to 0
[2024-01-22 13:46:25.475] [DEBUG] — export start
[2024-01-22 13:46:25.475] [DEBUG] - sub restart ref count to 0
[2024-01-22 13:46:25.475] [DEBUG] - export restart
[2024-01-22 13:46:25.480] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2024-01-22 13:46:25.481] [INFO] Trace ID: 2b90751a-b8e8-11ee-8eb5-b499bab66c38
[2024-01-22 13:46:25.481] [INFO] If you want to view detailed obd logs, please run: obd display-trace 2b90751a-b8e8-11ee-8eb5-b499bab66c38
[2024-01-22 13:46:25.482] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 3
[2024-01-22 13:46:25.482] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 2
[2024-01-22 13:46:25.482] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[2024-01-22 13:46:25.483] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2024-01-22 13:46:25.483] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2024-01-22 13:46:25.483] [DEBUG] - exclusive lock /root/.obd/lock/deploy_wisdombase release, count 0
[2024-01-22 13:46:25.483] [DEBUG] - unlock /root/.obd/lock/deploy_wisdombase
[2024-01-22 13:46:25.483] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2024-01-22 13:46:25.484] [DEBUG] - unlock /root/.obd/lock/global

麻烦发下 、observer.log日志 还有配置文件

配置文件在哪个位置?

obd cluster list 可以看得到路径

log_config.rar (4.9 MB)

重新obd cluster restart xxx后,其它都起来能连接 了,就差ocp-express了,trace如下:
ocp_trace.txt (259.9 KB)

OCP-Express启动失败 这个方式试下。

1 个赞

根据你提供的帖子解决了,原来是BUG,按帖子操作如下可行,感谢。
在集群成功启动但ocp-express失败的情况下,单独stop ocp-express 再 start ocp-exrepss,则ocp-express启动成功。

obd cluster stop 集群名称 -c ocp-express
obd cluster start 集群名称 -c ocp-express

1 个赞