使用obd cluster restart obcluster -s 重启某个节点时，由于密码的原因进入循环，管理员密码有设置

小白一枚 · 2023 年11 月 14 日 10:53

【使用环境】生产环境
【 OB or 其他组件】 OBD 重启节点
【使用版本】4.1.0.1
【问题描述】使用obd cluster restart obcluster -s 重启某个节点时，重启不成功，查看日志，应该是密码问题
【复现路径】服务器重启，某个节点停止OB的服务
【问题现象及影响】
OB集群某个节点不可用。从日志看，connect执行了二次，第一次连接时有密码，第二次密码变成空了。
obd cluster restart obcluster -s 192.XXX.XXX.XXX
日志内容：
[2023-11-14 02:40:01.598] [INFO] Connect to observer
[2023-11-14 02:40:01.598] [DEBUG] – Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.1.0.1-102000042023061314.el7-d03fafa6fa8ceb0636e4db05b5b5f6c3ac2256a3
[2023-11-14 02:40:01.598] [DEBUG] — import connect
[2023-11-14 02:40:01.622] [DEBUG] — add connect ref count to 1
[2023-11-14 02:40:01.622] [DEBUG] ---- connect 192.XXX.XXX.XXX -P2881 -uroot -pXXXXXXX9
[2023-11-14 02:40:04.631] [DEBUG] ---- connect 192.XXX.XXX.XXX -P2881 -uroot -p
[2023-11-14 02:40:04.638] [DEBUG] — sub connect ref count to 0
[2023-11-14 02:40:04.638] [DEBUG] — export connect
[2023-11-14 02:40:04.727] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:04.730] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:06.732] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:06.734] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:08.736] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:08.739] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:10.741] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:10.744] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:12.746] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:12.749] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:14.751] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:14.754] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:16.756] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:16.759] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:18.761] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:18.764] [DEBUG] ---- OBD-5000: use oceanbase execute failed
【附件】

王利博 · 2023 年11 月 14 日 11:04

麻烦提供下完整得obd日志和observer.log日志

小白一枚 · 2023 年11 月 14 日 11:44

日志链接。https://nextera-chat.oss-cn-shanghai.aliyuncs.com/oceanbase.zip

秃蛙 · 2023 年11 月 14 日 13:59

看日志是有杀掉进程，是未完成执行了Ctrl+C了吧。

小白一枚 · 2023 年11 月 14 日 14:34

是的，因为反复出现下面的内容，就强行中止了：
[2023-11-14 02:40:04.730] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:06.732] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:06.734] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:08.736] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:08.739] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:10.741] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:10.744] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:12.746] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:12.749] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:14.751] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:14.754] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:16.756] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:16.759] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:18.761] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:18.764] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:20.766] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:20.769] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:22.771] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:22.774] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:24.776] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:24.779] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:26.781] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:26.784] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:28.786] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:28.789] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:30.791] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:30.793] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:32.796] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:32.798] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:34.800] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:34.803] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:36.805] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:36.808] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:38.810] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:38.813] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:40.815] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:40.818] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:42.820] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:42.823] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:44.824] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:44.826] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:46.829] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:46.831] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:48.834] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:48.836] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:50.838] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:50.841] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:52.843] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:52.845] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:54.847] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:54.850] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:56.852] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:56.855] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:58.857] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:58.860] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:00.862] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:00.865] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:02.867] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:02.870] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:04.872] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:04.875] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:06.877] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:06.880] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:08.882] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:08.884] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:10.885] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:10.888] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:12.890] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:12.893] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:14.895] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:14.898] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:16.900] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:16.903] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:18.905] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:18.908] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:20.910] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:20.913] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed

秃蛙 · 2023 年11 月 14 日 14:37

这个打印符合预期的，如果涉及组件较多会循环打印，可以增加-c参数指定组件。例如：obd cluster restart obcluster -c oceanbase-ce -s xx.xx.xx.xx

小白一枚 · 2023 年11 月 14 日 14:45

哦，那我试一试

小白一枚 · 2023 年11 月 14 日 15:45

应该还是有问题的，我已经执行了50分钟，一直在提示：
[2023-11-14 02:40:04.730] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:06.732] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:06.734] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:08.736] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None

OBD-5000之前的连接是没有密码的，是不是因为这个？但实际上是设置了密码。
[2023-11-14 14:51:50.794] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] — add connect ref count to 1
[2023-11-14 14:51:50.795] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] ---- connect 192.XXX.XXX.72 -P2881 -uroot -p7xxxxxxxx9
[2023-11-14 14:51:53.805] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] ---- connect 192.XXX.XXX.72 -P2881 -uroot -p
[2023-11-14 14:51:53.812] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] — sub connect ref count to 0
[2023-11-14 14:51:53.812] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] — export connect

秃蛙 · 2023 年11 月 14 日 16:13

你有登录数据库设置过集群密码吗？

mysql -h192.XXX.XXX.72 -P2881 -uroot -p7XXXXXX9 能否登录？

小白一枚 · 2023 年11 月 14 日 16:22

这个我之前都是正常，前段时间服务器有重启，所以就出现问题了

小白一枚 · 2023 年11 月 14 日 16:24

这个节点的状态是INACTIVE。所以我尝试使用obd cluster restart obcluster -s 来重启节点

秃蛙 · 2023 年11 月 14 日 17:33

INACTIVE节点的observer.log麻烦提供下

小白一枚 · 2023 年11 月 14 日 17:37

https://nextera-chat.oss-cn-shanghai.aliyuncs.com/oceanbase.zip ](https://nextera-chat.oss-cn-shanghai.aliyuncs.com/oceanbase.zip放的就是INACTIVE节点的observer.log

秃蛙 · 2023 年11 月 15 日 10:06

麻烦把rs主节点的observer.log和rootservice.log提供下