【 使用环境 】生产环境
【 OB or 其他组件 】 OBD 重启节点
【 使用版本 】4.1.0.1
【问题描述】使用obd cluster restart obcluster -s 重启某个节点时,重启不成功,查看日志,应该是密码问题
【复现路径】服务器重启,某个节点停止OB的服务
【问题现象及影响】
OB集群某个节点不可用。从日志看,connect执行了二次,第一次连接时有密码,第二次密码变成空了。
obd cluster restart obcluster -s 192.XXX.XXX.XXX
日志内容:
[2023-11-14 02:40:01.598] [INFO] Connect to observer
[2023-11-14 02:40:01.598] [DEBUG] – Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.1.0.1-102000042023061314.el7-d03fafa6fa8ceb0636e4db05b5b5f6c3ac2256a3
[2023-11-14 02:40:01.598] [DEBUG] — import connect
[2023-11-14 02:40:01.622] [DEBUG] — add connect ref count to 1
[2023-11-14 02:40:01.622] [DEBUG] ---- connect 192.XXX.XXX.XXX -P2881 -uroot -pXXXXXXX9
[2023-11-14 02:40:04.631] [DEBUG] ---- connect 192.XXX.XXX.XXX -P2881 -uroot -p
[2023-11-14 02:40:04.638] [DEBUG] — sub connect ref count to 0
[2023-11-14 02:40:04.638] [DEBUG] — export connect
[2023-11-14 02:40:04.727] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:04.730] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:06.732] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:06.734] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:08.736] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:08.739] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:10.741] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:10.744] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:12.746] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:12.749] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:14.751] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:14.754] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:16.756] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:16.759] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:18.761] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:18.764] [DEBUG] ---- OBD-5000: use oceanbase execute failed
【附件】
麻烦提供下完整得obd日志和observer.log日志
是的,因为反复出现下面的内容,就强行中止了:
[2023-11-14 02:40:04.730] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:06.732] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:06.734] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:08.736] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:08.739] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:10.741] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:10.744] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:12.746] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:12.749] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:14.751] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:14.754] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:16.756] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:16.759] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:18.761] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:18.764] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:20.766] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:20.769] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:22.771] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:22.774] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:24.776] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:24.779] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:26.781] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:26.784] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:28.786] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:28.789] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:30.791] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:30.793] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:32.796] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:32.798] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:34.800] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:34.803] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:36.805] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:36.808] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:38.810] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:38.813] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:40.815] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:40.818] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:42.820] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:42.823] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:44.824] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:44.826] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:46.829] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:46.831] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:48.834] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:48.836] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:50.838] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:50.841] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:52.843] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:52.845] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:54.847] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:54.850] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:56.852] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:56.855] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:58.857] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:58.860] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:00.862] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:00.865] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:02.867] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:02.870] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:04.872] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:04.875] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:06.877] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:06.880] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:08.882] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:08.884] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:10.885] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:10.888] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:12.890] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:12.893] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:14.895] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:14.898] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:16.900] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:16.903] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:18.905] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:18.908] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:41:20.910] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:41:20.913] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
这个打印符合预期的,如果涉及组件较多会循环打印,可以增加-c参数指定组件。 例如:obd cluster restart obcluster -c oceanbase-ce -s xx.xx.xx.xx
哦,那我试一试
应该还是有问题的,我已经执行了50分钟,一直在提示:
[2023-11-14 02:40:04.730] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:06.732] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
[2023-11-14 02:40:06.734] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- OBD-5000: use oceanbase execute failed
[2023-11-14 02:40:08.736] [19978402-8297-11ee-b366-ac1f6bf105c7] [DEBUG] ---- execute sql: use oceanbase. args: None
OBD-5000之前的连接是没有密码的,是不是因为这个?但实际上是设置了密码。
[2023-11-14 14:51:50.794] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] — add connect ref count to 1
[2023-11-14 14:51:50.795] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] ---- connect 192.XXX.XXX.72 -P2881 -uroot -p7xxxxxxxx9
[2023-11-14 14:51:53.805] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] ---- connect 192.XXX.XXX.72 -P2881 -uroot -p
[2023-11-14 14:51:53.812] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] — sub connect ref count to 0
[2023-11-14 14:51:53.812] [46d67e6e-82ba-11ee-8fa6-ac1f6bf40430] [DEBUG] — export connect
你有登录数据库设置过集群密码吗?
mysql -h192.XXX.XXX.72 -P2881 -uroot -p7XXXXXX9 能否登录?
这个我之前都是正常,前段时间服务器有重启,所以就出现问题了
这个节点的状态是INACTIVE。所以我尝试使用obd cluster restart obcluster -s 来重启节点
INACTIVE节点的observer.log麻烦提供下
https://nextera-chat.oss-cn-shanghai.aliyuncs.com/oceanbase.zip ](https://nextera-chat.oss-cn-shanghai.aliyuncs.com/oceanbase.zip放的就是INACTIVE节点的observer.log
麻烦把rs主节点的observer.log和rootservice.log提供下