obshell rolling upgrade 节点启动超时

OB rolling upgrade 4.4.2.2-102000022026071514.el7

Reinstall and restart observer ID: 1111


2026-08-03 20:34:54

 build start service time map

2026-08-03 20:34:54

 stop ob

2026-08-03 20:34:54

 Get observer Pid

2026-08-03 20:34:56

 Kill observer process 824849

2026-08-03 20:35:00

 Check observer process

2026-08-03 20:35:00

 Successfully killed the observer process

2026-08-03 20:35:00

 reinstall ob

2026-08-03 20:35:00

 copy new ob from '/data/oceanbase/cluster/oceanbase/upgrade/rpm-1785496539502/x86_64/4.3.5.6/home/admin/oceanbase' to '/data/oceanbase/cluster/oceanbase'

2026-08-03 20:35:00

 start ob

2026-08-03 20:35:00

 check if first start

2026-08-03 20:35:00

 not first start, skip require check

2026-08-03 20:35:00

 generate start cmd

2026-08-03 20:35:00

 start cmd: export LD_LIBRARY_PATH='/data/oceanbase/cluster/oceanbase/lib'; /data/oceanbase/cluster/oceanbase/bin/observer   

2026-08-03 20:36:11

 wait all observer available

2026-08-03 20:41:15

 ERROR: [OB.Cluster.AsyncOperationTimeout]: wait all observer available timeout

2026-08-03 20:41:15

 ERROR: [OB.Cluster.AsyncOperationTimeout]: wait all observer available timeout

使用 obshell web 界面上更新时,由于数据库启动过慢超时了(启动成功但耗时10分钟)导致失败,有办法调整超时时间吗?

问了下chatgpt,obshell 检测 ob 启动是每60s轮询一次,持续5轮共 5min,当前启动耗时10分钟。
虚拟服务器节点,硬盘太慢真是抱歉:sob:

无法取消更新 或 跳过当前子任务:
设置 DAG 回滚失败:节点 Restart obshell 不能回滚
跳过节点失败:节点 Reinstall and restart observer 不能跳过


2026-08-03 04:36:15.916697|INFO|SERVER|OB_SERVER_INIT_BEGIN|0|0|2249178|observer|Y0-0000000000000001-0-0|init|ob_server.cpp:245|"[server_start 1/18] observer init begin."

2026-08-03 04:36:15.964814|INFO|LIB|OB_SERVER_SYSLOG_SERVICE_INIT_BEGIN|0|0|2249178|observer|Y0-0000000000000001-0-0|init|ob_log.cpp:1514|"[server_start 2/18] observer syslog service init begin."

2026-08-03 04:36:16.012970|INFO|LIB|OB_SERVER_SYSLOG_SERVICE_INIT_SUCCESS|0|0|2249178|observer|Y0-0000000000000001-0-0|init|ob_log.cpp:1649|"[server_start 3/18] observer syslog service init success."

2026-08-03 04:36:39.218984|INFO|SERVER|OB_SERVER_INIT_SUCCESS|0|0|2249178|observer|Y0-0000000000000001-0-0|init|ob_server.cpp:562|"[server_start 4/18] observer init success."

2026-08-03 04:36:39.219300|INFO|SERVER|OB_SERVER_START_BEGIN|0|0|2249178|observer|Y0-0000000000000001-0-0|start|ob_server.cpp:926|"[server_start 5/18] observer start begin."

2026-08-03 04:36:39.219416|INFO|SERVER|OB_SERVER_INSTANCE_START_BEGIN|0|0|2249178|observer|Y0-0000000000000001-0-0|start|ob_server.cpp:944|"[server_start 6/18] observer instance start begin."

2026-08-03 04:36:39.811751|INFO|STORAGE_BLKMGR|OB_SERVER_BLOCK_MANAGER_START_BEGIN|0|0|2249178|observer|Y0-0000000000000001-0-0|start|ob_block_manager.cpp:183|"[server_start 7/18] block manager start begin."

2026-08-03 04:36:39.919946|INFO|STORAGE_BLKMGR|OB_SERVER_BLOCK_MANAGER_START_SUCCESS|0|0|2249178|observer|Y0-0000000000000001-0-0|start|ob_block_manager.cpp:218|"[server_start 8/18] block manager start success."

2026-08-03 04:38:59.683323|INFO|SERVER|OB_SERVER_WAIT_SCHEMA_READY_SUCCESS|0|0|2249178|observer|YB42AC198511-0006581D159004AB-0-0|check_if_schema_ready|ob_server.cpp:1328| REACH SYSLOG RATE LIMIT [bandwidth]

2026-08-03 04:39:05.684690|INFO|SERVER|OB_SERVER_CHECK_USER_TENANT_SCHEMA_REFRESHED_BEGIN|0|0|2249178|observer|YB42AC198511-0006581D159004AB-0-0|check_user_tenant_schema_refreshed|ob_server.cpp:3404| REACH SYSLOG RATE LIMIT [bandwidth]

2026-08-03 04:39:36.379464|INFO|SERVER|OB_SERVER_CHECK_USER_TENANT_SCHEMA_REFRESHED_FINISH|0|0|2249178|observer|YB42AC198511-0006581D159004AB-0-0|check_user_tenant_schema_refreshed|ob_server.cpp:3452| REACH SYSLOG RATE LIMIT [bandwidth]

2026-08-03 04:45:28.527457|INFO|SERVER|OB_SERVER_CHECK_LOG_REPLAY_OVER_FINISH|0|0|2249178|observer|YB42AC198511-0006581D159004AB-0-0|check_log_replay_over|ob_server.cpp:3485|"[server_start 17/18] observer check log replay over finish."

2026-08-03 04:45:28.559677|INFO|SERVER|OB_SERVER_START_SUCCESS|0|0|2249178|observer|YB42AC198511-0006581D159004AB-0-0|start|ob_server.cpp:1249|"[server_start 18/18] observer start success."

2026-08-03 04:45:28.559759|INFO|SERVER|OB_SERVER_WAIT_BEGIN|0|0|2249178|observer|YB42AC198511-0006581D159004AB-0-0|wait|ob_server.cpp:1725|"observer process wait begin."
6 个赞

能截个图看看么?具体什么样的操作

感谢回复,截图如下

1 个赞

能把完整的日志取一下么?obshell的日志 observer的日志都发一下 要启动时候的日志

2 个赞

顶顶

observer 的启动日志有提示 The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)

使用 obd cluster init4env cluster 调整上述参数后,observer 的启动时间缩短到了 5min内,问题解决了。

2 个赞

日志对解决问题还是非常关键的,学习了

1 个赞

确实

指标,日志,跟踪

调整observer的启动时间。