OceanBase CE 4.3.5.4 单机 OBD 部署启动失败，observer 恢复阶段报 OB_LAST_LOG_RUINNED，fail to replay create tenant(ret=-4052, tenant_id=1002)

OceanBase 技术问题
#1

环境：

  • 时间：2026-07-21
  • 部署方式：OBD 单机部署
  • 版本：oceanbase-ce-4.3.5.4
  • 场景：complex_oltp
  • observer IP：单节点
  • mysql_port：2881
  • rpc_port：2882
  • data dir：/home/netmarch/data/1
  • observer dir：/home/netmarch/myoceanbase/oceanbase

现象：

  • 执行 obd cluster start myoceanbase 后，observer 进程能拉起，但 OBD 持续连接 :2881 失败，最终报：
    OBD-1006: Failed to connect to oceanbase-ce

  • observer 日志中关键报错：
    fail to replay create tenant(ret=-4052, tenant_id=1002)
    fail to start server storage meta service(ret=-4052, ret="OB_LAST_LOG_RUINNED")
    observer instance start fail

  • 根因看起来不是端口、密码或磁盘空间问题，而是启动恢复阶段回放 tenant_1002 的存储元数据/日志失败

  • 移走 /home/netmarch/data/1/slog/tenant_1002 后，故障形态变为元数据不一致，tenant_1002 仍被系统内部 SQL 引用，实例仍不可用

  • 目标是保数据，不接受直接重建实例

11 个赞
#3
  1. obd 启动失败主报错

[ERROR] OBD-1006: Failed to connect to oceanbase-ce
pymysql.err.OperationalError: (2003, “Can’t connect to MySQL server on ‘xxx.xxx.xxx.xxx’ ([Errno 111] Connection refused)”)

  1. observer 原始根因

EDIAG [STORAGE] handle_tenant_create_commit_ … fail to replay create tenant(ret=-4052, tenant_id=1002)
WDIAG [STORAGE] start_replay … fail to apply repaly result(ret=-4052)
WDIAG [STORAGE] start … fail to start replayer(ret=-4052)
EDIAG [SERVER] start … fail to start server storage meta service(ret=-4052, ret=“OB_LAST_LOG_RUINNED”)
ERROR [SERVER] start … observer instance start fail

  1. 误操作后现场变化

tenant_id=1002 相关内部 SQL 仍在访问：
select tenant_id, id, type, value, recovery_until_scn from __all_log_restore_source where tenant_id=1002 and id=1

同时出现：
Table ‘oceanbase.__all_ls_status’ doesn’t exist
OB_LS_LOCATION_NOT_EXIST
OB_SCHEMA_EAGAIN
OB_TENANT_NOT_EXIST

已经做过的排查

  • dmesg 未发现明显 I/O / ext4 / xfs / disk error
  • 磁盘空间充足：/ 剩余约 104G
  • /home/netmarch/data/1 总大小约 120G
  • /home/netmarch/data/1/slog/tenant_1002 大小约 3.0G
  • 已打包 tenant_1002 的 slog 证据包：/home/netmarch/ob_fail_2026-07-21/tenant_1002_slog.tar.gz
  • 本机无 LVM snapshot：
    lvs 显示 #SN=0
  • 当前无可用本地回退点
5 个赞
#4
  1. 对于 CE 4.3.5.4，OB_LAST_LOG_RUINNED + fail to replay create tenant(ret=-4052, tenant_id=1002) 是否有官方支持的保数据修复路径？
  2. 是否可以对 tenant_1002 的 slog / storage meta 做定向修复或跳过回放，而不破坏其余租户与系统租户元数据一致性？
  3. 需要我们继续补充哪些文件或目录（如 slog/server、tenant_1002 对应 palf/clog、observer.log 全量、trace 等）？
3 个赞
#5

提供一下obd.log日志

3 个赞
#6

[2026-07-21 10:59:31.145] [400911a8-84f3-11f1-b4b1-000c2988f524] [DEBUG] - cmd: []
[2026-07-21 10:59:31.145] [400911a8-84f3-11f1-b4b1-000c2988f524] [DEBUG] - opts: {}
[2026-07-21 10:59:31.145] [400911a8-84f3-11f1-b4b1-000c2988f524] [DEBUG] - mkdir /root/.obd/lock/
[2026-07-21 10:59:31.146] [400911a8-84f3-11f1-b4b1-000c2988f524] [DEBUG] - set lock mode to NO_LOCK(0)
[2026-07-21 10:59:31.146] [400911a8-84f3-11f1-b4b1-000c2988f524] [DEBUG] - Get deploy list
[2026-07-21 10:59:31.146] [400911a8-84f3-11f1-b4b1-000c2988f524] [DEBUG] - mkdir /root/.obd/cluster/
[2026-07-21 10:59:31.146] [400911a8-84f3-11f1-b4b1-000c2988f524] [DEBUG] - mkdir /root/.obd/config_parser/
[2026-07-21 10:59:31.150] [400911a8-84f3-11f1-b4b1-000c2988f524] [INFO] ±---------------------------------------------------------------+
[2026-07-21 10:59:31.151] [400911a8-84f3-11f1-b4b1-000c2988f524] [INFO] | Cluster List |
[2026-07-21 10:59:31.151] [400911a8-84f3-11f1-b4b1-000c2988f524] [INFO] ±------------±-------------------------------±----------------+
[2026-07-21 10:59:31.151] [400911a8-84f3-11f1-b4b1-000c2988f524] [INFO] | Name | Configuration Path | Status (Cached) |
[2026-07-21 10:59:31.151] [400911a8-84f3-11f1-b4b1-000c2988f524] [INFO] ±------------±-------------------------------±----------------+
[2026-07-21 10:59:31.151] [400911a8-84f3-11f1-b4b1-000c2988f524] [INFO] | myoceanbase | /root/.obd/cluster/myoceanbase | stopped |
[2026-07-21 10:59:31.151] [400911a8-84f3-11f1-b4b1-000c2988f524] [INFO] ±------------±-------------------------------±----------------+
[2026-07-21 10:59:31.151] [400911a8-84f3-11f1-b4b1-000c2988f524] [INFO] Trace ID: 400911a8-84f3-11f1-b4b1-000c2988f524
[2026-07-21 10:59:31.151] [400911a8-84f3-11f1-b4b1-000c2988f524] [INFO] If you want to view detailed obd logs, please run: obd display-trace 400911a8-84f3-11f1-b4b1-000c2988f524
root@netmarch:/mnt# tail -300 ~/.obd/log/obd
[2026-07-21 08:53:41.960] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add ocp_tenant_check ref count to 1
[2026-07-21 08:53:41.960] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub ocp_tenant_check ref count to 0
[2026-07-21 08:53:41.960] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export ocp_tenant_check
[2026-07-21 08:53:41.960] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin oceanbase-ce-py_script_ocp_tenant_check-4.0.0.0 result: True
[2026-07-21 08:53:41.960] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo, count 4
[2026-07-21 08:53:41.962] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching start_check_pre plugin for components …
[2026-07-21 08:53:41.962] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching start_check_pre plugin for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:41.962] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for obproxy-ce-py_script_start_check_pre-4.2.3 for obproxy-ce-4.3.5.0
[2026-07-21 08:53:41.962] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin obproxy-ce-py_script_start_check_pre-4.2.3 for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:41.962] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import start_check_pre
[2026-07-21 08:53:41.963] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add start_check_pre ref count to 1
[2026-07-21 08:53:41.963] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub start_check_pre ref count to 0
[2026-07-21 08:53:41.963] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export start_check_pre
[2026-07-21 08:53:41.963] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin obproxy-ce-py_script_start_check_pre-4.2.3 result: True
[2026-07-21 08:53:41.963] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching password_check plugin for components …
[2026-07-21 08:53:41.963] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching password_check plugin for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:41.964] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for obproxy-ce-py_script_password_check-3.1.0 for obproxy-ce-4.3.5.0
[2026-07-21 08:53:41.964] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin obproxy-ce-py_script_password_check-3.1.0 for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:41.964] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import password_check
[2026-07-21 08:53:41.965] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add password_check ref count to 1
[2026-07-21 08:53:41.965] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub password_check ref count to 0
[2026-07-21 08:53:41.965] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export password_check
[2026-07-21 08:53:41.965] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin obproxy-ce-py_script_password_check-3.1.0 result: True
[2026-07-21 08:53:41.965] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching status_check plugin for components …
[2026-07-21 08:53:41.965] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching status_check plugin for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:41.965] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for obproxy-ce-py_script_status_check-3.1.0 for obproxy-ce-4.3.5.0
[2026-07-21 08:53:41.965] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin obproxy-ce-py_script_status_check-3.1.0 for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:41.965] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import status_check
[2026-07-21 08:53:41.966] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add status_check ref count to 1
[2026-07-21 08:53:41.966] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – netmarch@172.16.0.253 execute: cat /home/netmarch/myoceanbase/obproxy/run/obproxy-172.16.0.253-2883.pid
[2026-07-21 08:53:42.001] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – exited code 1, error output:
[2026-07-21 08:53:42.001] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] cat: /home/netmarch/myoceanbase/obproxy/run/obproxy-172.16.0.253-2883.pid: No such file or directory
[2026-07-21 08:53:42.001] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG]
[2026-07-21 08:53:42.002] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub status_check ref count to 0
[2026-07-21 08:53:42.002] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export status_check
[2026-07-21 08:53:42.002] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin obproxy-ce-py_script_status_check-3.1.0 result: True
[2026-07-21 08:53:42.002] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching work_dir_check plugin for components …
[2026-07-21 08:53:42.002] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching work_dir_check plugin for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:42.002] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for obproxy-ce-py_script_work_dir_check-3.1.0 for obproxy-ce-4.3.5.0
[2026-07-21 08:53:42.002] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin obproxy-ce-py_script_work_dir_check-3.1.0 for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:42.002] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import work_dir_check
[2026-07-21 08:53:42.003] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add work_dir_check ref count to 1
[2026-07-21 08:53:42.003] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub work_dir_check ref count to 0
[2026-07-21 08:53:42.003] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export work_dir_check
[2026-07-21 08:53:42.003] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin obproxy-ce-py_script_work_dir_check-3.1.0 result: True
[2026-07-21 08:53:42.003] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching proxy_id_check plugin for components …
[2026-07-21 08:53:42.004] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching proxy_id_check plugin for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:42.004] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for obproxy-ce-py_script_proxy_id_check-4.2.3 for obproxy-ce-4.3.5.0
[2026-07-21 08:53:42.004] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin obproxy-ce-py_script_proxy_id_check-4.2.3 for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:42.004] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import proxy_id_check
[2026-07-21 08:53:42.005] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add proxy_id_check ref count to 1
[2026-07-21 08:53:42.005] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub proxy_id_check ref count to 0
[2026-07-21 08:53:42.005] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export proxy_id_check
[2026-07-21 08:53:42.005] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin obproxy-ce-py_script_proxy_id_check-4.2.3 result: True
[2026-07-21 08:53:42.005] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching port_check plugin for components …
[2026-07-21 08:53:42.005] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching port_check plugin for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:42.005] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for obproxy-ce-py_script_port_check-4.3.0 for obproxy-ce-4.3.5.0
[2026-07-21 08:53:42.005] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin obproxy-ce-py_script_port_check-4.3.0 for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:42.005] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import port_check
[2026-07-21 08:53:42.006] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add port_check ref count to 1
[2026-07-21 08:53:42.006] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – 172.16.0.253 port check
[2026-07-21 08:53:42.006] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – netmarch@172.16.0.253 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B43’ | awk -F’ ’ ‘{print $3}’ | uniq
[2026-07-21 08:53:42.055] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – exited code 0
[2026-07-21 08:53:42.055] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – netmarch@172.16.0.253 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B44’ | awk -F’ ’ ‘{print $3}’ | uniq
[2026-07-21 08:53:42.107] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – exited code 0
[2026-07-21 08:53:42.107] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – netmarch@172.16.0.253 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B45’ | awk -F’ ’ ‘{print $3}’ | uniq
[2026-07-21 08:53:42.160] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – exited code 0
[2026-07-21 08:53:42.160] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub port_check ref count to 0
[2026-07-21 08:53:42.160] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export port_check
[2026-07-21 08:53:42.160] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin obproxy-ce-py_script_port_check-4.3.0 result: True
[2026-07-21 08:53:42.161] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching start template for components …
[2026-07-21 08:53:42.161] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call workflow oceanbase-ce-py_script_workflow_start-4.2.5.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:42.162] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import start
[2026-07-21 08:53:42.162] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add start ref count to 1
[2026-07-21 08:53:42.163] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub start ref count to 0
[2026-07-21 08:53:42.163] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export start
[2026-07-21 08:53:42.163] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin oceanbase-ce-py_script_workflow_start-4.2.5.0 result: True
[2026-07-21 08:53:42.163] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for oceanbase-ce-py_script_workflow_start-4.2.5.0 for oceanbase-ce-4.2.5.0
[2026-07-21 08:53:42.163] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching start template for components …
[2026-07-21 08:53:42.163] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call workflow obproxy-ce-py_script_workflow_start-3.2.1 for obproxy-ce-4.3.5.0-3.el7-f17b277b681adb1c86bfc3cfda369ad88896da9d
[2026-07-21 08:53:42.164] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import start
[2026-07-21 08:53:42.164] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add start ref count to 1
[2026-07-21 08:53:42.164] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub start ref count to 0
[2026-07-21 08:53:42.164] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export start
[2026-07-21 08:53:42.164] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin obproxy-ce-py_script_workflow_start-3.2.1 result: True
[2026-07-21 08:53:42.164] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for obproxy-ce-py_script_workflow_start-3.2.1 for obproxy-ce-3.2.1
[2026-07-21 08:53:42.165] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching configserver_pre plugin for components …
[2026-07-21 08:53:42.165] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching configserver_pre plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:42.165] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for oceanbase-ce-py_script_configserver_pre-3.1.0 for oceanbase-ce-4.3.5.4
[2026-07-21 08:53:42.165] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin oceanbase-ce-py_script_configserver_pre-3.1.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:42.166] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import configserver_pre
[2026-07-21 08:53:42.167] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add configserver_pre ref count to 1
[2026-07-21 08:53:42.167] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub configserver_pre ref count to 0
[2026-07-21 08:53:42.167] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export configserver_pre
[2026-07-21 08:53:42.167] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin oceanbase-ce-py_script_configserver_pre-3.1.0 result: True
[2026-07-21 08:53:42.167] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching start_pre plugin for components …
[2026-07-21 08:53:42.167] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching start_pre plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:42.168] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for oceanbase-ce-py_script_start_pre-4.2.5.0 for oceanbase-ce-4.3.5.4
[2026-07-21 08:53:42.168] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin oceanbase-ce-py_script_start_pre-4.2.5.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:42.168] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import start_pre
[2026-07-21 08:53:42.170] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add start_pre ref count to 1
[2026-07-21 08:53:42.171] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [INFO] cluster scenario: complex_oltp
[2026-07-21 08:53:42.171] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – netmarch@172.16.0.253 execute: ls /home/netmarch/data/1/clog/tenant_1/
[2026-07-21 08:53:42.209] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – exited code 0
[2026-07-21 08:53:42.209] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – netmarch@172.16.0.253 execute: cat /home/netmarch/myoceanbase/oceanbase/run/observer.pid
[2026-07-21 08:53:42.256] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – exited code 0
[2026-07-21 08:53:42.256] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – netmarch@172.16.0.253 execute: ls /proc/9934
[2026-07-21 08:53:42.304] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – exited code 0
[2026-07-21 08:53:42.305] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub start_pre ref count to 0
[2026-07-21 08:53:42.305] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export start_pre
[2026-07-21 08:53:42.305] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin oceanbase-ce-py_script_start_pre-4.2.5.0 result: True
[2026-07-21 08:53:42.305] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching start plugin for components …
[2026-07-21 08:53:42.305] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching start plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:42.305] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-4.3.5.4
[2026-07-21 08:53:42.306] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin oceanbase-ce-py_script_start-3.1.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:42.306] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import start
[2026-07-21 08:53:42.306] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add start ref count to 1
[2026-07-21 08:53:42.307] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [INFO] Start observer
[2026-07-21 08:53:42.438] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – need_bootstrap: False
[2026-07-21 08:53:42.438] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub start ref count to 0
[2026-07-21 08:53:42.438] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export start
[2026-07-21 08:53:42.438] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin oceanbase-ce-py_script_start-3.1.0 result: True
[2026-07-21 08:53:42.438] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching health_check plugin for components …
[2026-07-21 08:53:42.438] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching health_check plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:42.439] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for oceanbase-ce-py_script_health_check-3.1.0 for oceanbase-ce-4.3.5.4
[2026-07-21 08:53:42.439] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin oceanbase-ce-py_script_health_check-3.1.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:42.439] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import health_check
[2026-07-21 08:53:42.440] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add health_check ref count to 1
[2026-07-21 08:53:42.440] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [INFO] observer program health check
[2026-07-21 08:53:45.444] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – 172.16.0.253 program health check
[2026-07-21 08:53:45.975] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – netmarch@172.16.0.253 execute: cat /home/netmarch/myoceanbase/oceanbase/run/observer.pid
[2026-07-21 08:53:45.979] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – exited code 0
[2026-07-21 08:53:45.980] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – netmarch@172.16.0.253 execute: ls /proc/9934
[2026-07-21 08:53:46.028] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – exited code 0
[2026-07-21 08:53:46.028] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – 172.16.0.253 observer[pid: 9934] started
[2026-07-21 08:53:46.092] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub health_check ref count to 0
[2026-07-21 08:53:46.093] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export health_check
[2026-07-21 08:53:46.093] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin oceanbase-ce-py_script_health_check-3.1.0 result: True
[2026-07-21 08:53:46.093] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching connect plugin for components …
[2026-07-21 08:53:46.093] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Searching connect plugin for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:46.093] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.3.5.4
[2026-07-21 08:53:46.093] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - Call plugin oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.3.5.4-104000042025090916.el7-a30a15dd7a80f6acd0f113993bb70b2e56d40f80
[2026-07-21 08:53:46.094] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - import connect
[2026-07-21 08:53:46.094] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - add connect ref count to 1
[2026-07-21 08:53:46.095] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [INFO] Connect to observer
[2026-07-21 08:53:46.095] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:53:49.097] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:53:52.100] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:53:55.344] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:53:58.347] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:54:01.350] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:54:04.354] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:54:07.358] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:54:10.361] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:54:13.364] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:54:16.368] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:54:20.268] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:54:23.272] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:54:29.552] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:54:32.556] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:54:35.820] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:54:38.824] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:54:41.828] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:54:44.832] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:54:47.836] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:54:50.840] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:54:53.844] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p

[2026-07-21 08:57:30.418] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:57:36.328] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:57:39.332] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:57:45.908] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:57:48.912] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:57:51.916] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:57:54.920] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:57:57.924] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:00.928] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:58:03.932] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:06.936] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:58:09.940] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:12.944] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:58:15.948] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:19.412] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:58:22.416] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:25.420] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:58:28.424] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:31.428] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:58:34.432] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:37.436] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:58:40.440] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:43.444] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:58:46.449] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:49.452] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:58:52.456] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:58:57.316] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:59:02.002] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:59:05.004] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:59:08.008] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:59:11.012] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:59:17.411] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:59:20.416] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:59:23.420] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p
[2026-07-21 08:59:26.424] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] – mysql connect 172.16.0.253 -P2881 -uroot -p******
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] Traceback (most recent call last):
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “core.py”, line 2277, in start_cluster
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “core.py”, line 2348, in _start_cluster
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “core.py”, line 256, in run_workflow
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “core.py”, line 298, in run_plugin_template
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “core.py”, line 347, in call_plugin
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “_plugin.py”, line 348, in call
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “_plugin.py”, line 304, in _new_func
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 68, in connect
[2026-07-21 08:59:26.426] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] cursor = Cursor(ip=server.ip, port=server_config.get(‘mysql_port’, 2881), tenant=’’, password=****** if password is not None else ‘’, stdio=stdio)
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “_stdio.py”, line 1203, in wrapper
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “_stdio.py”, line 1164, in func_wrapper
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “tool.py”, line 789, in init
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “tool.py”, line 823, in _connect
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “pymysql/connections.py”, line 353, in init
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “pymysql/connections.py”, line 633, in connect
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “pymysql/connections.py”, line 907, in _request_authentication
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “pymysql/connections.py”, line 725, in _read_packet
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “pymysql/protocol.py”, line 221, in raise_for_error
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] File “pymysql/err.py”, line 143, in raise_mysql_exception
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] pymysql.err.OperationalError: (8001, ‘Server is initializing’)
[2026-07-21 08:59:26.427] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR]
[2026-07-21 08:59:29.484] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [ERROR] [ERROR] OBD-1006: Failed to connect to oceanbase-ce
[2026-07-21 08:59:32.389] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - sub connect ref count to 0
[2026-07-21 08:59:32.390] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - export connect
[2026-07-21 08:59:32.390] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - plugin oceanbase-ce-py_script_connect-3.1.0 result: False
[2026-07-21 08:59:32.394] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 3
[2026-07-21 08:59:32.394] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 2
[2026-07-21 08:59:32.394] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[2026-07-21 08:59:32.394] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2026-07-21 08:59:32.395] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2026-07-21 08:59:32.395] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - exclusive lock /root/.obd/lock/deploy_myoceanbase release, count 0
[2026-07-21 08:59:32.395] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - unlock /root/.obd/lock/deploy_myoceanbase
[2026-07-21 08:59:32.395] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2026-07-21 08:59:32.395] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [DEBUG] - unlock /root/.obd/lock/global
[2026-07-21 08:59:32.395] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [INFO] See 错误码及解决方案 - ob-deployer - OceanBase .
[2026-07-21 08:59:32.395] [aa17ad6e-84e1-11f1-87ca-000c2988f524] [INFO] Trace ID: aa17ad6e-84e1-11f1-87ca-000c2988f524

3 个赞
#7

我截取的够吗，不够我换个号发完整的

3 个赞
#8

为啥不发个TXT文档呢

3 个赞
#9

obd.txt (931.3 KB)
谢谢提醒，这是日志

3 个赞
#10

围观学习

3 个赞
#11

ps -ef | grep observer

ss -lntp | grep 2881
提供一下 启动时间的observer.log日志

3 个赞
#12

observer.txt (30.0 MB)

77A25799-A7E3-425e-8E1D-B6D065E91A77
77A25799-A7E3-425e-8E1D-B6D065E91A771581×235 28.2 KB

是这些

2 个赞
#13

服务重新关闭 查一下进程是否还在 如果在kill 掉
ps -ef | grep observer

ss -lntp | grep 2881
在启动一下 发一下obd.log observer.log日志 这个日志不全 无法判断具体的问题
看着是 slog回放没有完成 进程在服务不可用
slog replay hasn’t finished, cannot start loop task

ObTmpFileFlushTG does not work before server slog replay finished

2 个赞
#14

observer.log.gz (4.1 MB)
obd.txt (244.0 KB)

8C3FB296-CB47-4e7b-9C46-2E61B7BB52B0
8C3FB296-CB47-4e7b-9C46-2E61B7BB52B0993×269 19.4 KB

这是单次启动的日志，因为observer启动日志增长很快，只获取了启动到卡在Connect to observer差不多时间的日志

1 个赞
#15

启动的过程你是不是有 Ctrl+C 中断操作呀

1 个赞
#16

是的，不过中断后面的obd日志应该和前面发出来差不多，是observer日志需要继续写入直到启动结束报错那时吗？

1 个赞
#17

– 1. 正常停止（不要 kill -9）
obd cluster stop myoceanbase

– 2. 确认进程已退出
ps -ef | grep observer

– 3. 确认内存充足后启动，并耐心等待
obd cluster start myoceanbase

  1. 不要再 Ctrl+C，让 observer 完整恢复（大数据量可能要 10～30 分钟甚至更久）
  2. 先释放内存，或把 memory_limit 从 43G 降到 30G～32G（与 36G 空闲内存匹配）
    报错信息 OBD-2000: not enough memory (Free: 36G, Need: 43G)
    配置 memory_limit=43G ，但机器空闲仅约 36G。日志里还有 sys_total_wash_size=-39804608512 （约 -37GB），说明内存已明显吃紧，会拖慢恢复。
  3. 用下面命令监控恢复进度：
    –观察 SLOG 回放和 schema 恢复
    tail -f /home/netmarch/myoceanbase/oceanbase/log/observer.log |
    grep -E “slog replay|schema|serving|8001|finish replay”

– 进程和端口
ps -ef | grep observer
ss -lntp | grep -E ‘2881|2882’
4. 恢复完成后再测连接：
mysql -h172.16.0.253 -P2881 -uroot -p -e “select 1”

直接报错信息

tenant schema not refreshed yet (ret=-8001, OB_SERVER_IS_INIT)

server is not serving (ret=-8001, GCTX.status_=1)

tenant schema is not ready, need wait (ret=-4076)

OB_LS_LOCATION_NOT_EXIST (ret=-4721)

SCHEMA NOT REFRESHED (level:SERIOUS)

2 个赞
#18

谢谢，我去试试

1 个赞
#19

给国产数据库多点时间成长 。

#20

分析的很详细啊，学习学习

#21

6666