docker OBD-1006: Failed to connect to oceanbase-ce

OceanBase 技术问题
#1

【 使用版本 】ob 4.3.5.0
【问题描述】

  1. ob服务启动通过docker run
sudo docker run -p 2881:2881 \
  --name obstandalone \
  -e MODE=MINI \
  -e OB_TENANT_PASSWORD=ob123456 \
  -e OB_SYS_PASSWORD=ob123456 \
  --cpus="4" \
  --memory="8g" \
  -v /data/zhangx/oceanbase:/var/lib/oceanbase \
  -d quay.io/oceanbase/oceanbase-ce:4.3.5.1-101000042025031818

2.之前一直正常运行，今天服务器重启了，再重启ob docker 服务时候提示报错，因为之前docker未固定ip（172.17.0.6），后面启动生成了新的ip（172.17.0.2），接着把docker内 ~/.obd/cluster/obcluster 内文件ip改为了新的ip 172.17.0.6 =》 172.17.0.2
，config.yaml 如下

oceanbase-ce:
  servers:
  - 172.17.0.2
  global:
    home_path: /root/ob
    mysql_port: 2881
    rpc_port: 2882
    zone: zone1
    cluster_id: 1
    appname: obcluster
    memory_limit: 6G
    system_memory: 1G
    datafile_size: 5G
    log_disk_size: 5G
    root_password: ob123456
    scenario: express_oltp
    obconfig_url:
    cpu_count: 16
    production_mode: false
    syslog_level: INFO
    enable_syslog_wf: false
    enable_syslog_recycle: true
    max_syslog_file_count: 4
    enable_rich_error_msg: true
  1. 改好后重启docker，提示新的错误
[ERROR] OBD-1006: Failed to connect to oceanbase-ce


[2025-04-18 10:23:55.123] [DEBUG] -- starting 172.17.0.2 observer
[2025-04-18 10:23:55.123] [DEBUG] -- root@172.17.0.2 export LD_LIBRARY_PATH='/root/ob/lib:'
[2025-04-18 10:23:55.123] [DEBUG] -- root@172.17.0.2 execute: cd /root/ob; /root/ob/bin/observer -p 2881 -P 2882 -z 'zone1' -n 'obcluster' -c 1 -d '/root/ob/store' -l 'INFO' -I '172.17.0.2' -o __min_full_resource_pool_memory=2147483648,memory_limit='6G',system_memory='1G',datafile_size='5G',log_disk_size='5G',cpu_count=16,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,enable_rich_error_msg=True,enable_record_trace_log=False 
[2025-04-18 10:23:55.208] [DEBUG] -- exited code 0
[2025-04-18 10:23:55.208] [DEBUG] -- root@172.17.0.2 delete env LD_LIBRARY_PATH
[2025-04-18 10:23:55.208] [DEBUG] -- need_bootstrap: False
[2025-04-18 10:23:55.208] [DEBUG] - sub start ref count to 0
[2025-04-18 10:23:55.208] [DEBUG] - export start
[2025-04-18 10:23:55.208] [DEBUG] - plugin oceanbase-ce-py_script_start-3.1.0 result: True
[2025-04-18 10:23:55.208] [DEBUG] - Searching health_check plugin for components ...
[2025-04-18 10:23:55.208] [DEBUG] - Searching health_check plugin for oceanbase-ce-4.3.5.0-100000202024123117.el8-a81cd6413fc9c42f14fdf71c55fc319dfa6cab60
[2025-04-18 10:23:55.209] [DEBUG] - Found for oceanbase-ce-py_script_health_check-3.1.0 for oceanbase-ce-4.3.5.0
[2025-04-18 10:23:55.209] [DEBUG] - Call plugin oceanbase-ce-py_script_health_check-3.1.0 for oceanbase-ce-4.3.5.0-100000202024123117.el8-a81cd6413fc9c42f14fdf71c55fc319dfa6cab60
[2025-04-18 10:23:55.209] [DEBUG] - import health_check
[2025-04-18 10:23:55.209] [DEBUG] - add health_check ref count to 1
[2025-04-18 10:23:55.209] [INFO] observer program health check
[2025-04-18 10:23:58.212] [DEBUG] -- 172.17.0.2 program health check
[2025-04-18 10:23:58.212] [DEBUG] -- root@172.17.0.2 execute: cat /root/ob/run/observer.pid 
[2025-04-18 10:23:58.217] [DEBUG] -- exited code 0
[2025-04-18 10:23:58.218] [DEBUG] -- root@172.17.0.2 execute: ls /proc/202 
[2025-04-18 10:23:58.265] [DEBUG] -- exited code 0
[2025-04-18 10:23:58.265] [DEBUG] -- 172.17.0.2 observer[pid: 202] started
[2025-04-18 10:23:58.265] [DEBUG] - sub health_check ref count to 0
[2025-04-18 10:23:58.266] [DEBUG] - export health_check
[2025-04-18 10:23:58.266] [DEBUG] - plugin oceanbase-ce-py_script_health_check-3.1.0 result: True
[2025-04-18 10:23:58.266] [DEBUG] - Searching connect plugin for components ...
[2025-04-18 10:23:58.266] [DEBUG] - Searching connect plugin for oceanbase-ce-4.3.5.0-100000202024123117.el8-a81cd6413fc9c42f14fdf71c55fc319dfa6cab60
[2025-04-18 10:23:58.267] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.3.5.0
[2025-04-18 10:23:58.267] [DEBUG] - Call plugin oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.3.5.0-100000202024123117.el8-a81cd6413fc9c42f14fdf71c55fc319dfa6cab60
[2025-04-18 10:23:58.267] [DEBUG] - import connect
[2025-04-18 10:23:58.268] [DEBUG] - add connect ref count to 1
[2025-04-18 10:23:58.268] [INFO] Connect to observer
[2025-04-18 10:23:58.268] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:01.273] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:04.276] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:07.280] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:10.283] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:13.286] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:16.290] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:19.293] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:22.297] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:25.300] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:28.304] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:31.307] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:34.311] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:37.314] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:40.317] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:43.320] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:46.323] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:49.326] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:52.329] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:24:55.332] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:24:58.335] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:01.339] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:04.342] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:07.346] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:10.349] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:13.353] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:16.357] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:19.360] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:22.363] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:25.367] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:28.370] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:31.374] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:34.377] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:37.381] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:40.384] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:43.387] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:46.391] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:49.394] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:52.397] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:25:55.401] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:25:58.404] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:01.407] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:04.410] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:07.414] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:10.417] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:13.421] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:16.424] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:19.427] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:22.431] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:25.434] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:28.438] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:31.441] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:34.445] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:37.448] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:40.451] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:43.454] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:46.458] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:49.461] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:52.464] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:26:55.468] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:26:58.471] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:01.474] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:04.478] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:07.481] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:10.484] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:13.487] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:16.491] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:19.494] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:22.497] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:25.500] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:28.504] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:31.507] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:34.510] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:37.514] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:40.517] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:43.521] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:46.524] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:49.527] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:52.531] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:27:55.534] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:27:58.538] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:01.541] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:04.544] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:07.547] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:10.551] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:13.554] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:16.557] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:19.561] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:22.564] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:25.567] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:28.571] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:31.574] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:34.577] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:37.581] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:40.584] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:43.588] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:46.591] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:49.595] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:52.598] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:55.602] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p
[2025-04-18 10:28:58.605] [DEBUG] -- connect 172.17.0.2 -P2881 -uroot -p******
[2025-04-18 10:28:58.605] [ERROR] Traceback (most recent call last):
[2025-04-18 10:28:58.605] [ERROR]   File "core.py", line 2090, in start_cluster
[2025-04-18 10:28:58.606] [ERROR]   File "core.py", line 2140, in _start_cluster
[2025-04-18 10:28:58.606] [ERROR]   File "core.py", line 228, in run_workflow
[2025-04-18 10:28:58.606] [ERROR]   File "core.py", line 270, in run_plugin_template
[2025-04-18 10:28:58.606] [ERROR]   File "core.py", line 315, in call_plugin
[2025-04-18 10:28:58.606] [ERROR]   File "_plugin.py", line 347, in __call__
[2025-04-18 10:28:58.606] [ERROR]   File "_plugin.py", line 304, in _new_func
[2025-04-18 10:28:58.606] [ERROR]   File "/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py", line 69, in connect
[2025-04-18 10:28:58.606] [ERROR]     cursor = Cursor(ip=server.ip, port=server_config.get('mysql_port', 2881), tenant='', password=****** if password is not None else '', stdio=stdio)
[2025-04-18 10:28:58.606] [ERROR]   File "_stdio.py", line 1012, in wrapper
[2025-04-18 10:28:58.606] [ERROR]   File "_stdio.py", line 999, in func_wrapper
[2025-04-18 10:28:58.606] [ERROR]   File "tool.py", line 763, in __init__
[2025-04-18 10:28:58.606] [ERROR]   File "tool.py", line 793, in _connect
[2025-04-18 10:28:58.606] [ERROR]   File "pymysql/connections.py", line 353, in __init__
[2025-04-18 10:28:58.606] [ERROR]   File "pymysql/connections.py", line 633, in connect
[2025-04-18 10:28:58.606] [ERROR]   File "pymysql/connections.py", line 907, in _request_authentication
[2025-04-18 10:28:58.606] [ERROR]   File "pymysql/connections.py", line 725, in _read_packet
[2025-04-18 10:28:58.606] [ERROR]   File "pymysql/protocol.py", line 221, in raise_for_error
[2025-04-18 10:28:58.606] [ERROR]   File "pymysql/err.py", line 143, in raise_mysql_exception
[2025-04-18 10:28:58.606] [ERROR] pymysql.err.OperationalError: (8001, 'Server is initializing\n[0.0.0.0:0] [2025-04-18 10:28:58.605513] [Y0-0006330AEE616957-0-0]')
[2025-04-18 10:28:58.606] [ERROR] 
[2025-04-18 10:29:01.609] [ERROR] OBD-1006: Failed to connect to oceanbase-ce
[2025-04-18 10:29:01.609] [DEBUG] - sub connect ref count to 0
[2025-04-18 10:29:01.609] [DEBUG] - export connect
[2025-04-18 10:29:01.609] [DEBUG] - plugin oceanbase-ce-py_script_connect-3.1.0 result: False
[2025-04-18 10:29:01.610] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[2025-04-18 10:29:01.610] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2025-04-18 10:29:01.610] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2025-04-18 10:29:01.610] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obcluster release, count 0
[2025-04-18 10:29:01.610] [DEBUG] - unlock /root/.obd/lock/deploy_obcluster
[2025-04-18 10:29:01.610] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2025-04-18 10:29:01.610] [DEBUG] - unlock /root/.obd/lock/global
[2025-04-18 10:29:01.610] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2025-04-18 10:29:01.610] [INFO] Trace ID: 3a9a50ba-1c3f-11f0-9df7-0242ac110002
[2025-04-18 10:29:01.610] [INFO] If you want to view detailed obd logs, please run: obd display-trace 3a9a50ba-1c3f-11f0-9df7-0242ac110002

在docker内验证过密码变量和之前的一样
问：
1.是重启服务器导致ob的密码变了么
2.针对这种情况如何解决（修复？再起一个服务把数据迁移过去？重新重置密码？）具体如何做，谢谢

#3

observer.log中发现有还是连接老的ip 172.17.0.6，要修改哪里呢

[2025-04-18 14:15:41.418465] WDIAG [SHARE.PT] do_detect_master_rs_ls_ (ob_rpc_ls_table.cpp:296) [492][T1_TimerWK3][T1][YB42AC110002-0006330E06D8EC4E-0-0] [lt=2][errcode=0] fail to send rpc(tmp_ret=-4122, tmp_ret="OB_RPC_POST_ERROR", cluster_id=1, addr="172.17.0.6:2882", timeout=10305932, arg={addr:"172.17.0.6:2882", cluster_id:1})
#4

自问自答了，自己解决了，希望对有需要的人有帮助

1.容器重启后ip变更报错 [ERROR] OBD-1013: root@172.17.0.6 connect failed: time out

2.解决方式 将容器ip改为原来的ip，改配置的话坑比较多

创建个自己的网络

docker network create --subnet=172.17.0.0/16 mynet1

把原有的断网

docker network disconnect bridge 1f86eea88ccf

ip改为之前的

docker network connect --ip 172.17.0.6 mynet1 1f86eea88ccf

在验证在ip生效没有

docker inspect 1f86eea88ccf | grep “IPAddress”

3.重启容器，完活