oceanbase 分布式集群 bootstrap失败

OceanBase 技术问题
#1

【 使用环境 】生产环境 or 测试环境
【 OB or 其他组件 】obd部署分布式集群
【 使用版本 】4.3.5
【问题描述】obd cluster start xxx集群时，cluster bootstrap超时
【复现路径】问题出现前后相关操作
【附件及日志】
yaml配置如下：
user:
username: admin
password: admin
oceanbase-ce:
servers:

  • name: server1
    ip: 192.168.2.130
  • name: server2
    ip: 192.168.2.62
  • name: server3
    ip: 192.168.2.109
  • name: server4
    ip: 192.168.2.198
    global:
    production_mode: false
    devname: enp193s0f3
    cluster_id: 1
    system_memory: 6G
    datafile_size: 10G # Size of the data file.
    log_disk_size: 2G # The size of disk space used by the clog files.
    syslog_level: INFO # System log level. The default value is INFO.
    enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
    enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
    max_syslog_file_count: 4 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    appname: obdist
    root_password: Tt2uRvLpjueQdv3xeNjM
    ocp_agent_monitor_password: 1wGyiR9yOq
    proxyro_password: h2wzHw12Oy
    server1:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
    home_path: /home/admin/oceanbase-ce
    data_dir: /mnt/ssd2/zj/data
    redo_dir: /mnt/ssd2/zj/redo
    zone: zone1
    server2:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
    home_path: /home/admin/oceanbase-ce
    data_dir: /mnt/ssd2/zj/data
    redo_dir: /mnt/ssd2/zj/redo
    zone: zone2
    server3:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
    home_path: /home/admin/oceanbase-ce
    data_dir: /mnt/ssd2/zj/data
    redo_dir: /mnt/ssd2/zj/redo
    zone: zone3
    server4:
    mysql_port: 2881 # External port for OceanBase Database. The default value is 2881. DO NOT change this value after the cluster is started.
    rpc_port: 2882 # Internal port for OceanBase Database. The default value is 2882. DO NOT change this value after the cluster is started.
    home_path: /home/admin/oceanbase-ce
    data_dir: /mnt/ssd2/zj/data
    redo_dir: /mnt/ssd2/zj/redo
    zone: zone4
    obproxy-ce:
    depends:
  • oceanbase-ce
    servers:
  • 192.168.2.130
  • 192.168.2.62
  • 192.168.2.109
  • 192.168.2.198
    global:
    listen_port: 2883 # External port. The default value is 2883.
    prometheus_listen_port: 2884 # The Prometheus port. The default value is 2884.
    home_path: /home/admin/obproxy
    enable_cluster_checkout: false
    cluster_name: obcluster
    skip_proxy_sys_private_check: true
    enable_strict_kernel_release: false
    obproxy_sys_password: PMIdynuRzD
    obagent:
    depends:
  • oceanbase-ce
    servers:
  • name: server1
    ip: 192.168.2.130
  • name: server2
    ip: 192.168.2.62
  • name: server3
    ip: 192.168.2.109
  • name: server4
    ip: 192.168.2.198
    global:
    home_path: /home/admin/obagent
    ob_monitor_status: active
    http_basic_auth_password: hRQlEw096
    prometheus:
    depends:
  • obagent
    servers:
  • 192.168.2.130
    global:
    home_path: /home/zj/prometheus
    basic_auth_users:
    admin: RhnapIz6Np
    grafana:
    depends:
  • prometheus
    servers:
  • 192.168.2.130
    global:
    home_path: /home/zj/grafana
    login_password: oceanbase

同时发现，observer的2881端口不通
nc -zv 192.168.2.109 2881
Ncat: Version 7.92 ( https://nmap.org/ncat )
Ncat: TIMEOUT.

【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试，在发帖时输入 [@论坛小助手] 即可召唤小助手，欢迎试用！

#3

根据文档库中的信息，以下是可能导致 cluster bootstrap 超时以及 observer 的 2881 端口不通的原因及解决方法：

1. 端口冲突或被占用

  • 原因：如果配置文件中存在端口冲突或端口已被其他进程占用，则可能导致 bootstrap 失败。
  • 解决方法：
    • 检查所有节点的 2881 和 2882 端口是否被占用。可以使用以下命令检查端口占用情况：
      netstat -tuln | grep 2881
netstat -tuln | grep 2882
    • 如果发现端口被占用，可以选择结束该端口的进程，或者更换为未被占用的端口。参考文档片段 2 中的方法一和方法二进行端口修改。

2. 网络连通性问题

  • 原因：如果集群节点之间的网络不可达，observer 无法正常通信，可能导致 bootstrap 超时。
  • 解决方法：
    • 检查节点间的网络连通性。可以使用 ping 或 nc 命令测试节点间网络是否可达：
      ping 192.168.2.109
nc -zv 192.168.2.109 2881
    • 如果网络不可达，请检查防火墙规则或网络配置，确保所有节点之间可以通过指定端口通信。

3. 配置文件问题

  • 原因：配置文件中的某些参数设置不当可能导致 bootstrap 失败。例如，devname 参数可能配置错误，导致网络接口选择不正确。
  • 解决方法：
    • 检查 yaml 配置文件中的 devname 参数是否正确。确保每个节点的 devname 对应实际可用的网络接口。
    • 使用以下命令查看节点的网络接口列表，并确认 devname 是否正确：
      ip addr show

4. 集群 ID 不匹配

  • 原因：如果多个集群共享相同的 cluster_id，可能会导致 bootstrap 失败。
  • 解决方法：
    • 确保所有节点的 cluster_id 一致且唯一。可以在 yaml 配置文件中统一设置 cluster_id 参数：
      global:
  cluster_id: 1
    • 如果发现 cluster_id 不匹配，参考文档片段 5 的解决方式，停止部署错误的集群。

5. 资源不足

  • 原因：如果节点的 CPU、内存或磁盘资源不足，可能导致 bootstrap 过程超时。
  • 解决方法：
    • 检查各节点的资源使用情况。可以使用以下命令查看资源状态：
      free -m  # 查看内存使用情况
df -h    # 查看磁盘使用情况
top      # 查看 CPU 使用情况
    • 如果资源不足，请释放资源或调整配置文件中的 system_memory、datafile_size 和 log_disk_size 参数。

6. 日志分析

  • 原因：通过分析 obd 日志可以进一步定位问题。
  • 解决方法：
    • 查看 obd 的详细日志以获取更多错误信息：
      obd log <deploy_name>
    • 根据日志中的具体错误信息采取相应的解决措施。

总结

根据上述分析，请按照以下步骤逐步排查问题：

  1. 检查端口占用情况并确保网络连通性。
  2. 核对配置文件中的参数设置是否正确。
  3. 确保 cluster_id 唯一且一致。
  4. 检查节点资源是否充足。
  5. 分析 obd 日志以获取更多错误信息。

如果以上方法仍无法解决问题，请提供详细的日志信息以便进一步分析。

更多信息请查看:

  1. 错误码
  2. OceanBase 安装部署工具 V2.8.0
  3. 使用 obd 接管集群
  4. 8000 ~ 8999
  5. 使用 OCP 进行运维
  6. 容灾架构及容灾级别
  7. 监控告警
1 个赞
#5

log 如下

[2025-03-13 17:04:52.853889] INFO [SQL.RESV] resolve (ob_bootstrap_resolver.cpp:116) [942805][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=5] shared_storage_node is NULL(ret=0, ret=“OB_SUCCESS”)
[2025-03-13 17:04:52.853906] INFO [SQL.ENG] execute (ob_alter_system_executor.cpp:1803) [942805][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=8] bootstrap timeout(rpc_timeout=999946223)
[2025-03-13 17:04:52.853988] INFO [SERVER] bootstrap (ob_service.cpp:1601) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=33] bootstrap timeout(timeout=600000000, worker_timeout_ts=1741857692800168)
[2025-03-13 17:04:52.854085] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=8][errcode=-4122] check_blacklist failed(ret=-4122)
[2025-03-13 17:04:52.854093] WDIAG [BOOTSTRAP] check_all_server_bootstrap_mode_match (ob_bootstrap.cpp:559) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=6][errcode=-4122] fail to check deployment mode match(ret=-4122)
[2025-03-13 17:04:52.854095] WDIAG [BOOTSTRAP] check_all_server_bootstrap_mode_match (ob_bootstrap.cpp:566) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=2][errcode=-4122] STEP_2.1:check_all_server_bootstrap_mode_match execute fail, ret=-4122, cost=99
[2025-03-13 17:04:52.854098] WDIAG [BOOTSTRAP] prepare_bootstrap (ob_bootstrap.cpp:267) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=2][errcode=-4122] fail to check all server bootstrap mode match(ret=-4122, ret=“OB_RPC_POST_ERROR”)
[2025-03-13 17:04:52.854104] WDIAG [BOOTSTRAP] prepare_bootstrap (ob_bootstrap.cpp:293) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=6][errcode=-4122] STEP_2.2:prepare_bootstrap execute fail, ret=-4122, cost=9
[2025-03-13 17:04:52.854107] EDIAG [BOOTSTRAP] bootstrap (ob_service.cpp:1631) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=2][errcode=-4122] failed to prepare boot strap(rs_list=[region:“sys_region”, zone:“zone1”, server:“192.168.2.130:2882”, region:“sys_region”, zone:“zone2”, server:“192.168.2.62:2882”, region:“sys_region”, zone:“zone3”, server:“192.168.2.109:2882”, region:“sys_region”, zone:“zone4”, server:“192.168.2.198:2882”], ret=-4122) BACKTRACE:0x11df612c 0x1191a664 0x11c1ef2c 0x11c1e6ec 0x11c1e37c 0x11c1e844 0xd741534 0xd7401b8 0xcf92d24 0x11ea8dcc 0x1169ff20 0x11690164 0xc97f0b4 0x1c4be8bc 0xffffa50a5138 0xffffa510c6dc
[2025-03-13 17:04:52.854153] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [942805][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=7][errcode=-4122] execute rpc fail(addr=“192.168.2.130:2882”, pcode=257, ret=-4122, timeout=999946223)
[2025-03-13 17:04:52.854164] WDIAG log_user_error_and_warn (ob_poc_rpc_proxy.cpp:249) [942805][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=9][errcode=-4122]
[2025-03-13 17:04:52.854169] WDIAG [SQL.ENG] execute (ob_alter_system_executor.cpp:1811) [942805][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=3][errcode=-4122] rpc proxy bootstrap failed(ret=-4122, rpc_timeout=999946223)

1 个赞
#6

先看下yaml配置问题log_disk_size太小了，麻烦修改下。
存在rpc问题，看一下所有节点防火墙是否都关闭了端口问题需进行打通

#7

根据您提供的日志信息，以下是导致 cluster bootstrap 超时的具体原因及解决方法：

1. bootstrap timeout 错误

  • 日志分析：
    [2025-03-13 17:04:52.853906] INFO  [SQL.ENG] execute (ob_alter_system_executor.cpp:1803) [942805][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=8] bootstrap timeout(rpc_timeout=999946223)
[2025-03-13 17:04:52.853988] INFO  [SERVER] bootstrap (ob_service.cpp:1601) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=33] bootstrap timeout(timeout=600000000, worker_timeout_ts=1741857692800168)
    • 日志显示 bootstrap 过程超时，可能是由于节点间通信延迟或网络问题导致。
  • 解决方法：
    • 检查所有节点之间的网络连通性，确保节点间的 RPC 端口（默认为 2882）可以正常通信。
    • 使用以下命令测试节点间端口连通性：
      nc -zv <IP> 2882
    • 如果发现网络延迟较高或连接失败，请优化网络配置或调整防火墙规则。

2. check_blacklist failed 错误

  • 日志分析：
    [2025-03-13 17:04:52.854085] WDIAG [RPC] send (ob_poc_rpc_proxy.h:150) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=8][errcode=-4122] check_blacklist failed(ret=-4122)
    • 日志显示在检查黑名单时失败，可能是因为某些节点被错误地列入了黑名单。
  • 解决方法：
    • 确保所有节点的配置文件中没有将其他节点列入黑名单。
    • 检查 yaml 配置文件中的 depends 和 servers 部分，确保所有节点都被正确包含。

3. check_all_server_bootstrap_mode_match 错误

  • 日志分析：
    [2025-03-13 17:04:52.854093] WDIAG [BOOTSTRAP] check_all_server_bootstrap_mode_match (ob_bootstrap.cpp:559) [942806][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=6][errcode=-4122] fail to check deployment mode match(ret=-4122)
    • 日志显示在检查所有节点的启动模式是否匹配时失败，可能是由于某些节点的配置不一致。
  • 解决方法：
    • 确保所有节点的 production_mode 参数设置一致。例如，在 yaml 配置文件中统一设置为 false 或 true：
      global:
  production_mode: false
    • 如果某些节点已经运行在不同的模式下，请停止这些节点并重新部署。

4. execute rpc fail 错误

  • 日志分析：
    [2025-03-13 17:04:52.854153] WDIAG [RPC] send (ob_poc_rpc_proxy.h:176) [942805][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=7][errcode=-4122] execute rpc fail(addr="192.168.2.130:2882", pcode=257, ret=-4122, timeout=999946223)
    • 日志显示在执行 RPC 请求时失败，可能是由于目标节点未正常启动或网络问题。
  • 解决方法：
    • 检查目标节点（如 192.168.2.130）是否正常启动，并确保其监听的端口（2882）可以被访问。
    • 使用以下命令检查目标节点的状态：
      obd cluster status <deploy_name>
    • 如果发现目标节点未启动，请尝试重新启动该节点：
      obd server start <server_name>

5. shared_storage_node is NULL 提示

  • 日志分析：
    [2025-03-13 17:04:52.853889] INFO  [SQL.RESV] resolve (ob_bootstrap_resolver.cpp:116) [942805][T1_L0_G0][T1][YB42C0A80282-000630359D1B58EE-0-0] [lt=5] shared_storage_node is NULL(ret=0, ret="OB_SUCCESS")
    • 日志提示共享存储节点为空，这通常与分布式集群的存储配置有关。
  • 解决方法：
    • 确保所有节点的 data_dir 和 redo_dir 配置正确，并且路径可用。
    • 检查磁盘空间是否充足：
      df -h
    • 如果使用共享存储，请确保所有节点可以正确访问共享存储路径。

总结

根据上述分析，请按照以下步骤逐步排查问题：

  1. 检查所有节点之间的网络连通性，确保 RPC 端口（2882）可以正常通信。
  2. 确保所有节点的配置文件一致，特别是 production_mode 和 depends 部分。
  3. 检查目标节点是否正常启动，并确保其监听的端口可以被访问。
  4. 确保所有节点的存储路径配置正确，并且磁盘空间充足。
  5. 如果问题仍未解决，请提供更详细的日志信息以便进一步分析。

如果需要更多帮助，可以联系 OceanBase 的人工答疑团队获取支持。

详情请参考:

  1. 排序 limit 场景性能优化
  2. 命令行参数介绍
  3. 按表恢复相关视图介绍

(小助手的答复已结束，如未能解决您的问题，请继续提问并等待其他同学的回复，谢谢！)

1 个赞
#8

那这是搞了4台服务器，搞4个zone？要搞奇数个节点，搞3台

1 个赞
#9

所有节点防火墙均未启用，同时我调大了log_disk_size，并将节点数量修改为了三个，其中一台作为中控机，在start集群时卡在了connect to observer阶段

image
image731×201 14.4 KB

image
image732×192 9.67 KB

image
image
image726×187 9.03 KB

image
image998×349 32.8 KB

obd日志显示不断的尝试连接，但是最终超时了

1 个赞
#10

该问题已解决，ibmc 2881端口未打开，导致server未连上

1 个赞
#11

当我完成connect to observer以及cluster bootstrap后，
当我进行obshell bootstrap时，出现了卡顿，应当如何解决？

obd日志：

image
image2081×245 25.7 KB

1 个赞
#13

image
image1456×250 13.1 KB

1 个赞
#14

麻烦发一份完整的obd日志+obshell日志
obshell日志： observer的log目录里有个log_obshell目录，里面的就是 obshell 的日志。

1 个赞
#15

某个节点的obshell日志：
2025-03-14T15:47:10.274 INFO [1156691] [F000000000000000] [cmd/handle.go:148] current obshell version is 4.2.5.0-12024121011
2025-03-14T15:47:10.274 INFO [1156691] [F000000000000000] [cmd/handle.go:171] check backup binary /home/admin/oceanbase-ce/etc/obshell
2025-03-14T15:47:10.291 INFO [1156691] [F000000000000000] [cmd/handle.go:182] backup binary version is 4.2.5.0-12024121011
2025-03-14T15:47:10.307 INFO [1156691] [F000000000000000] [server/init.go:81] initialize logger
2025-03-14T15:47:10.307 INFO [1156691] [F000000000000000] [global/variable.go:70] homePath is /home/admin/oceanbase-ce
2025-03-14T15:47:10.307 INFO [1156691] [F000000000000000] [global/variable.go:63] architecture is aarch64
2025-03-14T15:47:10.308 INFO [1156691] [F000000000000000] [server/init.go:266] Check if obshell is in upgrade mode.
2025-03-14T15:47:10.308 INFO [1156691] [F000000000000000] [server/init.go:88] initialize sqlite
2025-03-14T15:47:10.308 INFO [1156691] [F000000000000000] [sqlite/builder.go:67] open sqlite succeed
2025-03-14T15:47:10.369 INFO [1156691] [F000000000000000] [server/init.go:142] initialize agent
2025-03-14T15:47:10.370 INFO [1156691] [F000000000000000] [server/init.go:146] meta from sqlite is :0
2025-03-14T15:47:10.370 INFO [1156691] [F000000000000000] [ob/etc.go:57] load ob config from config file
2025-03-14T15:47:10.370 INFO [1156691] [F000000000000000] [ob/etc.go:83] get conf from ob conf file /home/admin/oceanbase-ce/etc/observer.config.bin
2025-03-14T15:47:10.370 INFO [1156691] [F000000000000000] [ob/etc.go:118] get conf from ob conf file, ip: 192.168.2.109, zone: zone2, mysqlPort: 10000, rpcPort: 10001
2025-03-14T15:47:10.372 INFO [1156691] [F000000000000000] [server/init.go:197] check agent info
2025-03-14T15:47:10.374 INFO [1156691] [F000000000000000] [server/init.go:167] initialize agent status
2025-03-14T15:47:10.377 INFO [1156691] [F000000000000000] [server/init.go:172] update base info
2025-03-14T15:47:10.378 INFO [1156691] [F000000000000000] [server/init.go:229] init server
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [server/init.go:236] server config is {0.0.0.0 2886 0.0.0.0:2886 /home/admin/oceanbase-ce/run false}
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CreateSubStartDagTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckSubStartDagReadyTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:RetrySubStartDagTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitSubStartDagFinishTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:StartZoneTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:PassSubStartDagTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckDagStageTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckObserverForStartTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AlterStartServerTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitPassOperatorTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:GetConnForEStartTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CreateSubStopDagTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckSubStopDagReadyTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:RetrySubStopDagTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitSubStopDagFinishTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:PassSubStopDagTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckDagStageTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ExecStopSqlTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitPassOperatorTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:MinorFreezeTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:UpdateOBClusterConfigTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:UpdateOBServerConfigTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:IntegrateObConfigTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:DeployTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:DestroyTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:StartObserverTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:StopObserverTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ClusterBoostrapTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:MigrateTableTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ModifyPwdTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:MigrateDataTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ConvertFollowerToClusterAgentTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AgentSyncTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ConvertMasterToClusterAgentTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AgentBeScalingOutTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitDeployRetryTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitStartRetryTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WatchDagTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ScalingAgentUpdateBinaryTask
025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CreateLocalScaleOutDagTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitScalingReadyTask
2025-03-14T15:47:10.379 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitRemoteDeployTaskFinish
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitRemoteStartTaskFinish
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:PrevCheckTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AddNewZoneTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:StartNewZoneTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AddServerTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AddAgentTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:FinishTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:SetAgentToScaleInTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:DeleteAgentTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:TryToInformToKillObserverTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:TryToInformToKillObserversTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:KillObserverTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckMultiPaxosMemberAliveTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:DeleteObserverTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitDeleteServerSuccessTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:DeleteZoneTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:StartObserverForScaleInRollbackTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CreateUpgradeDirTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:GetAllRequiredPkgsTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckAllRequiredPkgsTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:InstallAllRequiredPkgsTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:RemoveUpgradeCheckDirTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:BackupAgentForUpgradeTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:InstallNewAgentTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:RestartAgentTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:UpgradePostTableMaintainTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:UpgradeToClusterAgentVersionTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:TakeOverAgentUpdateBinaryTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckEnvTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:BackupParametersTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ExecScriptTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:StopZoneTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ReinstallAndRestartObTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:StartOneZoneTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:RestoreParametersTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:SetBackupConfigTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckBackupConfigTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CheckDestTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:OpenArchiveLogTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:StartBackupTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitBackupTaskFinish
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:PreRestoreCheckTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:StartRestoreTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitRestoreFinshTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ActiveTenantTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:UpgradeTenantTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CancelRestoreTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:DropResourcePoolTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AgentJoinSelfTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AgentJoinMasterTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AgentBeFollowerTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AgentToSingleTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:RemoveFollowerAgentTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AgentRemoveFollowerRPCTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AgentRemoveMasterTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:SendFollowerRemoveSelfRPCTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:CreateTenantTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ModifyPrimaryZoneTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:SetRootPwdTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:SetTenantTimeZoneTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:SetTenantParamterTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:OptimizeTenantTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:DropTenantTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:RecycleTenantTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:BatchCreateResourcePoolTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AlterLocalityTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:SplitResourcePoolTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:BatchDropResourcePoolTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AlterResourcePoolUnitNumTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:AlterResourcePoolUnitConfTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ModifyTenantWhitelistTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:WaitForPurgeFinishedTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:ImportScriptForTenantTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [task/task.go:537] Register Task:DropResourcePoolTask
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [engine/enter.go:37] local task engine starting …
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [engine/enter.go:46] local task engine started
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [web/server.go:123] listen unix socket on /home/admin/oceanbase-ce/run/obshell.sock
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [server/run.go:73] restore secure info
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [secure/crypto.go:72] restore private key from sqlite successed
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [server/run.go:83] restore secure info successed, check password in sqlite
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [secure/crypto.go:123] get password from environment variable
2025-03-14T15:47:10.380 INFO [1156691] [F000000000000000] [server/run.go:91] check password in sqlite successed
2025-03-14T15:47:10.381 INFO [1156691] [F000000000000000] [server/run.go:199] take over flag is 1
2025-03-14T15:47:10.381 INFO [1156691] [F000000000000000] [oceanbase/loader.go:66] initialzie oceanbase instance without db name …
2025-03-14T15:47:10.381 INFO [1156691] [F000000000000000] [oceanbase/builder.go:74] try connect oceanbase: 1
2025-03-14T15:47:10.381 INFO [1156691] [LS00000000000000] [runtime/asm_arm64.s:1172] scheduler starting
2025-03-14T15:47:10.381 INFO [1156691] [F000000000000000] [secure/crypto.go:137] current password is temporary, will dump it into sqlite
2025-03-14T15:47:10.403 INFO [1156691] [F000000000000000] [oceanbase/loader.go:173] current config is nil, update db instance
2025-03-14T15:47:10.403 INFO [1156691] [F000000000000000] [oceanbase/loader.go:76] init oceanbase instance without db name success
2025-03-14T15:47:10.403 INFO [1156691] [F000000000000000] [oceanbase/loader.go:92] initialzie oceanbase instance …
2025-03-14T15:47:10.403 INFO [1156691] [F000000000000000] [oceanbase/builder.go:74] try connect oceanbase: 1
2025-03-14T15:47:10.411 INFO [1156691] [F000000000000000] [oceanbase/loader.go:185] db name changed, update db instance
2025-03-14T15:47:10.411 INFO [1156691] [F000000000000000] [oceanbase/loader.go:102] init oceanbase instance success
2025-03-14T15:47:11.386 INFO [1156691] [F000000000000000] [secure/crypto.go:143] dump temporary password into sqlite successed
2025-03-14T15:47:11.386 INFO [1156691] [F000000000000000] [server/takeover.go:42] start to take over or rebuild
2025-03-14T15:47:11.426 INFO [1156691] [F000000000000000] [oceanbase/builder.go:121] create database ocs succeed
2025-03-14T15:47:12.264 INFO [1156691] [F000000000000000] [oceanbase/builder.go:191] auto migrate ob tables succeed
2025-03-14T15:47:12.284 INFO [1156691] [F000000000000000] [server/takeover.go:84] agent with ip 192.168.2.109 and rpc port 10001 not found, need to take over
2025-03-14T15:47:12.284 INFO [1156691] [F000000000000000] [ob/upgrade_agent.go:207] check target version
2025-03-14T15:47:12.301 INFO [1156691] [F000000000000000] [ob/take_over.go:50] lock cluster status succeed
2025-03-14T15:47:12.331 INFO [1156691] [F000000000000000] [ob/take_over.go:47] unlock cluster status succeed
2025-03-14T15:47:12.331 INFO [1156691] [F000000000000000] [web/server.go:146] listen tcp socket on 0.0.0.0:2886
2025-03-14T15:47:12.331 INFO [1156691] [F000000000000000] [web/server.go:179] run tcp server
2025-03-14T15:47:12.331 INFO [1156691] [F000000000000000] [web/server.go:229] set web server state to 2
2025-03-14T15:47:12.331 INFO [1156691] [F000000000000000] [web/server.go:187] listen tcp socket on 0.0.0.0:2886

#16

麻烦弄个附件发出来看一下

#17

obd bootstrap部分日志:
[2025-03-14 15:46:01.167] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - import bootstrap
[2025-03-14 15:46:01.167] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - add bootstrap ref count to 1
[2025-03-14 15:46:01.167] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – bootstrap for components: dict_keys([‘oceanbase-ce’, ‘obproxy-ce’, ‘obagent’, ‘prometheus’, ‘grafana’])
[2025-03-14 15:46:01.167] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – execute sql: set session ob_query_timeout=1000000000
[2025-03-14 15:46:01.168] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – execute sql: set session ob_query_timeout=1000000000. args: None
[2025-03-14 15:46:01.168] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [INFO] Cluster bootstrap
[2025-03-14 15:46:01.169] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – execute sql: alter system bootstrap REGION “sys_region” ZONE “zone1” SERVER “192.168.2.62:10001”,REGION “sys_region” ZONE “zone2” SERVER “192.168.2.109:10001”,REGION “sys_region” ZONE “zone3” SERVER “192.168.2.198:10001”. args: None
[2025-03-14 15:47:05.589] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – execute sql: alter user “root” IDENTIFIED BY %s. args: [‘’]
[2025-03-14 15:47:05.730] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – execute sql: select * from oceanbase.__all_server. args: None
[2025-03-14 15:47:05.732] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - sub bootstrap ref count to 0
[2025-03-14 15:47:05.732] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - export bootstrap
[2025-03-14 15:47:05.732] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - plugin oceanbase-ce-py_script_bootstrap-3.1.0 result: True
[2025-03-14 15:47:05.732] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Searching user_pre plugin for components …
[2025-03-14 15:47:05.732] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Searching user_pre plugin for oceanbase-ce-4.3.5.0-100000202024123117.el7-7366be71e093dcb2498f7921212a5b44caa4499a
[2025-03-14 15:47:05.732] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Found for oceanbase-ce-py_script_user_pre-4.2.1.0 for oceanbase-ce-4.3.5.0
[2025-03-14 15:47:05.732] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Call plugin oceanbase-ce-py_script_user_pre-4.2.1.0 for oceanbase-ce-4.3.5.0-100000202024123117.el7-7366be71e093dcb2498f7921212a5b44caa4499a
[2025-03-14 15:47:05.733] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - import user_pre
[2025-03-14 15:47:05.733] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - add user_pre ref count to 1
[2025-03-14 15:47:05.733] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – bootstrap for components: dict_keys([‘oceanbase-ce’, ‘obproxy-ce’, ‘obagent’, ‘prometheus’, ‘grafana’])
[2025-03-14 15:47:05.733] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – scale out for components: []
[2025-03-14 15:47:05.734] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - sub user_pre ref count to 0
[2025-03-14 15:47:05.734] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - export user_pre
[2025-03-14 15:47:05.734] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - plugin oceanbase-ce-py_script_user_pre-4.2.1.0 result: True
[2025-03-14 15:47:05.734] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Searching create_user plugin for components …
[2025-03-14 15:47:05.734] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Searching create_user plugin for oceanbase-ce-4.3.5.0-100000202024123117.el7-7366be71e093dcb2498f7921212a5b44caa4499a
[2025-03-14 15:47:05.735] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Found for oceanbase-ce-py_script_create_user-4.0.0.0 for oceanbase-ce-4.3.5.0
[2025-03-14 15:47:05.735] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Call plugin oceanbase-ce-py_script_create_user-4.0.0.0 for oceanbase-ce-4.3.5.0-100000202024123117.el7-7366be71e093dcb2498f7921212a5b44caa4499a
[2025-03-14 15:47:05.735] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - import create_user
[2025-03-14 15:47:05.736] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - add create_user ref count to 1
[2025-03-14 15:47:05.736] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – execute sql: create user if not exists ‘proxyro’ IDENTIFIED BY %s;. args: ['’]
[2025-03-14 15:47:05.780] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – execute sql: grant all on . to ‘proxyro’ WITH GRANT OPTION;. args: []
[2025-03-14 15:47:05.828] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – execute sql: create user if not exists ‘ocp_monitor’ IDENTIFIED BY %s;. args: ['’]
[2025-03-14 15:47:05.868] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – execute sql: grant all on . to ‘ocp_monitor’ WITH GRANT OPTION;. args: []
[2025-03-14 15:47:05.909] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - sub create_user ref count to 0
[2025-03-14 15:47:05.909] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - export create_user
[2025-03-14 15:47:05.909] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - plugin oceanbase-ce-py_script_create_user-4.0.0.0 result: True
[2025-03-14 15:47:05.909] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Searching obshell_start plugin for components …
[2025-03-14 15:47:05.909] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Searching obshell_start plugin for oceanbase-ce-4.3.5.0-100000202024123117.el7-7366be71e093dcb2498f7921212a5b44caa4499a
[2025-03-14 15:47:05.909] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Found for oceanbase-ce-py_script_obshell_start-4.2.1.4 for oceanbase-ce-4.3.5.0
[2025-03-14 15:47:05.909] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Call plugin oceanbase-ce-py_script_obshell_start-4.2.1.4 for oceanbase-ce-4.3.5.0-100000202024123117.el7-7366be71e093dcb2498f7921212a5b44caa4499a
[2025-03-14 15:47:05.909] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - import obshell_start
[2025-03-14 15:47:05.910] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - add obshell_start ref count to 1
[2025-03-14 15:47:05.910] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – start_obshell: True
[2025-03-14 15:47:05.910] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [INFO] obshell start
[2025-03-14 15:47:05.910] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.62 execute: cat /home/admin/oceanbase-ce/run/obshell.pid
[2025-03-14 15:47:05.925] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 1, error output:
[2025-03-14 15:47:05.925] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] cat: /home/admin/oceanbase-ce/run/obshell.pid: No such file or directory
[2025-03-14 15:47:05.925] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG]
[2025-03-14 15:47:05.925] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.62 export OB_ROOT_PASSWORD=
[2025-03-14 15:47:05.925] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – start obshell: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/obshell admin start --ip 192.168.2.62 --port 2886
[2025-03-14 15:47:05.926] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.62 execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/obshell admin start --ip 192.168.2.62 --port 2886
[2025-03-14 15:47:10.075] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 0
[2025-03-14 15:47:10.075] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.109 execute: cat /home/admin/oceanbase-ce/run/obshell.pid
[2025-03-14 15:47:10.089] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 1, error output:
[2025-03-14 15:47:10.089] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] cat: /home/admin/oceanbase-ce/run/obshell.pid: No such file or directory
[2025-03-14 15:47:10.089] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG]
[2025-03-14 15:47:10.089] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.109 export OB_ROOT_PASSWORD=******
[2025-03-14 15:47:10.089] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – start obshell: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/obshell admin start --ip 192.168.2.109 --port 2886
[2025-03-14 15:47:10.089] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.109 execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/obshell admin start --ip 192.168.2.109 --port 2886
[2025-03-14 15:47:13.241] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 0
[2025-03-14 15:47:13.241] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.198 execute: cat /home/admin/oceanbase-ce/run/obshell.pid
[2025-03-14 15:47:13.255] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 1, error output:
[2025-03-14 15:47:13.255] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] cat: /home/admin/oceanbase-ce/run/obshell.pid: No such file or directory
[2025-03-14 15:47:13.255] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG]
[2025-03-14 15:47:13.255] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.198 export OB_ROOT_PASSWORD=******
[2025-03-14 15:47:13.255] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – start obshell: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/obshell admin start --ip 192.168.2.198 --port 2886
[2025-03-14 15:47:13.255] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.198 execute: cd /home/admin/oceanbase-ce; /home/admin/oceanbase-ce/bin/obshell admin start --ip 192.168.2.198 --port 2886
[2025-03-14 15:47:16.428] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 0
[2025-03-14 15:47:16.454] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [INFO] obshell program health check
[2025-03-14 15:47:16.455] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.62 execute: cat /home/admin/oceanbase-ce/run/obshell.pid
[2025-03-14 15:47:16.469] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 0
[2025-03-14 15:47:16.470] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – Get server1(192.168.2.62) obshell[pid: 935784]
[2025-03-14 15:47:16.470] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.62 execute: ls /proc/935784
[2025-03-14 15:47:16.531] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 0
[2025-03-14 15:47:16.532] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – server1(192.168.2.62) obshell[pid: 935784] started
[2025-03-14 15:47:16.532] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.109 execute: cat /home/admin/oceanbase-ce/run/obshell.pid
[2025-03-14 15:47:16.545] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 0
[2025-03-14 15:47:16.546] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – Get server2(192.168.2.109) obshell[pid: 1156691]
[2025-03-14 15:47:16.546] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.109 execute: ls /proc/1156691
[2025-03-14 15:47:16.603] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 0
[2025-03-14 15:47:16.603] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – server2(192.168.2.109) obshell[pid: 1156691] started
[2025-03-14 15:47:16.603] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.198 execute: cat /home/admin/oceanbase-ce/run/obshell.pid
[2025-03-14 15:47:16.617] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 0
[2025-03-14 15:47:16.617] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – Get server3(192.168.2.198) obshell[pid: 890677]
[2025-03-14 15:47:16.617] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – admin@192.168.2.198 execute: ls /proc/890677
[2025-03-14 15:47:16.675] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – exited code 0
[2025-03-14 15:47:16.675] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] – server3(192.168.2.198) obshell[pid: 890677] started
[2025-03-14 15:47:16.715] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - sub obshell_start ref count to 0
[2025-03-14 15:47:16.715] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - export obshell_start
[2025-03-14 15:47:16.715] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - plugin oceanbase-ce-py_script_obshell_start-4.2.1.4 result: True
[2025-03-14 15:47:16.715] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Searching obshell_bootstrap plugin for components …
[2025-03-14 15:47:16.715] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Searching obshell_bootstrap plugin for oceanbase-ce-4.3.5.0-100000202024123117.el7-7366be71e093dcb2498f7921212a5b44caa4499a
[2025-03-14 15:47:16.716] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Found for oceanbase-ce-py_script_obshell_bootstrap-4.2.1.4 for oceanbase-ce-4.3.5.0
[2025-03-14 15:47:16.716] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - Call plugin oceanbase-ce-py_script_obshell_bootstrap-4.2.1.4 for oceanbase-ce-4.3.5.0-100000202024123117.el7-7366be71e093dcb2498f7921212a5b44caa4499a
[2025-03-14 15:47:16.716] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - import obshell_bootstrap
[2025-03-14 15:47:17.186] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [DEBUG] - add obshell_bootstrap ref count to 1
[2025-03-14 15:47:17.187] [4ab13c28-00a8-11f0-82a1-fa163ead067e] [INFO] obshell bootstrap

#18

你上面发的日志片段并没有报错。麻烦弄个附件发出来吧

#19

obshell log
obshell.log (18.7 KB)

#21

obd日志[obd.txt|attachment](upload://snAHR6GzfGM6OUFTcFmscRXV4w0.txt) (57.0 KB)

#22

我发现obshell使用了2886端口，但是这个端口是关闭的，如何修改这个默认的2886端口号

image
image1193×58 5.04 KB

#23

多谢各位，问题已解决，由于obshell的端口被强了，导致没有办法完成端口的打开。分布式集群顺利部署完毕。

image
image1262×1188 52.2 KB

#24

后续是自己打通obshell端口解决了么