4.3.3 OBD-2002: Failed to start 127.0.0.1 observer

OceanBase
#1

【 使用环境 】测试环境
【 OB or 其他组件 】obd
【 使用版本 】
【问题描述】obd demo 安装试用，服务无法启动
【复现路径】按照官方文档安装试用环境时出错
【附件及日志】推荐使用OceanBase敏捷诊断工具obdiag收集诊断信息,详情参见链接（右键跳转查看）：

sub start_check ref count to 0
[2024-10-18 02:50:43.226] [DEBUG] - export start_check
[2024-10-18 02:50:43.227] [DEBUG] - plugin oceanbase-ce-py_script_start_check-4.3.0.0 result: True
[2024-10-18 02:50:43.227] [DEBUG] - Call obagent-py_script_start_check-4.2.2 for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e
[2024-10-18 02:50:43.227] [DEBUG] - import start_check
[2024-10-18 02:50:43.229] [DEBUG] - add start_check ref count to 1
[2024-10-18 02:50:43.230] [INFO] Check before start obagent
[2024-10-18 02:50:43.232] [DEBUG] -- local execute: cat /root/obagent/run/ob_agentd.pid 
[2024-10-18 02:50:43.236] [DEBUG] -- exited code 1, error output:
[2024-10-18 02:50:43.236] [DEBUG] cat: /root/obagent/run/ob_agentd.pid: No such file or directory
[2024-10-18 02:50:43.236] [DEBUG] 
[2024-10-18 02:50:43.237] [DEBUG] -- 127.0.0.1 port check
[2024-10-18 02:50:43.237] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1F99' | awk -F' ' '{print $3}' | uniq 
[2024-10-18 02:50:43.246] [DEBUG] -- exited code 0
[2024-10-18 02:50:43.246] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1F98' | awk -F' ' '{print $3}' | uniq 
[2024-10-18 02:50:43.254] [DEBUG] -- exited code 0
[2024-10-18 02:50:43.362] [DEBUG] - sub start_check ref count to 0
[2024-10-18 02:50:43.363] [DEBUG] - export start_check
[2024-10-18 02:50:43.363] [DEBUG] - plugin obagent-py_script_start_check-4.2.2 result: True
[2024-10-18 02:50:43.363] [DEBUG] - Call prometheus-py_script_start_check-2.37.1 for prometheus-2.37.1-10000102022110211.el7-58913c7606f05feb01bc1c6410346e5fc31cf263
[2024-10-18 02:50:43.363] [DEBUG] - import start_check
[2024-10-18 02:50:43.365] [DEBUG] - add start_check ref count to 1
[2024-10-18 02:50:43.366] [INFO] Check before start prometheus
[2024-10-18 02:50:43.366] [DEBUG] -- local execute: cat /root/prometheus/run/prometheus.pid 
[2024-10-18 02:50:43.371] [DEBUG] -- exited code 1, error output:
[2024-10-18 02:50:43.371] [DEBUG] cat: /root/prometheus/run/prometheus.pid: No such file or directory
[2024-10-18 02:50:43.371] [DEBUG] 
[2024-10-18 02:50:43.371] [DEBUG] -- 127.0.0.1 port check
[2024-10-18 02:50:43.371] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':2382' | awk -F' ' '{print $3}' | uniq 
[2024-10-18 02:50:43.381] [DEBUG] -- exited code 0
[2024-10-18 02:50:43.499] [DEBUG] - sub start_check ref count to 0
[2024-10-18 02:50:43.499] [DEBUG] - export start_check
[2024-10-18 02:50:43.499] [DEBUG] - plugin prometheus-py_script_start_check-2.37.1 result: True
[2024-10-18 02:50:43.499] [DEBUG] - Call obproxy-ce-py_script_start_check-4.3.0 for obproxy-ce-4.3.1.0-4.el7-835f4803c1f4da186439323b66c51db4662678a3
[2024-10-18 02:50:43.499] [DEBUG] - import start_check
[2024-10-18 02:50:43.501] [DEBUG] - add start_check ref count to 1
[2024-10-18 02:50:43.502] [INFO] Check before start obproxy
[2024-10-18 02:50:43.503] [DEBUG] -- local execute: cat /root/obproxy-ce/run/obproxy-127.0.0.1-2883.pid 
[2024-10-18 02:50:43.507] [DEBUG] -- exited code 1, error output:
[2024-10-18 02:50:43.507] [DEBUG] cat: /root/obproxy-ce/run/obproxy-127.0.0.1-2883.pid: No such file or directory
[2024-10-18 02:50:43.507] [DEBUG] 
[2024-10-18 02:50:43.507] [DEBUG] -- 127.0.0.1 port check
[2024-10-18 02:50:43.507] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B43' | awk -F' ' '{print $3}' | uniq 
[2024-10-18 02:50:43.516] [DEBUG] -- exited code 0
[2024-10-18 02:50:43.517] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B44' | awk -F' ' '{print $3}' | uniq 
[2024-10-18 02:50:43.525] [DEBUG] -- exited code 0
[2024-10-18 02:50:43.526] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B45' | awk -F' ' '{print $3}' | uniq 
[2024-10-18 02:50:43.534] [DEBUG] -- exited code 0
[2024-10-18 02:50:43.633] [DEBUG] - sub start_check ref count to 0
[2024-10-18 02:50:43.633] [DEBUG] - export start_check
[2024-10-18 02:50:43.634] [DEBUG] - plugin obproxy-ce-py_script_start_check-4.3.0 result: True
[2024-10-18 02:50:43.634] [DEBUG] - Call grafana-py_script_start_check-7.5.17 for grafana-7.5.17-1-1bf1f338d3a3445d8599dc6902e7aeed4de4e0d6
[2024-10-18 02:50:43.634] [DEBUG] - import start_check
[2024-10-18 02:50:43.636] [DEBUG] - add start_check ref count to 1
[2024-10-18 02:50:43.636] [INFO] Check before start grafana
[2024-10-18 02:50:43.637] [DEBUG] -- local execute: cat /root/grafana/run/grafana.pid 
[2024-10-18 02:50:43.641] [DEBUG] -- exited code 1, error output:
[2024-10-18 02:50:43.641] [DEBUG] cat: /root/grafana/run/grafana.pid: No such file or directory
[2024-10-18 02:50:43.641] [DEBUG] 
[2024-10-18 02:50:43.642] [DEBUG] -- 127.0.0.1 port check
[2024-10-18 02:50:43.642] [DEBUG] -- local execute: bash -c 'cat /proc/net/{udp*,tcp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0BB8' | awk -F' ' '{print $3}' | uniq 
[2024-10-18 02:50:43.651] [DEBUG] -- exited code 0
[2024-10-18 02:50:43.768] [DEBUG] - sub start_check ref count to 0
[2024-10-18 02:50:43.768] [DEBUG] - export start_check
[2024-10-18 02:50:43.768] [DEBUG] - plugin grafana-py_script_start_check-7.5.17 result: True
[2024-10-18 02:50:43.768] [DEBUG] - Call oceanbase-ce-py_script_start-4.3.0.0 for oceanbase-ce-4.3.3.0-100000142024101215.el7-11df93533c605aaae003f0ee27635f8ffa66aaa6
[2024-10-18 02:50:43.768] [DEBUG] - import start
[2024-10-18 02:50:43.771] [DEBUG] - add start ref count to 1
[2024-10-18 02:50:43.772] [INFO] cluster scenario: express_oltp
[2024-10-18 02:50:43.772] [INFO] Start observer
[2024-10-18 02:50:43.773] [DEBUG] -- local execute: ls /root/oceanbase-ce/store/clog/tenant_1/ 
[2024-10-18 02:50:43.777] [DEBUG] -- exited code 2, error output:
[2024-10-18 02:50:43.778] [DEBUG] ls: cannot access /root/oceanbase-ce/store/clog/tenant_1/: No such file or directory
[2024-10-18 02:50:43.778] [DEBUG] 
[2024-10-18 02:50:43.778] [DEBUG] -- local execute: cat /root/oceanbase-ce/run/observer.pid 
[2024-10-18 02:50:43.782] [DEBUG] -- exited code 1, error output:
[2024-10-18 02:50:43.782] [DEBUG] cat: /root/oceanbase-ce/run/observer.pid: No such file or directory
[2024-10-18 02:50:43.782] [DEBUG] 
[2024-10-18 02:50:43.782] [DEBUG] -- 127.0.0.1 start command construction
[2024-10-18 02:50:43.782] [DEBUG] -- update _enable_defensive_check to 0 because of scenario
[2024-10-18 02:50:43.782] [DEBUG] -- update enable_syslog_recycle to 1 because of scenario
[2024-10-18 02:50:43.782] [DEBUG] -- update max_syslog_file_count to 300 because of scenario
[2024-10-18 02:50:43.783] [DEBUG] -- starting 127.0.0.1 observer
[2024-10-18 02:50:43.783] [DEBUG] -- root@127.0.0.1 export LD_LIBRARY_PATH='/root/oceanbase-ce/lib:'
[2024-10-18 02:50:43.783] [DEBUG] -- local execute: cd /root/oceanbase-ce; /root/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -p 2881 -P 2882 -z 'zone1' -n 'demo' -c 1729234236 -d '/root/oceanbase-ce/store' -I '127.0.0.1' -o __min_full_resource_pool_memory=1073741824,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit='6G',system_memory='1G',cpu_count=16,datafile_size='2G',datafile_maxsize='8G',datafile_next='2G',log_disk_size='14G',_enable_defensive_check=0,enable_syslog_recycle=1 
[2024-10-18 02:50:43.808] [DEBUG] -- exited code 132, error output:
[2024-10-18 02:50:43.808] [DEBUG] /bin/sh: line 1: 65010 Illegal instruction     /root/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -p 2881 -P 2882 -z 'zone1' -n 'demo' -c 1729234236 -d '/root/oceanbase-ce/store' -I '127.0.0.1' -o __min_full_resource_pool_memory=1073741824,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit='6G',system_memory='1G',cpu_count=16,datafile_size='2G',datafile_maxsize='8G',datafile_next='2G',log_disk_size='14G',_enable_defensive_check=0,enable_syslog_recycle=1
[2024-10-18 02:50:43.808] [DEBUG] 
[2024-10-18 02:50:43.809] [DEBUG] -- root@127.0.0.1 delete env LD_LIBRARY_PATH
[2024-10-18 02:50:43.903] [ERROR] OBD-2002: Failed to start 127.0.0.1 observer: /bin/sh: line 1: 65010 Illegal instruction     /root/oceanbase-ce/bin/observer -r '127.0.0.1:2882:2881' -p 2881 -P 2882 -z 'zone1' -n 'demo' -c 1729234236 -d '/root/oceanbase-ce/store' -I '127.0.0.1' -o __min_full_resource_pool_memory=1073741824,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit='6G',system_memory='1G',cpu_count=16,datafile_size='2G',datafile_maxsize='8G',datafile_next='2G',log_disk_size='14G',_enable_defensive_check=0,enable_syslog_recycle=1
[2024-10-18 02:50:43.904] [ERROR] 
[2024-10-18 02:50:43.904] [DEBUG] - sub start ref count to 0
[2024-10-18 02:50:43.904] [DEBUG] - export start
[2024-10-18 02:50:43.904] [DEBUG] - plugin oceanbase-ce-py_script_start-4.3.0.0 result: None
[2024-10-18 02:50:43.904] [ERROR] oceanbase-ce start failed
[2024-10-18 02:50:43.908] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2024-10-18 02:50:43.908] [INFO] Trace ID: 438ef79a-8d1d-11ef-880b-0cda411d72d3
[2024-10-18 02:50:43.908] [INFO] If you want to view detailed obd logs, please run: obd display-trace 438ef79a-8d1d-11ef-880b-0cda411d72d3
[2024-10-18 02:50:43.910] [DEBUG] - unlock /root/.obd/lock/global
[2024-10-18 02:50:43.910] [DEBUG] - unlock /root/.obd/lock/deploy_demo
[2024-10-18 02:50:43.910] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo

【备注】基于 LLM 和开源文档 RAG 的论坛小助手已开放测试，在发帖时输入 [@论坛小助手] 即可召唤小助手，欢迎试用！

2 个赞
#3

根据文档库中的信息，OBD-2002 错误通常表示在启动 Observer 时遇到了问题。从您提供的日志来看，Observer 在启动时收到了 “Illegal instruction” 的错误，这通常意味着二进制文件与当前运行的 CPU 架构不兼容，或者存在其他硬件兼容性问题。 [2]

解决方案

  1. 检查 CPU 架构

    • 确认您的 CPU 架构是否与您下载的 OceanBase 版本兼容。您可以使用以下命令检查您的 CPU 架构：
      uname -m
    • 如果您的 CPU 架构是 x86_64，确保您下载的是适用于 x86_64 的 OceanBase 版本。如果是 ARM 架构，确保下载的是 ARM 版本。

  2. 检查二进制文件完整性

    • 确认下载的二进制文件是否完整且未损坏。您可以重新下载并重新安装 OceanBase，确保文件的完整性。

  3. 检查系统依赖

    • 确保您的系统已经安装了所有必要的依赖项。您可以参考官方文档中的系统要求部分，确保所有依赖项都已正确安装。

  4. 检查系统资源

    • 确认您的系统有足够的资源（如内存和磁盘空间）来运行 OceanBase。您可以使用以下命令检查系统资源：
      free -m
df -h

  5. 查看详细日志

    • 您可以使用 obd display-trace 命令查看更详细的日志，以便进一步诊断问题：
      obd display-trace 438ef79a-8d1d-11ef-880b-0cda411d72d3

如果以上步骤仍无法解决问题，建议您联系 OceanBase 的技术支持人员以获取进一步的帮助。 [2]

检索到的文档如下:

  1. 4000 ~ 4499
  2. 错误码
  3. 通过 OBD 白屏部署 OceanBase 集群
  4. OceanBase 安装部署工具 V1.6.0
  5. 0001 ~ 3999
  6. 使用 systemd 部署 OceanBase 数据库
2 个赞
#4

麻烦按照小助手的建议先排查下，确认您的 CPU 架构是否与您下载的 OceanBase 版本兼容

2 个赞
#5

CPU：Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz
操作系统：CentOS Linux release 7.9.2009 (Core)
安装包：oceanbase-all-in-one-4.3.3_20241014.el7.x86_64.tar.gz

2 个赞
#6

根据您提供的信息，您的 CPU 架构是 x86_64，操作系统是 CentOS 7.9，安装包是 oceanbase-all-in-one-4.3.3_20241014.el7.x86_64.tar.gz。这些信息表明您的环境应该是兼容的。然而，您仍然遇到了 “Illegal instruction” 错误，这通常意味着二进制文件与当前运行的 CPU 架构不完全匹配，或者存在其他硬件兼容性问题。

进一步排查步骤

  1. 检查 CPU 特性

    • 确认您的 CPU 是否支持所有必要的指令集。您可以使用以下命令查看 CPU 信息：
      lscpu
    • 特别关注 Flags 部分，确保您的 CPU 支持所有必要的指令集，如 SSE4_1、SSE4_2、AVX 等。

  2. 重新下载安装包

    • 确认下载的安装包没有损坏。您可以重新下载并重新安装 OceanBase，确保文件的完整性。您可以从 OceanBase 软件下载中心 下载最新的 all-in-one 安装包。

  3. 检查系统依赖

    • 确保您的系统已经安装了所有必要的依赖项。您可以参考官方文档中的系统要求部分，确保所有依赖项都已正确安装。[[2]]

  4. 检查系统资源

    • 确认您的系统有足够的资源（如内存和磁盘空间）来运行 OceanBase。您可以使用以下命令检查系统资源：
      free -m
df -h

  5. 查看详细日志

    • 您可以使用 obd display-trace 命令查看更详细的日志，以便进一步诊断问题：
      obd display-trace 438ef79a-8d1d-11ef-880b-0cda411d72d3

  6. 检查环境变量

    • 确认环境变量设置正确，特别是 LD_LIBRARY_PATH。您可以使用以下命令检查和设置环境变量：
      echo $LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/root/oceanbase-ce/lib:$LD_LIBRARY_PATH

示例步骤

  1. 查看 CPU 信息

    lscpu

  2. 重新下载安装包

    wget https://www.oceanbase.com/softwarecenter/path/to/oceanbase-all-in-one-4.3.3_20241014.el7.x86_64.tar.gz
tar -xzf oceanbase-all-in-one-4.3.3_20241014.el7.x86_64.tar.gz
cd oceanbase-all-in-one/bin/
./install.sh
source ~/.oceanbase-all-in-one/bin/env.sh

  3. 检查系统依赖

    sudo yum install -y epel-release
sudo yum install -y gcc gcc-c++ make libaio-devel numactl-devel

  4. 检查系统资源

    free -m
df -h

  5. 查看详细日志

    obd display-trace 438ef79a-8d1d-11ef-880b-0cda411d72d3

  6. 检查环境变量

    echo $LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/root/oceanbase-ce/lib:$LD_LIBRARY_PATH

如果以上步骤仍无法解决问题，建议您联系 OceanBase 的技术支持人员以获取进一步的帮助。[[2]]

检索到的文档如下:

  1. 4000 ~ 4499
  2. 错误码
  3. 通过 OBD 白屏部署 OceanBase 集群
  4. OceanBase 安装部署工具 V1.6.0
  5. 0001 ~ 3999
  6. 使用 systemd 部署 OceanBase 数据库

具体信息可参考以下文档:

  1. 软硬件要求
  2. 快速体验 OceanBase 社区版
  3. 服务器配置
  4. 使用 obd.sh 执行 mysqltest
  5. 安装并配置 obd
  6. 单机部署 OceanBase 数据库
  7. 使用命令行部署 OceanBase 数据库生产环境
  8. 通过白屏部署 OceanBase 集群
  9. 通过 OBD 白屏部署 OceanBase 集群

(小助手的答复已结束，如未能解决您的问题，请继续提问并等待其他同学的回复，谢谢！)

2 个赞
#7

Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 15
Model: 6
Model name: Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz
Stepping: 3
CPU MHz: 2593.906
BogoMIPS: 5187.81
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
L3 cache: 16384K
NUMA node0 CPU(s): 0-15
Flags: fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc nopl xtopology eagerfpu pni cx16 x2apic aes hypervisor lahf_lm

2 个赞
#9

你发下obd的版本吧，我们尝试复现下这个问题

1 个赞
#10

OceanBase Deploy: 2.10.1
REVISION: 9870cf1b400be9002d851b9a165b043f1da4e0e7
BUILD_BRANCH: HEAD
BUILD_TIME: Oct 12 2024 16:15:27OURCE
Copyright (C) 2021 OceanBase
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

1 个赞
#11

image
image821×223 35.1 KB

这个架构的cpu比较新

麻烦看下uname -a

image
image898×57 15.6 KB

1 个赞
#13

你好，我这边没能复现。没你这个cpu架构的机器。你提供一下observer 的log. 目前看是observer 报参数问题。需要看observer log 确认哪个参数问题。

1 个赞
#14

Linux localhost.localdomain 3.10.0-1160.el7.x86_64 #1 SMP Mon Oct 19 16:18:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

1 个赞
#15

这个observer的log怎么输出， obdiag的几个指令都没有输出有效的信息

#16

observer log在安装的home目录下的oceanbase/log
obdiag报错什么

#17

image
image1896×332 40.2 KB

obd obdiag gather all 获取不到任何内容

#18

oceanbase-ce 4.3.2.1版本的可以运行起来，4.3.3.0的不行

#19

你好目前 432正常运行着么

#20

432是正常的

#21

可能是433不支持某些cpu架构