OceanBase无法启动

Gone222 · 2024 年9 月 24 日 11:33

但是我是 33 机器启动不起来，看 ob 给我的报错都是 warn 在另外两个节点的

Gone222 · 2024 年9 月 24 日 11:44

Gone222:

root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:03.993] [DEBUG] -- exited code 0
[2024-09-24 11:29:03.994] [DEBUG] -- root@49.52.27.33 execute: ls /proc/12246
[2024-09-24 11:29:04.049] [DEBUG] -- exited code 255, error output:

root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:03.993] [DEBUG] – exited code 0
[2024-09-24 11:29:03.994] [DEBUG] – root@49.52.27.33 execute: ls /proc/12246
[2024-09-24 11:29:04.049] [DEBUG] – exited code 255, error output: @论坛小助手 @素素好像就是这一步有问题？异常代码 255 是啥，这里没有明确的报错信息，所以有可能是进程数不够吗？

Gone222 · 2024 年9 月 24 日 11:45

@旭辉请问您 log 层面有什么比较明确的建议嘛

论坛小助手 · 2024 年9 月 24 日 11:45

根据您提供的日志信息，OceanBase 集群在启动过程中遇到了一些问题，主要集中在几个关键路径不存在的情况。具体来说，日志中多次提到 /data/1/clog/tenant_1/ 和 /data/1/sstable/block_file 这两个路径不存在。这可能是导致集群无法正常启动的原因之一。

针对这个问题，建议您检查以下几个方面：

数据路径配置：确保您的配置文件中指定的数据路径 /data/1 已经正确创建，并且具有相应的子目录结构。这些路径应该在所有服务器上都存在并且一致。 [8]
权限问题：确认这些路径的权限是否正确。OceanBase 需要足够的权限来创建和写入这些目录。通常情况下，这些目录应该属于运行 OceanBase 的用户（在您的配置中是 root 用户）。
集群状态：如果这是首次启动集群，那么这些路径不存在可能是正常的，因为它们会在集群初始化过程中被创建。但如果这是集群重启的一部分，那么这些路径应该已经存在。请检查集群的状态，确认之前是否有成功的启动记录。
日志文件：检查 /root/observer/run/observer.pid 文件是否存在。这个文件用来记录 OceanBase 进程的 PID，如果不存在，可能意味着之前的启动尝试并未成功。 [8]
网络配置：虽然日志中显示网络设备 enp95s0f0 存在，但还是建议检查网络配置是否正确，确保服务器之间可以正常通信。

综上所述，您可以按照以下步骤进行排查：

确认 /data/1 目录以及其子目录 clog 和 sstable 是否已经创建。
检查这些目录的权限是否正确，确保 root 用户拥有读写权限。
如果是首次启动，请确保集群的初始化过程完成。
查看集群的日志文件，寻找更详细的错误信息。

如果问题仍然存在，建议您联系 OceanBase 的技术支持团队，他们可以提供更深入的帮助。 [8]

详情请参考:

素素 · 2024 年9 月 24 日 12:02

这倒不是的，它只是ls输出异常了，就255了。不代表进程数不足。

素素 · 2024 年9 月 24 日 12:02

[2024-09-24 11:29:00.454] [INFO] Start observer
[2024-09-24 11:29:00.454] [DEBUG] -- root@49.52.27.33 execute: ls /data/1/clog/tenant_1/
[2024-09-24 11:29:00.469] [DEBUG] -- exited code 255, error output:
[2024-09-24 11:29:00.470] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-24 11:29:00.470] [DEBUG]
[2024-09-24 11:29:00.470] [DEBUG] -- root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:00.521] [DEBUG] -- exited code 1, error output:
[2024-09-24 11:29:00.521] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-24 11:29:00.521] [DEBUG]
[2024-09-24 11:29:00.521] [DEBUG] -- server1(49.52.27.33) start command construction
[2024-09-24 11:29:00.521] [DEBUG] -- root@49.52.27.34 execute: ls /data/1/clog/tenant_1/
[2024-09-24 11:29:00.533] [DEBUG] -- exited code 2, error output:
[2024-09-24 11:29:00.533] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-24 11:29:00.534] [DEBUG]
[2024-09-24 11:29:00.534] [DEBUG] -- root@49.52.27.34 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:00.585] [DEBUG] -- exited code 1, error output:
[2024-09-24 11:29:00.586] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-24 11:29:00.586] [DEBUG]
[2024-09-24 11:29:00.586] [DEBUG] -- server2(49.52.27.34) start command construction
[2024-09-24 11:29:00.586] [DEBUG] -- root@49.52.27.35 execute: ls /data/1/clog/tenant_1/
[2024-09-24 11:29:00.591] [DEBUG] -- exited code 2, error output:
[2024-09-24 11:29:00.592] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-24 11:29:00.592] [DEBUG]
[2024-09-24 11:29:00.592] [DEBUG] -- root@49.52.27.35 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:00.636] [DEBUG] -- exited code 1, error output:
[2024-09-24 11:29:00.636] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory
[2024-09-24 11:29:00.636] [DEBUG]
[2024-09-24 11:29:00.636] [DEBUG] -- server3(49.52.27.35) start command construction

这一段以上，是启动前检查检查，看看是否是observer已经启动，所以里面的exited code非0，不需要关注。

[2024-09-24 11:29:00.637] [DEBUG] -- starting server1(49.52.27.33) observer
[2024-09-24 11:29:00.637] [DEBUG] -- root@49.52.27.33 set env LD_LIBRARY_PATH to '/root/observer/lib:'
[2024-09-24 11:29:00.637] [DEBUG] -- root@49.52.27.33 execute: cd /root/observer; /root/observer/bin/observer -r '49.52.27.33:3882:3881;49.52.27.34:3882:3881;49.52.27.35:3882:3881' -p 3881 -P 3882 -z 'zone1' -c 1727148513 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='10G',system_memory='4G',datafile_size='10G',datafile_next='10G',datafile_maxsize='20G',log_disk_size='13G',enable_syslog_wf=False,max_syslog_file_count=4,obshell_port=3886
[2024-09-24 11:29:00.720] [DEBUG] -- exited code 0
[2024-09-24 11:29:00.721] [DEBUG] -- root@49.52.27.33 delete env LD_LIBRARY_PATH
[2024-09-24 11:29:00.722] [DEBUG] -- starting server2(49.52.27.34) observer
[2024-09-24 11:29:00.722] [DEBUG] -- root@49.52.27.34 set env LD_LIBRARY_PATH to '/root/observer/lib:'
[2024-09-24 11:29:00.723] [DEBUG] -- root@49.52.27.34 execute: cd /root/observer; /root/observer/bin/observer -r '49.52.27.33:3882:3881;49.52.27.34:3882:3881;49.52.27.35:3882:3881' -p 3881 -P 3882 -z 'zone2' -c 1727148513 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='10G',system_memory='4G',datafile_size='10G',datafile_next='10G',datafile_maxsize='20G',log_disk_size='13G',enable_syslog_wf=False,max_syslog_file_count=4,obshell_port=3886
[2024-09-24 11:29:00.806] [DEBUG] -- exited code 0
[2024-09-24 11:29:00.807] [DEBUG] -- root@49.52.27.34 delete env LD_LIBRARY_PATH
[2024-09-24 11:29:00.807] [DEBUG] -- starting server3(49.52.27.35) observer
[2024-09-24 11:29:00.807] [DEBUG] -- root@49.52.27.35 set env LD_LIBRARY_PATH to '/root/observer/lib:'
[2024-09-24 11:29:00.808] [DEBUG] -- root@49.52.27.35 execute: cd /root/observer; /root/observer/bin/observer -r '49.52.27.33:3882:3881;49.52.27.34:3882:3881;49.52.27.35:3882:3881' -p 3881 -P 3882 -z 'zone3' -c 1727148513 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='10G',system_memory='4G',datafile_size='10G',datafile_next='10G',datafile_maxsize='20G',log_disk_size='13G',enable_syslog_wf=False,max_syslog_file_count=4,obshell_port=3886
[2024-09-24 11:29:00.881] [DEBUG] -- exited code 0
[2024-09-24 11:29:00.882] [DEBUG] -- root@49.52.27.35 delete env LD_LIBRARY_PATH

这一段以上，是在真正启动了，执行了具体的启动命令，并且执行返回exited code为0. （此时代表的是执行命令成功，但是执行的结果在下一段进行检查）

[2024-09-24 11:29:00.977] [INFO] observer program health check
[2024-09-24 11:29:03.980] [DEBUG] -- server1(49.52.27.33) program health check
[2024-09-24 11:29:03.980] [DEBUG] -- root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:03.993] [DEBUG] -- exited code 0
[2024-09-24 11:29:03.994] [DEBUG] -- root@49.52.27.33 execute: ls /proc/12246
[2024-09-24 11:29:04.049] [DEBUG] -- exited code 255, error output:
[2024-09-24 11:29:04.049] [DEBUG]
[2024-09-24 11:29:04.050] [DEBUG] -- server2(49.52.27.34) program health check
[2024-09-24 11:29:04.050] [DEBUG] -- root@49.52.27.34 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:04.062] [DEBUG] -- exited code 0
[2024-09-24 11:29:04.063] [DEBUG] -- root@49.52.27.34 execute: ls /proc/2514
[2024-09-24 11:29:04.114] [DEBUG] -- exited code 0
[2024-09-24 11:29:04.115] [DEBUG] -- server2(49.52.27.34) observer[pid: 2514] started
[2024-09-24 11:29:04.115] [DEBUG] -- server3(49.52.27.35) program health check
[2024-09-24 11:29:04.115] [DEBUG] -- root@49.52.27.35 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:04.121] [DEBUG] -- exited code 0
[2024-09-24 11:29:04.122] [DEBUG] -- root@49.52.27.35 execute: ls /proc/105710
[2024-09-24 11:29:04.167] [DEBUG] -- exited code 0
[2024-09-24 11:29:04.167] [DEBUG] -- server3(49.52.27.35) observer[pid: 105710] started

这一段以上，在进行启动后检查。
如果是正常情况下，33 34 35都应该有Observer进程对应的pid，它就cat一下pid，然后到proc里ls一下，ok就返回exited code 0.
但是33没有启动成功，但是33的observer里cat /root/observer/run/observer.pid应该不为空，所以紧接着去proc里ls 12246, 发现12246进程（observer）不存在。

[2024-09-24 11:29:04.235] [WARNING] OBD-2002: Failed to start server1(49.52.27.33) observer

就类似于ps -ef | grep observer发现没有，它就OBD-2002报失败了
那要不你就33上直接手动启动看看会发生什么.

哇，你是用root启动的，那就root吧

su - root
export LD_LIBRARY_PATH=/root/observer/lib:$LD_LIBRARY_PATH
cd /root/observer
/root/observer/bin/observer -r '49.52.27.33:3882:3881;49.52.27.34:3882:3881;49.52.27.35:3882:3881' -p 3881 -P 3882 -z 'zone1' -c 1727148513 -d '/data/1' -i 'enp95s0f0' -o __min_full_resource_pool_memory=2147483648,memory_limit='10G',system_memory='4G',datafile_size='10G',datafile_next='10G',datafile_maxsize='20G',log_disk_size='13G',enable_syslog_wf=False,max_syslog_file_count=4,obshell_port=3886
看看会不会发生什么

如果没有的话，要么看看系统/var/log/messages里看看有没有说明为什么这个进程没有启动成功。
另外，可以掐着启动的时间点去看看observer.log发生了什么（整个的observer.log太大难以阅读，看对应时间点即可）

素素 · 2024 年9 月 24 日 12:18

OB官方也有个报错代码释义。

错误码

可以参考下其中的OBD-2002看一下。

辞霜 · 2024 年9 月 24 日 12:33

你的33节点是不是已经存在了一个ob数据库

素素 · 2024 年9 月 24 日 12:50

素素:

[2024-09-24 11:29:00.454] [INFO] Start observer
[2024-09-24 11:29:00.454] [DEBUG] -- root@49.52.27.33 execute: ls /data/1/clog/tenant_1/
[2024-09-24 11:29:00.469] [DEBUG] -- exited code 255, error output:
[2024-09-24 11:29:00.470] [DEBUG] ls: cannot access /data/1/clog/tenant_1/: No such file or directory
[2024-09-24 11:29:00.470] [DEBUG]
[2024-09-24 11:29:00.470] [DEBUG] -- root@49.52.27.33 execute: cat /root/observer/run/observer.pid
[2024-09-24 11:29:00.521] [DEBUG] -- exited code 1, error output:
[2024-09-24 11:29:00.521] [DEBUG] cat: /root/observer/run/observer.pid: No such file or directory

应该不会，在检查阶段，没有显示33存在pid

Gone222 · 2024 年9 月 24 日 13:21

结果是这样的，看起来是可以正常启动？但是在客户端输入 obd cluster start obtest就会显示端口占用了，但这样的话不知道初始化的密码。以及每一次都得手动这么启动嘛

Gone222 · 2024 年9 月 24 日 13:27

我就是卡着重装后立即启动的时间点，暂时日志量还好，但没看出明显的 error
observer.log (4.3 MB)

Gone222 · 2024 年9 月 24 日 13:33

@素素而且很奇怪的是，虽然显示 33 启动不起来，但是直接在 33 上 grep observer 也能看到启动项

旭辉 · 2024 年9 月 24 日 14:53

日志中有大量rpc fail，检查下是否有防火墙等网络策略限制了？

Gone222 · 2024 年9 月 24 日 16:04

衷心感谢上面几位老师的帮助，已解决！祝好～

旭辉 · 2024 年9 月 24 日 18:09

如果可以分享下解决方案就更好了

Gone222 · 2024 年9 月 24 日 19:33

@旭辉 @素素我觉得上面素素老师介绍的比较详细，尤其是对于 obd display 的信息的详解，他提醒我是 33 没有启动成功是因为进程 id，ls 检索不到，包括查阅了一下 observer 的日志，发现一直有一个get ls handle failed的问题，所以我在本地 check 一下，虽然 ls /proc/进程 id 有内容，但是执行 ls后输入 echo $?发现返回值是 255，而不是正常执行上一个指令后的返回值 0，所以定位了一下应该是本机由于权限没有很好管理的问题，被人篡改了一些 ls 的内容，例如重定义 ls 命令等，把 ls 改回默认的形式后就好了（所以有时候总会遇到一些莫名其妙的问题，这个完全是没想到被人改过，但是上面对于 log 内容的解释还是让我受益匪浅的）

素素 · 2024 年9 月 24 日 21:00

下午在忙，晚上没伞，还淋雨了。没注意看消息。

你很棒。加油。

素素 · 2024 年9 月 24 日 21:02

突然发现论坛里曾经有个类似的案例，但是交互信息里没有给出解决方案。

get ls handle failed

不过你在此处解决了，后来人如果遇到了就知道怎么解决了。

旭辉 · 2024 年9 月 25 日 09:28

感谢你的分享，其它同学遇到类似问题能看到这个解决方案 @素素老师上面对obd日志的分析非常正确，部署问题的一个痛点就是环境的不确定性，产品或者工具侧也在尽量优化使部署更加顺畅，欢迎有问题或建议及时提问及反馈。

AntTech_NXJMVL · 2024 年12 月 2 日 14:06

您好，我也遇到了类似的问题，但是细节似乎又不太一样，修改ls权限并没有效果
log如下

[1998-01-06 04:11:34.542] [DEBUG] -- root@9.88.71.96 execute: ls /home/obmini/data/clog/tenant_1/
[1998-01-06 04:11:34.593] [DEBUG] -- exited code 2, error output:
[1998-01-06 04:11:34.593] [DEBUG] ls: cannot access '/home/obmini/data/clog/tenant_1/': No such file or directory
[1998-01-06 04:11:34.593] [DEBUG]
[1998-01-06 04:11:34.593] [DEBUG] -- root@9.88.71.96 execute: cat /home/obmini/observer/run/observer.pid
[1998-01-06 04:11:34.683] [DEBUG] -- exited code 1, error output:
[1998-01-06 04:11:34.683] [DEBUG] cat: /home/obmini/observer/run/observer.pid: No such file or directory
[1998-01-06 04:11:34.683] [DEBUG]
[1998-01-06 04:11:34.683] [DEBUG] -- server1(9.88.71.96) port check
[1998-01-06 04:11:34.683] [DEBUG] -- root@9.88.71.96 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0F2E' | awk -F' ' '{print $3}' | uniq
[1998-01-06 04:11:34.826] [DEBUG] -- exited code 0
[1998-01-06 04:11:34.826] [DEBUG] -- root@9.88.71.96 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0F2F' | awk -F' ' '{print $3}' | uniq
[1998-01-06 04:11:34.970] [DEBUG] -- exited code 0
[1998-01-06 04:11:34.971] [DEBUG] -- root@9.88.71.96 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B4A' | awk -F' ' '{print $3}' | uniq
[1998-01-06 04:11:35.117] [DEBUG] -- exited code 0
[1998-01-06 04:11:35.117] [DEBUG] -- root@9.88.71.96 execute: ls /home/obmini/data/sstable/block_file
[1998-01-06 04:11:35.207] [DEBUG] -- exited code 2, error output:
[1998-01-06 04:11:35.208] [DEBUG] ls: cannot access '/home/obmini/data/sstable/block_file': No such file or directory
[1998-01-06 04:11:35.208] [DEBUG]
[1998-01-06 04:11:35.208] [DEBUG] -- root@9.88.71.96 execute: grep -e '^ *enp46s0:' /proc/net/dev
[1998-01-06 04:11:35.296] [DEBUG] -- exited code 0
[1998-01-06 04:11:35.296] [DEBUG] -- root@9.88.71.96 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[1998-01-06 04:11:35.385] [DEBUG] -- exited code 0
[1998-01-06 04:11:35.385] [DEBUG] -- root@9.88.71.96 execute: ulimit -a
[1998-01-06 04:11:35.478] [DEBUG] -- exited code 0
[1998-01-06 04:11:35.479] [WARNING] OBD-1007: (9.88.71.96) The recommended number of stack size is unlimited (Current value: 8192)
[1998-01-06 04:11:35.479] [DEBUG] -- root@9.88.71.96 execute: sysctl -a
[1998-01-06 04:11:35.612] [DEBUG] -- exited code 0
[1998-01-06 04:11:35.619] [DEBUG] -- root@9.88.71.96 execute: cat /proc/meminfo
[1998-01-06 04:11:35.703] [DEBUG] -- exited code 0
[1998-01-06 04:11:35.704] [DEBUG] -- root@9.88.71.96 execute: df --block-size=1024
[1998-01-06 04:11:35.792] [DEBUG] -- exited code 0
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /dev, total: 4194304 avail: 4194304
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /dev/shm, total: 168028930048 avail: 168028917760
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /run, total: 67211575296 avail: 67192000512
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 4194304 avail: 4194304
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /, total: 73391005696 avail: 20206366720
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /tmp, total: 168028934144 avail: 167941931008
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /boot, total: 1020702720 avail: 775974912
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /boot/efi, total: 627875840 avail: 621453312
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /home, total: 26692533919744 avail: 23685906448384
[1998-01-06 04:11:35.792] [DEBUG] -- get disk info for path /mnt/myimage, total: 17715562496 avail: 0
[1998-01-06 04:11:35.793] [DEBUG] -- root@9.88.71.96 execute: df --block-size=1024 /home/obmini/data
[1998-01-06 04:11:35.881] [DEBUG] -- exited code 0
[1998-01-06 04:11:35.881] [DEBUG] -- get disk info for path /home, total: 26692533919744 avail: 23685906448384
[1998-01-06 04:11:35.881] [DEBUG] -- root@9.88.71.96 execute: df --block-size=1024 /home/obmini/redo/clog
[1998-01-06 04:11:35.970] [DEBUG] -- exited code 0
[1998-01-06 04:11:35.970] [DEBUG] -- get disk info for path /home, total: 26692533919744 avail: 23685906448384
[1998-01-06 04:11:35.970] [DEBUG] -- disk: {'/dev': {'total': 4194304, 'avail': 4194304, 'need': 0}, '/dev/shm': {'total': 168028930048, 'avail': 168028917760, 'need': 0}, '/run': {'total': 67211575296, 'avail': 67192000512, 'need': 0}, '/sys/fs/cgroup': {'total': 4194304, 'avail': 4194304, 'need': 0}, '/': {'total': 73391005696, 'avail': 20206366720, 'need': 0}, '/tmp': {'total': 168028934144, 'avail': 167941931008, 'need': 0}, '/boot': {'total': 1020702720, 'avail': 775974912, 'need': 0}, '/boot/efi': {'total': 627875840, 'avail': 621453312, 'need': 0}, '/home': {'total': 26692533919744, 'avail': 23685906448384, 'need': 0}, '/mnt/myimage': {'total': 17715562496, 'avail': 0, 'need': 0}}
[1998-01-06 04:11:35.970] [WARNING] OBD-1012: (9.88.71.96) clog and data use the same disk (/home)
[1998-01-06 04:11:35.970] [DEBUG] -- root@9.88.71.96 execute: ping -W 1 -c 1 127.0.0.1
[1998-01-06 04:11:36.057] [DEBUG] -- exited code 0
[1998-01-06 04:11:36.058] [DEBUG] -- root@9.88.71.96 execute: date +%s%N
[1998-01-06 04:11:36.145] [DEBUG] -- exited code 0
[1998-01-06 04:11:36.145] [DEBUG] -- 9.88.71.96 time delta 43.193115234375
[1998-01-06 04:11:36.235] [INFO] [WARN] OBD-1007: (9.88.71.96) The recommended number of stack size is unlimited (Current value: 8192)
[1998-01-06 04:11:36.235] [INFO] [WARN] OBD-1012: (9.88.71.96) clog and data use the same disk (/home)
[1998-01-06 04:11:36.235] [INFO]
[1998-01-06 04:11:36.236] [DEBUG] - sub start_check ref count to 0
[1998-01-06 04:11:36.236] [DEBUG] - export start_check
[1998-01-06 04:11:36.236] [DEBUG] - plugin oceanbase-ce-py_script_start_check-4.2.1.4 result: True
[1998-01-06 04:11:36.236] [DEBUG] - Call obproxy-ce-py_script_start_check-4.3.0 for obproxy-ce-4.3.2.0-26.el7-fd779e401be448715254165b1a4f7205c4c1bda5
[1998-01-06 04:11:36.236] [DEBUG] - import start_check
[1998-01-06 04:11:36.237] [DEBUG] - add start_check ref count to 1
[1998-01-06 04:11:36.237] [INFO] Check before start obproxy
[1998-01-06 04:11:36.238] [DEBUG] -- root@9.88.71.96 execute: cat /data/obmini/obproxy/run/obproxy-9.88.71.96-3883.pid
[1998-01-06 04:11:36.286] [DEBUG] -- exited code 1, error output:
[1998-01-06 04:11:36.286] [DEBUG] cat: /data/obmini/obproxy/run/obproxy-9.88.71.96-3883.pid: No such file or directory
[1998-01-06 04:11:36.286] [DEBUG]
[1998-01-06 04:11:36.286] [DEBUG] -- 9.88.71.96 port check
[1998-01-06 04:11:36.286] [DEBUG] -- root@9.88.71.96 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0F2B' | awk -F' ' '{print $3}' | uniq
[1998-01-06 04:11:36.426] [DEBUG] -- exited code 0
[1998-01-06 04:11:36.427] [DEBUG] -- root@9.88.71.96 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0F2C' | awk -F' ' '{print $3}' | uniq
[1998-01-06 04:11:36.570] [DEBUG] -- exited code 0
[1998-01-06 04:11:36.570] [DEBUG] -- root@9.88.71.96 execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B45' | awk -F' ' '{print $3}' | uniq
[1998-01-06 04:11:36.713] [DEBUG] -- exited code 0
[1998-01-06 04:11:36.759] [DEBUG] - sub start_check ref count to 0
[1998-01-06 04:11:36.759] [DEBUG] - export start_check
[1998-01-06 04:11:36.759] [DEBUG] - plugin obproxy-ce-py_script_start_check-4.3.0 result: True
[1998-01-06 04:11:36.759] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.1.4 for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a
[1998-01-06 04:11:36.759] [DEBUG] - import start
[1998-01-06 04:11:36.761] [DEBUG] - add start ref count to 1
[1998-01-06 04:11:36.761] [INFO] Start observer
[1998-01-06 04:11:36.762] [DEBUG] -- root@9.88.71.96 execute: ls /home/obmini/data/clog/tenant_1/
[1998-01-06 04:11:36.813] [DEBUG] -- exited code 2, error output:
[1998-01-06 04:11:36.813] [DEBUG] ls: cannot access '/home/obmini/data/clog/tenant_1/': No such file or directory
[1998-01-06 04:11:36.813] [DEBUG]
[1998-01-06 04:11:36.813] [DEBUG] -- root@9.88.71.96 execute: cat /home/obmini/observer/run/observer.pid
[1998-01-06 04:11:36.904] [DEBUG] -- exited code 1, error output:
[1998-01-06 04:11:36.904] [DEBUG] cat: /home/obmini/observer/run/observer.pid: No such file or directory
[1998-01-06 04:11:36.904] [DEBUG]
[1998-01-06 04:11:36.904] [DEBUG] -- server1(9.88.71.96) start command construction
[1998-01-06 04:11:36.905] [DEBUG] -- starting server1(9.88.71.96) observer
[1998-01-06 04:11:36.905] [DEBUG] -- root@9.88.71.96 export LD_LIBRARY_PATH='/home/obmini/observer/lib:'
[1998-01-06 04:11:36.905] [DEBUG] -- root@9.88.71.96 execute: cd /home/obmini/observer; /home/obmini/observer/bin/observer -r '9.88.71.96:3887:3886' -p 3886 -P 3887 -z 'zone1' -n 'obmini' -c 1 -d '/home/obmini/data' -i 'enp46s0' -o __min_full_resource_pool_memory=2147483648,memory_limit='64G',system_memory='30G',datafile_size='192G',datafile_next='100G',datafile_maxsize='20G',log_disk_size='192G',cpu_count=16,enable_syslog_wf=False,max_syslog_file_count=4
[1998-01-06 04:11:37.015] [DEBUG] -- exited code 0
[1998-01-06 04:11:37.015] [DEBUG] -- root@9.88.71.96 delete env LD_LIBRARY_PATH
[1998-01-06 04:11:37.022] [DEBUG] -- start_obshell: False
[1998-01-06 04:11:37.022] [INFO] observer program health check
[1998-01-06 04:11:40.026] [DEBUG] -- server1(9.88.71.96) program health check
[1998-01-06 04:11:40.026] [DEBUG] -- root@9.88.71.96 execute: cat /home/obmini/observer/run/observer.pid
[1998-01-06 04:11:40.076] [DEBUG] -- exited code 1, error output:
[1998-01-06 04:11:40.076] [DEBUG] cat: /home/obmini/observer/run/observer.pid: No such file or directory
[1998-01-06 04:11:40.076] [DEBUG]
[1998-01-06 04:11:40.148] [WARNING] OBD-2002: Failed to start server1(9.88.71.96) observer
[1998-01-06 04:11:40.148] [DEBUG] - sub start ref count to 0
[1998-01-06 04:11:40.148] [DEBUG] - export start
[1998-01-06 04:11:40.148] [DEBUG] - plugin oceanbase-ce-py_script_start-4.2.1.4 result: False
[1998-01-06 04:11:40.148] [ERROR] oceanbase-ce start failed
[1998-01-06 04:11:40.150] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[1998-01-06 04:11:40.150] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[1998-01-06 04:11:40.150] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[1998-01-06 04:11:40.150] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obmini release, count 0
[1998-01-06 04:11:40.150] [DEBUG] - unlock /root/.obd/lock/deploy_obmini
[1998-01-06 04:11:40.150] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[1998-01-06 04:11:40.150] [DEBUG] - unlock /root/.obd/lock/global
[1998-01-06 04:11:40.150] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[1998-01-06 04:11:40.150] [INFO] Trace ID: 5c8a47b0-8609-11d1-9bbd-74a4b502e9c6
[1998-01-06 04:11:40.150] [INFO] If you want to view detailed obd logs, please run: obd display-trace 5c8a47b0-8609-11d1-9bbd-74a4b502e9c6

能帮忙分析一下原因吗