【 使用环境 】测试环境
【 OB or 其他组件 】observer 初始化
【 使用版本 】4.0
【问题描述】Cluster init failed
【复现路径】问题出现前后相关操作
【问题现象及影响】
OceanBase DataBase 数据库 4.1.0.0 失败查看详情
[2023-06-06 15:41:35.283] [DEBUG] - Found for oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-06-06 15:41:35.283] [DEBUG] - Searching start plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2023-06-06 15:41:35.283] [DEBUG] - Found for obproxy-ce-py_script_start-3.1.0 for obproxy-ce-4.1.0.0
[2023-06-06 15:41:35.283] [DEBUG] - Searching start plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2023-06-06 15:41:35.284] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-1.3.1
[2023-06-06 15:41:35.284] [DEBUG] - Searching start plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2023-06-06 15:41:35.284] [DEBUG] - Found for ocp-express-py_script_start-1.0.1 for ocp-express-1.0.1
[2023-06-06 15:41:35.284] [DEBUG] - Searching connect plugin for components …
[2023-06-06 15:41:35.284] [DEBUG] - Searching connect plugin for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-06 15:41:35.285] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.1.0.0
[2023-06-06 15:41:35.285] [DEBUG] - Searching connect plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2023-06-06 15:41:35.285] [DEBUG] - Found for obproxy-ce-py_script_connect-3.1.0 for obproxy-ce-4.1.0.0
[2023-06-06 15:41:35.285] [DEBUG] - Searching connect plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2023-06-06 15:41:35.286] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-1.3.1
[2023-06-06 15:41:35.286] [DEBUG] - Searching connect plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2023-06-06 15:41:35.286] [DEBUG] - Found for ocp-express-py_script_connect-1.0.1 for ocp-express-1.0.1
[2023-06-06 15:41:35.286] [DEBUG] - Searching bootstrap plugin for components …
[2023-06-06 15:41:35.286] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-06 15:41:35.287] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-06-06 15:41:35.287] [DEBUG] - Searching bootstrap plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2023-06-06 15:41:35.287] [DEBUG] - Found for obproxy-ce-py_script_bootstrap-3.1.0 for obproxy-ce-4.1.0.0
[2023-06-06 15:41:35.287] [DEBUG] - Searching bootstrap plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2023-06-06 15:41:35.288] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-1.3.1
[2023-06-06 15:41:35.288] [DEBUG] - Searching bootstrap plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2023-06-06 15:41:35.288] [DEBUG] - Found for ocp-express-py_script_bootstrap-1.0.1 for ocp-express-1.0.1
[2023-06-06 15:41:35.288] [DEBUG] - Searching display plugin for components …
[2023-06-06 15:41:35.288] [DEBUG] - Searching display plugin for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-06 15:41:35.289] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.1.0.0
[2023-06-06 15:41:35.289] [DEBUG] - Searching display plugin for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2023-06-06 15:41:35.289] [DEBUG] - Found for obproxy-ce-py_script_display-3.1.0 for obproxy-ce-4.1.0.0
[2023-06-06 15:41:35.289] [DEBUG] - Searching display plugin for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2023-06-06 15:41:35.290] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-1.3.1
[2023-06-06 15:41:35.290] [DEBUG] - Searching display plugin for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2023-06-06 15:41:35.290] [DEBUG] - Found for ocp-express-py_script_display-1.0.1 for ocp-express-1.0.1
[2023-06-06 15:41:35.290] [INFO] Load cluster param plugin
[2023-06-06 15:41:35.290] [DEBUG] - Searching param plugin for components …
[2023-06-06 15:41:35.291] [DEBUG] - Search param plugin for oceanbase-ce
[2023-06-06 15:41:35.291] [DEBUG] - Found for oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-06-06 15:41:35.291] [DEBUG] - Applying oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-06 15:41:35.292] [DEBUG] - Search param plugin for obproxy-ce
[2023-06-06 15:41:35.292] [DEBUG] - Found for obproxy-ce-param-3.1.0 for obproxy-ce-4.1.0.0
[2023-06-06 15:41:35.292] [DEBUG] - Applying obproxy-ce-param-3.1.0 for obproxy-ce-4.1.0.0-7.el7-2a9d9bf67f179dcca2a8c9e7c77373d94e7e2abe
[2023-06-06 15:41:35.292] [DEBUG] - Search param plugin for obagent
[2023-06-06 15:41:35.292] [DEBUG] - Found for obagent-param-1.3.0 for obagent-1.3.1
[2023-06-06 15:41:35.292] [DEBUG] - Applying obagent-param-1.3.0 for obagent-1.3.1-5.el7-ccfe93272a79ab1073e76f00580386c9c52e8324
[2023-06-06 15:41:35.293] [DEBUG] - Search param plugin for ocp-express
[2023-06-06 15:41:35.293] [DEBUG] - Found for ocp-express-param-1.0.1 for ocp-express-1.0.1
[2023-06-06 15:41:35.293] [DEBUG] - Applying ocp-express-param-1.0.1 for ocp-express-1.0.1-100000072023051917.el7-59eb8062858271a23080c824c98a72f9e5896235
[2023-06-06 15:41:35.293] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.0.0.0 for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-06 15:41:35.294] [INFO] Check before start observer
[2023-06-06 15:41:35.295] [DEBUG] – root@192.168.8.30 execute: ls /root/oceabase/oceanbase/store/clog/tenant_1/
[2023-06-06 15:41:35.350] [DEBUG] – exited code 2, error output:
[2023-06-06 15:41:35.350] [DEBUG] ls: cannot access /root/oceabase/oceanbase/store/clog/tenant_1/: No such file or directory
[2023-06-06 15:41:35.351] [DEBUG]
[2023-06-06 15:41:35.351] [DEBUG] – root@192.168.8.30 execute: cat /root/oceabase/oceanbase/run/observer.pid
[2023-06-06 15:41:35.463] [DEBUG] – exited code 1, error output:
[2023-06-06 15:41:35.463] [DEBUG] cat: /root/oceabase/oceanbase/run/observer.pid: No such file or directory
[2023-06-06 15:41:35.463] [DEBUG]
[2023-06-06 15:41:35.464] [DEBUG] – 192.168.8.30 port check
[2023-06-06 15:41:35.464] [DEBUG] – root@192.168.8.30 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-06-06 15:41:35.563] [DEBUG] – exited code 0
[2023-06-06 15:41:35.563] [DEBUG] – root@192.168.8.30 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-06-06 15:41:35.660] [DEBUG] – exited code 0
[2023-06-06 15:41:35.661] [DEBUG] – root@192.168.8.30 execute: ls /root/oceabase/oceanbase/store/sstable/block_file
[2023-06-06 15:41:35.761] [DEBUG] – exited code 2, error output:
[2023-06-06 15:41:35.761] [DEBUG] ls: cannot access /root/oceabase/oceanbase/store/sstable/block_file: No such file or directory
[2023-06-06 15:41:35.762] [DEBUG]
[2023-06-06 15:41:35.763] [DEBUG] – root@192.168.8.31 execute: ls /root/oceabase/oceanbase/store/clog/tenant_1/
[2023-06-06 15:41:35.791] [DEBUG] – exited code 2, error output:
[2023-06-06 15:41:35.791] [DEBUG] ls: 无法访问/root/oceabase/oceanbase/store/clog/tenant_1/: 没有那个文件或目录
[2023-06-06 15:41:35.792] [DEBUG]
[2023-06-06 15:41:35.792] [DEBUG] – root@192.168.8.31 execute: cat /root/oceabase/oceanbase/run/observer.pid
[2023-06-06 15:41:35.864] [DEBUG] – exited code 1, error output:
[2023-06-06 15:41:35.864] [DEBUG] cat: /root/oceabase/oceanbase/run/observer.pid: 没有那个文件或目录
[2023-06-06 15:41:35.864] [DEBUG]
[2023-06-06 15:41:35.865] [DEBUG] – 192.168.8.31 port check
[2023-06-06 15:41:35.866] [DEBUG] – root@192.168.8.31 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-06-06 15:41:35.951] [DEBUG] – exited code 0
[2023-06-06 15:41:35.951] [DEBUG] – root@192.168.8.31 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-06-06 15:41:36.025] [DEBUG] – exited code 0
[2023-06-06 15:41:36.026] [DEBUG] – root@192.168.8.31 execute: ls /root/oceabase/oceanbase/store/sstable/block_file
[2023-06-06 15:41:36.094] [DEBUG] – exited code 2, error output:
[2023-06-06 15:41:36.094] [DEBUG] ls: 无法访问/root/oceabase/oceanbase/store/sstable/block_file: 没有那个文件或目录
[2023-06-06 15:41:36.094] [DEBUG]
[2023-06-06 15:41:36.095] [DEBUG] – root@192.168.8.32 execute: ls /root/oceabase/oceanbase/store/clog/tenant_1/
[2023-06-06 15:41:36.118] [DEBUG] – exited code 2, error output:
[2023-06-06 15:41:36.119] [DEBUG] ls: 无法访问/root/oceabase/oceanbase/store/clog/tenant_1/: 没有那个文件或目录
[2023-06-06 15:41:36.119] [DEBUG]
[2023-06-06 15:41:36.119] [DEBUG] – root@192.168.8.32 execute: cat /root/oceabase/oceanbase/run/observer.pid
[2023-06-06 15:41:36.185] [DEBUG] – exited code 1, error output:
[2023-06-06 15:41:36.186] [DEBUG] cat: /root/oceabase/oceanbase/run/observer.pid: 没有那个文件或目录
[2023-06-06 15:41:36.186] [DEBUG]
[2023-06-06 15:41:36.186] [DEBUG] – 192.168.8.32 port check
[2023-06-06 15:41:36.186] [DEBUG] – root@192.168.8.32 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B41’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-06-06 15:41:36.262] [DEBUG] – exited code 0
[2023-06-06 15:41:36.263] [DEBUG] – root@192.168.8.32 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{print $2,$10}’ | grep ‘00000000:0B42’ | awk -F’ ’ ‘{print $2}’ | uniq
[2023-06-06 15:41:36.342] [DEBUG] – exited code 0
[2023-06-06 15:41:36.343] [DEBUG] – root@192.168.8.32 execute: ls /root/oceabase/oceanbase/store/sstable/block_file
[2023-06-06 15:41:36.409] [DEBUG] – exited code 2, error output:
[2023-06-06 15:41:36.409] [DEBUG] ls: 无法访问/root/oceabase/oceanbase/store/sstable/block_file: 没有那个文件或目录
[2023-06-06 15:41:36.409] [DEBUG]
[2023-06-06 15:41:36.409] [DEBUG] – root@192.168.8.30 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2023-06-06 15:41:36.453] [DEBUG] – exited code 0
[2023-06-06 15:41:36.453] [DEBUG] – root@192.168.8.30 execute: ulimit -a
[2023-06-06 15:41:36.539] [DEBUG] – exited code 0
[2023-06-06 15:41:36.540] [DEBUG] – root@192.168.8.30 execute: cat /proc/meminfo
[2023-06-06 15:41:36.630] [DEBUG] – exited code 0
[2023-06-06 15:41:36.630] [DEBUG] – root@192.168.8.30 execute: df --block-size=1024
[2023-06-06 15:41:36.723] [DEBUG] – exited code 0
[2023-06-06 15:41:36.723] [DEBUG] – get disk info for path /dev, total: 8310882304 avail: 8310882304
[2023-06-06 15:41:36.723] [DEBUG] – get disk info for path /dev/shm, total: 8328204288 avail: 8328204288
[2023-06-06 15:41:36.723] [DEBUG] – get disk info for path /run, total: 8328204288 avail: 8318201856
[2023-06-06 15:41:36.723] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 8328204288 avail: 8328204288
[2023-06-06 15:41:36.723] [DEBUG] – get disk info for path /, total: 102081433600 avail: 79569133568
[2023-06-06 15:41:36.723] [DEBUG] – get disk info for path /boot, total: 1063256064 avail: 812859392
[2023-06-06 15:41:36.723] [DEBUG] – get disk info for path /run/user/1000, total: 1665642496 avail: 1665642496
[2023-06-06 15:41:36.723] [DEBUG] – get disk info for path /run/user/42, total: 1665642496 avail: 1665630208
[2023-06-06 15:41:36.724] [DEBUG] – get disk info for path /run/user/0, total: 1665642496 avail: 1665642496
[2023-06-06 15:41:36.724] [DEBUG] – root@192.168.8.30 execute: df --block-size=1024 /root/oceabase/oceanbase/store
[2023-06-06 15:41:36.816] [DEBUG] – exited code 0
[2023-06-06 15:41:36.816] [DEBUG] – get disk info for path /, total: 102081433600 avail: 79569133568
[2023-06-06 15:41:36.816] [DEBUG] – root@192.168.8.30 execute: df --block-size=1024 /root/oceabase/oceanbase/store/clog
[2023-06-06 15:41:36.912] [DEBUG] – exited code 0
[2023-06-06 15:41:36.913] [DEBUG] – get disk info for path /, total: 102081433600 avail: 79569133568
[2023-06-06 15:41:36.913] [DEBUG] – disk: {’/dev’: {‘total’: 8310882304, ‘avail’: 8310882304, ‘need’: 0}, ‘/dev/shm’: {‘total’: 8328204288, ‘avail’: 8328204288, ‘need’: 0}, ‘/run’: {‘total’: 8328204288, ‘avail’: 8318201856, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 8328204288, ‘avail’: 8328204288, ‘need’: 0}, ‘/’: {‘total’: 102081433600, ‘avail’: 79569133568, ‘need’: 0}, ‘/boot’: {‘total’: 1063256064, ‘avail’: 812859392, ‘need’: 0}, ‘/run/user/1000’: {‘total’: 1665642496, ‘avail’: 1665642496, ‘need’: 0}, ‘/run/user/42’: {‘total’: 1665642496, ‘avail’: 1665630208, ‘need’: 0}, ‘/run/user/0’: {‘total’: 1665642496, ‘avail’: 1665642496, ‘need’: 0}}
[2023-06-06 15:41:36.913] [WARNING] OBD-1012: (192.168.8.30) clog and data use the same disk (/)
[2023-06-06 15:41:36.913] [DEBUG] – root@192.168.8.31 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2023-06-06 15:41:36.945] [DEBUG] – exited code 0
[2023-06-06 15:41:36.946] [DEBUG] – root@192.168.8.31 execute: ulimit -a
[2023-06-06 15:41:37.015] [DEBUG] – exited code 0
[2023-06-06 15:41:37.016] [DEBUG] – root@192.168.8.31 execute: cat /proc/meminfo
[2023-06-06 15:41:37.085] [DEBUG] – exited code 0
[2023-06-06 15:41:37.086] [DEBUG] – root@192.168.8.31 execute: df --block-size=1024
[2023-06-06 15:41:37.156] [DEBUG] – exited code 0
[2023-06-06 15:41:37.157] [DEBUG] – get disk info for path /dev, total: 8310882304 avail: 8310882304
[2023-06-06 15:41:37.157] [DEBUG] – get disk info for path /dev/shm, total: 8328204288 avail: 8328204288
[2023-06-06 15:41:37.157] [DEBUG] – get disk info for path /run, total: 8328204288 avail: 8318246912
[2023-06-06 15:41:37.157] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 8328204288 avail: 8328204288
[2023-06-06 15:41:37.157] [DEBUG] – get disk info for path /, total: 102081433600 avail: 84332998656
[2023-06-06 15:41:37.157] [DEBUG] – get disk info for path /boot, total: 1063256064 avail: 755970048
[2023-06-06 15:41:37.157] [DEBUG] – get disk info for path /run/user/1000, total: 1665642496 avail: 1665642496
[2023-06-06 15:41:37.158] [DEBUG] – get disk info for path /run/user/42, total: 1665642496 avail: 1665630208
[2023-06-06 15:41:37.158] [DEBUG] – get disk info for path /run/user/0, total: 1665642496 avail: 1665642496
[2023-06-06 15:41:37.158] [DEBUG] – get disk info for path /run/user/1001, total: 1665642496 avail: 1665642496
[2023-06-06 15:41:37.158] [DEBUG] – root@192.168.8.31 execute: df --block-size=1024 /root/oceabase/oceanbase/store
[2023-06-06 15:41:37.222] [DEBUG] – exited code 0
[2023-06-06 15:41:37.223] [DEBUG] – get disk info for path /, total: 102081433600 avail: 84332998656
[2023-06-06 15:41:37.223] [DEBUG] – root@192.168.8.31 execute: df --block-size=1024 /root/oceabase/oceanbase/store/clog
[2023-06-06 15:41:37.287] [DEBUG] – exited code 0
[2023-06-06 15:41:37.288] [DEBUG] – get disk info for path /, total: 102081433600 avail: 84332998656
[2023-06-06 15:41:37.288] [DEBUG] – disk: {’/dev’: {‘total’: 8310882304, ‘avail’: 8310882304, ‘need’: 0}, ‘/dev/shm’: {‘total’: 8328204288, ‘avail’: 8328204288, ‘need’: 0}, ‘/run’: {‘total’: 8328204288, ‘avail’: 8318246912, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 8328204288, ‘avail’: 8328204288, ‘need’: 0}, ‘/’: {‘total’: 102081433600, ‘avail’: 84332998656, ‘need’: 0}, ‘/boot’: {‘total’: 1063256064, ‘avail’: 755970048, ‘need’: 0}, ‘/run/user/1000’: {‘total’: 1665642496, ‘avail’: 1665642496, ‘need’: 0}, ‘/run/user/42’: {‘total’: 1665642496, ‘avail’: 1665630208, ‘need’: 0}, ‘/run/user/0’: {‘total’: 1665642496, ‘avail’: 1665642496, ‘need’: 0}, ‘/run/user/1001’: {‘total’: 1665642496, ‘avail’: 1665642496, ‘need’: 0}}
[2023-06-06 15:41:37.288] [WARNING] OBD-1012: (192.168.8.31) clog and data use the same disk (/)
[2023-06-06 15:41:37.289] [DEBUG] – root@192.168.8.32 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2023-06-06 15:41:37.317] [DEBUG] – exited code 0
[2023-06-06 15:41:37.318] [DEBUG] – root@192.168.8.32 execute: ulimit -a
[2023-06-06 15:41:37.387] [DEBUG] – exited code 0
[2023-06-06 15:41:37.388] [DEBUG] – root@192.168.8.32 execute: cat /proc/meminfo
[2023-06-06 15:41:37.465] [DEBUG] – exited code 0
[2023-06-06 15:41:37.466] [DEBUG] – root@192.168.8.32 execute: df --block-size=1024
[2023-06-06 15:41:37.539] [DEBUG] – exited code 0
[2023-06-06 15:41:37.540] [DEBUG] – get disk info for path /dev, total: 8310890496 avail: 8310890496
[2023-06-06 15:41:37.540] [DEBUG] – get disk info for path /dev/shm, total: 8328212480 avail: 8328212480
[2023-06-06 15:41:37.540] [DEBUG] – get disk info for path /run, total: 8328212480 avail: 8318304256
[2023-06-06 15:41:37.540] [DEBUG] – get disk info for path /sys/fs/cgroup, total: 8328212480 avail: 8328212480
[2023-06-06 15:41:37.541] [DEBUG] – get disk info for path /, total: 102081433600 avail: 92817354752
[2023-06-06 15:41:37.541] [DEBUG] – get disk info for path /boot, total: 1063256064 avail: 812859392
[2023-06-06 15:41:37.541] [DEBUG] – get disk info for path /run/user/1000, total: 1665646592 avail: 1665646592
[2023-06-06 15:41:37.541] [DEBUG] – get disk info for path /run/user/42, total: 1665646592 avail: 1665634304
[2023-06-06 15:41:37.541] [DEBUG] – get disk info for path /run/user/0, total: 1665646592 avail: 1665646592
[2023-06-06 15:41:37.541] [DEBUG] – root@192.168.8.32 execute: df --block-size=1024 /root/oceabase/oceanbase/store
[2023-06-06 15:41:37.612] [DEBUG] – exited code 0
[2023-06-06 15:41:37.612] [DEBUG] – get disk info for path /, total: 102081433600 avail: 92817354752
[2023-06-06 15:41:37.612] [DEBUG] – root@192.168.8.32 execute: df --block-size=1024 /root/oceabase/oceanbase/store/clog
[2023-06-06 15:41:37.686] [DEBUG] – exited code 0
[2023-06-06 15:41:37.687] [DEBUG] – get disk info for path /, total: 102081433600 avail: 92817354752
[2023-06-06 15:41:37.688] [DEBUG] – disk: {’/dev’: {‘total’: 8310890496, ‘avail’: 8310890496, ‘need’: 0}, ‘/dev/shm’: {‘total’: 8328212480, ‘avail’: 8328212480, ‘need’: 0}, ‘/run’: {‘total’: 8328212480, ‘avail’: 8318304256, ‘need’: 0}, ‘/sys/fs/cgroup’: {‘total’: 8328212480, ‘avail’: 8328212480, ‘need’: 0}, ‘/’: {‘total’: 102081433600, ‘avail’: 92817354752, ‘need’: 0}, ‘/boot’: {‘total’: 1063256064, ‘avail’: 812859392, ‘need’: 0}, ‘/run/user/1000’: {‘total’: 1665646592, ‘avail’: 1665646592, ‘need’: 0}, ‘/run/user/42’: {‘total’: 1665646592, ‘avail’: 1665634304, ‘need’: 0}, ‘/run/user/0’: {‘total’: 1665646592, ‘avail’: 1665646592, ‘need’: 0}}
[2023-06-06 15:41:37.688] [WARNING] OBD-1012: (192.168.8.32) clog and data use the same disk (/)
[2023-06-06 15:41:37.691] [DEBUG] – root@192.168.8.32 execute: cat /proc/net/dev
[2023-06-06 15:41:37.758] [DEBUG] – exited code 0
[2023-06-06 15:41:37.759] [DEBUG] – root@192.168.8.32 execute: cat /proc/net/dev
[2023-06-06 15:41:37.833] [DEBUG] – exited code 0
[2023-06-06 15:41:37.834] [DEBUG] – root@192.168.8.32 execute: cat /proc/net/dev
[2023-06-06 15:41:37.907] [DEBUG] – exited code 0
[2023-06-06 15:41:37.908] [DEBUG] – root@192.168.8.30 execute: ping -W 1 -c 1 -I eth0 192.168.8.31
[2023-06-06 15:41:37.970] [DEBUG] – exited code 0
[2023-06-06 15:41:37.970] [DEBUG] – root@192.168.8.30 execute: ping -W 1 -c 1 -I eth0 192.168.8.32
[2023-06-06 15:41:38.074] [DEBUG] – exited code 0
[2023-06-06 15:41:38.075] [DEBUG] – root@192.168.8.31 execute: ping -W 1 -c 1 -I eth0 192.168.8.30
[2023-06-06 15:41:38.118] [DEBUG] – exited code 0
[2023-06-06 15:41:38.118] [DEBUG] – root@192.168.8.31 execute: ping -W 1 -c 1 -I eth0 192.168.8.32
[2023-06-06 15:41:38.191] [DEBUG] – exited code 0
[2023-06-06 15:41:38.193] [DEBUG] – root@192.168.8.32 execute: ping -W 1 -c 1 -I eth0 192.168.8.30
[2023-06-06 15:41:38.234] [DEBUG] – exited code 0
[2023-06-06 15:41:38.235] [DEBUG] – root@192.168.8.32 execute: ping -W 1 -c 1 -I eth0 192.168.8.31
[2023-06-06 15:41:38.311] [DEBUG] – exited code 0
[2023-06-06 15:41:38.311] [DEBUG] – root@192.168.8.30 execute: date +%s%N
[2023-06-06 15:41:38.362] [DEBUG] – exited code 0
[2023-06-06 15:41:38.363] [DEBUG] – 192.168.8.30 time delta -3.0576171875
[2023-06-06 15:41:38.363] [DEBUG] – root@192.168.8.31 execute: date +%s%N
[2023-06-06 15:41:38.399] [DEBUG] – exited code 0
[2023-06-06 15:41:38.400] [DEBUG] – 192.168.8.31 time delta -3.918701171875
[2023-06-06 15:41:38.400] [DEBUG] – root@192.168.8.32 execute: date +%s%N
[2023-06-06 15:41:38.433] [DEBUG] – exited code 0
[2023-06-06 15:41:38.434] [DEBUG] – 192.168.8.32 time delta 2.466552734375
[2023-06-06 15:41:38.434] [INFO] [WARN] OBD-1012: (192.168.8.30) clog and data use the same disk (/)
[2023-06-06 15:41:38.434] [INFO] [WARN] OBD-1012: (192.168.8.31) clog and data use the same disk (/)
[2023-06-06 15:41:38.434] [INFO] [WARN] OBD-1012: (192.168.8.32) clog and data use the same disk (/)
[2023-06-06 15:41:38.434] [INFO]
[2023-06-06 15:41:38.435] [DEBUG] - Call oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-06 15:41:38.435] [DEBUG] - import start
[2023-06-06 15:41:38.439] [DEBUG] - add start ref count to 1
[2023-06-06 15:41:38.440] [INFO] Start observer
[2023-06-06 15:41:38.440] [DEBUG] – root@192.168.8.30 execute: ls /root/oceabase/oceanbase/store/clog/tenant_1/
[2023-06-06 15:41:38.483] [DEBUG] – exited code 2, error output:
[2023-06-06 15:41:38.483] [DEBUG] ls: cannot access /root/oceabase/oceanbase/store/clog/tenant_1/: No such file or directory
[2023-06-06 15:41:38.483] [DEBUG]
[2023-06-06 15:41:38.484] [DEBUG] – root@192.168.8.30 execute: cat /root/oceabase/oceanbase/run/observer.pid
[2023-06-06 15:41:38.580] [DEBUG] – exited code 1, error output:
[2023-06-06 15:41:38.580] [DEBUG] cat: /root/oceabase/oceanbase/run/observer.pid: No such file or directory
[2023-06-06 15:41:38.581] [DEBUG]
[2023-06-06 15:41:38.581] [DEBUG] – 192.168.8.30 start command construction
[2023-06-06 15:41:38.581] [DEBUG] – root@192.168.8.31 execute: ls /root/oceabase/oceanbase/store/clog/tenant_1/
[2023-06-06 15:41:38.615] [DEBUG] – exited code 2, error output:
[2023-06-06 15:41:38.616] [DEBUG] ls: 无法访问/root/oceabase/oceanbase/store/clog/tenant_1/: 没有那个文件或目录
[2023-06-06 15:41:38.616] [DEBUG]
[2023-06-06 15:41:38.616] [DEBUG] – root@192.168.8.31 execute: cat /root/oceabase/oceanbase/run/observer.pid
[2023-06-06 15:41:38.684] [DEBUG] – exited code 1, error output:
[2023-06-06 15:41:38.684] [DEBUG] cat: /root/oceabase/oceanbase/run/observer.pid: 没有那个文件或目录
[2023-06-06 15:41:38.684] [DEBUG]
[2023-06-06 15:41:38.685] [DEBUG] – 192.168.8.31 start command construction
[2023-06-06 15:41:38.685] [DEBUG] – root@192.168.8.32 execute: ls /root/oceabase/oceanbase/store/clog/tenant_1/
[2023-06-06 15:41:38.723] [DEBUG] – exited code 2, error output:
[2023-06-06 15:41:38.723] [DEBUG] ls: 无法访问/root/oceabase/oceanbase/store/clog/tenant_1/: 没有那个文件或目录
[2023-06-06 15:41:38.724] [DEBUG]
[2023-06-06 15:41:38.724] [DEBUG] – root@192.168.8.32 execute: cat /root/oceabase/oceanbase/run/observer.pid
[2023-06-06 15:41:38.800] [DEBUG] – exited code 1, error output:
[2023-06-06 15:41:38.800] [DEBUG] cat: /root/oceabase/oceanbase/run/observer.pid: 没有那个文件或目录
[2023-06-06 15:41:38.800] [DEBUG]
[2023-06-06 15:41:38.801] [DEBUG] – 192.168.8.32 start command construction
[2023-06-06 15:41:38.802] [DEBUG] – starting 192.168.8.30 observer
[2023-06-06 15:41:38.802] [DEBUG] – root@192.168.8.30 set env LD_LIBRARY_PATH to ‘/root/oceabase/oceanbase/lib:’
[2023-06-06 15:41:38.802] [DEBUG] – root@192.168.8.30 execute: cd /root/oceabase/oceanbase; /root/oceabase/oceanbase/bin/observer -r ‘192.168.8.30:2882:2881;192.168.8.31:2882:2881;192.168.8.32:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -n ‘oceabase’ -c 1 -d ‘/root/oceabase/oceanbase/store’ -o __min_full_resource_pool_memory=1073741824,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit=‘6G’,datafile_size=‘20G’,system_memory=‘1G’,log_disk_size=‘15G’,cpu_count=16
[2023-06-06 15:41:39.001] [DEBUG] – exited code 0
[2023-06-06 15:41:39.002] [DEBUG] – root@192.168.8.30 set env LD_LIBRARY_PATH to ‘’
[2023-06-06 15:41:39.002] [DEBUG] – starting 192.168.8.31 observer
[2023-06-06 15:41:39.002] [DEBUG] – root@192.168.8.31 set env LD_LIBRARY_PATH to ‘/root/oceabase/oceanbase/lib:’
[2023-06-06 15:41:39.002] [DEBUG] – root@192.168.8.31 execute: cd /root/oceabase/oceanbase; /root/oceabase/oceanbase/bin/observer -r ‘192.168.8.30:2882:2881;192.168.8.31:2882:2881;192.168.8.32:2882:2881’ -p 2881 -P 2882 -z ‘zone2’ -n ‘oceabase’ -c 1 -d ‘/root/oceabase/oceanbase/store’ -o __min_full_resource_pool_memory=1073741824,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit=‘6G’,datafile_size=‘20G’,system_memory=‘1G’,log_disk_size=‘15G’,cpu_count=16
[2023-06-06 15:41:39.288] [DEBUG] – exited code 0
[2023-06-06 15:41:39.289] [DEBUG] – root@192.168.8.31 set env LD_LIBRARY_PATH to ‘’
[2023-06-06 15:41:39.289] [DEBUG] – starting 192.168.8.32 observer
[2023-06-06 15:41:39.290] [DEBUG] – root@192.168.8.32 set env LD_LIBRARY_PATH to ‘/root/oceabase/oceanbase/lib:’
[2023-06-06 15:41:39.290] [DEBUG] – root@192.168.8.32 execute: cd /root/oceabase/oceanbase; /root/oceabase/oceanbase/bin/observer -r ‘192.168.8.30:2882:2881;192.168.8.31:2882:2881;192.168.8.32:2882:2881’ -p 2881 -P 2882 -z ‘zone3’ -n ‘oceabase’ -c 1 -d ‘/root/oceabase/oceanbase/store’ -o __min_full_resource_pool_memory=1073741824,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit=‘6G’,datafile_size=‘20G’,system_memory=‘1G’,log_disk_size=‘15G’,cpu_count=16
[2023-06-06 15:41:39.487] [DEBUG] – exited code 0
[2023-06-06 15:41:39.488] [DEBUG] – root@192.168.8.32 set env LD_LIBRARY_PATH to ‘’
[2023-06-06 15:41:39.489] [INFO] observer program health check
[2023-06-06 15:41:42.491] [DEBUG] – 192.168.8.30 program health check
[2023-06-06 15:41:42.492] [DEBUG] – root@192.168.8.30 execute: cat /root/oceabase/oceanbase/run/observer.pid
[2023-06-06 15:41:42.547] [DEBUG] – exited code 0
[2023-06-06 15:41:42.547] [DEBUG] – root@192.168.8.30 execute: ls /proc/6081
[2023-06-06 15:41:42.651] [DEBUG] – exited code 0
[2023-06-06 15:41:42.651] [DEBUG] – 192.168.8.30 observer[pid: 6081] started
[2023-06-06 15:41:42.651] [DEBUG] – 192.168.8.31 program health check
[2023-06-06 15:41:42.651] [DEBUG] – root@192.168.8.31 execute: cat /root/oceabase/oceanbase/run/observer.pid
[2023-06-06 15:41:42.681] [DEBUG] – exited code 0
[2023-06-06 15:41:42.681] [DEBUG] – root@192.168.8.31 execute: ls /proc/4850
[2023-06-06 15:41:42.750] [DEBUG] – exited code 0
[2023-06-06 15:41:42.750] [DEBUG] – 192.168.8.31 observer[pid: 4850] started
[2023-06-06 15:41:42.750] [DEBUG] – 192.168.8.32 program health check
[2023-06-06 15:41:42.750] [DEBUG] – root@192.168.8.32 execute: cat /root/oceabase/oceanbase/run/observer.pid
[2023-06-06 15:41:42.779] [DEBUG] – exited code 0
[2023-06-06 15:41:42.780] [DEBUG] – root@192.168.8.32 execute: ls /proc/4632
[2023-06-06 15:41:42.847] [DEBUG] – exited code 0
[2023-06-06 15:41:42.848] [DEBUG] – 192.168.8.32 observer[pid: 4632] started
[2023-06-06 15:41:42.848] [DEBUG] - sub start ref count to 0
[2023-06-06 15:41:42.848] [DEBUG] - export start
[2023-06-06 15:41:42.848] [DEBUG] - Call oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-06 15:41:42.848] [DEBUG] - import connect
[2023-06-06 15:41:42.911] [DEBUG] - add connect ref count to 1
[2023-06-06 15:41:42.912] [INFO] Connect to observer
[2023-06-06 15:41:42.912] [DEBUG] – connect 192.168.8.30 -P2881 -uroot -pmaotuan123S
[2023-06-06 15:41:43.054] [DEBUG] – connect 192.168.8.31 -P2881 -uroot -pmaotuan123S
[2023-06-06 15:41:53.068] [DEBUG] – connect 192.168.8.32 -P2881 -uroot -pmaotuan123S
[2023-06-06 15:41:56.231] [DEBUG] – connect 192.168.8.30 -P2881 -uroot -p
[2023-06-06 15:41:56.234] [DEBUG] - sub connect ref count to 0
[2023-06-06 15:41:56.234] [DEBUG] - export connect
[2023-06-06 15:41:56.235] [INFO] Initialize oceanbase-ce
[2023-06-06 15:41:56.236] [DEBUG] - Call oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.1.0.0-101010022023051821.el7-21271468e0dee7aaf3d4eff4c4bf5e07421ef6fe
[2023-06-06 15:41:56.236] [DEBUG] - import bootstrap
[2023-06-06 15:41:56.239] [DEBUG] - add bootstrap ref count to 1
[2023-06-06 15:41:56.239] [DEBUG] – execute sql: set session ob_query_timeout=1000000000
[2023-06-06 15:41:56.239] [DEBUG] – execute sql: set session ob_query_timeout=1000000000. args: None
[2023-06-06 15:41:56.243] [DEBUG] – execute sql: alter system bootstrap REGION “sys_region” ZONE “zone1” SERVER “192.168.8.30:2882”,REGION “sys_region” ZONE “zone2” SERVER “192.168.8.31:2882”,REGION “sys_region” ZONE “zone3” SERVER “192.168.8.32:2882”. args: None
[2023-06-06 15:58:36.271] [DEBUG] – OBD-5000: alter system bootstrap REGION “sys_region” ZONE “zone1” SERVER “192.168.8.30:2882”,REGION “sys_region” ZONE “zone2” SERVER “192.168.8.31:2882”,REGION “sys_region” ZONE “zone3” SERVER “192.168.8.32:2882” execute failed
[2023-06-06 15:58:36.274] [ERROR] Traceback (most recent call last):
[2023-06-06 15:58:36.274] [ERROR] File “service/handler/deployment_handler.py”, line 314, in _do_install
[2023-06-06 15:58:36.274] [ERROR] File “core.py”, line 1676, in _start_cluster
[2023-06-06 15:58:36.275] [ERROR] File “core.py”, line 184, in call_plugin
[2023-06-06 15:58:36.275] [ERROR] File “_plugin.py”, line 323, in call
[2023-06-06 15:58:36.275] [ERROR] File “_plugin.py”, line 285, in _new_func
[2023-06-06 15:58:36.275] [ERROR] File “/root/.obd/plugins/oceanbase-ce/4.0.0.0/bootstrap.py”, line 78, in bootstrap
[2023-06-06 15:58:36.275] [ERROR] raise_cursor.execute(sql, exc_level=‘verbose’)
[2023-06-06 15:58:36.275] [ERROR] File “_stdio.py”, line 885, in func_wrapper
[2023-06-06 15:58:36.275] [ERROR] File “/root/.obd/plugins/oceanbase-ce/3.1.0/connect.py”, line 91, in execute
[2023-06-06 15:58:36.275] [ERROR] self.cursor.execute(sql, args)
[2023-06-06 15:58:36.275] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/cursors.py”, line 148, in execute
[2023-06-06 15:58:36.275] [ERROR] result = self._query(query)
[2023-06-06 15:58:36.275] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/cursors.py”, line 310, in _query
[2023-06-06 15:58:36.275] [ERROR] conn.query(q)
[2023-06-06 15:58:36.276] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 548, in query
[2023-06-06 15:58:36.276] [ERROR] self._affected_rows = self._read_query_result(unbuffered=unbuffered)
[2023-06-06 15:58:36.276] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 775, in _read_query_result
[2023-06-06 15:58:36.276] [ERROR] result.read()
[2023-06-06 15:58:36.276] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 1156, in read
[2023-06-06 15:58:36.276] [ERROR] first_packet = self.connection._read_packet()
[2023-06-06 15:58:36.276] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/connections.py”, line 725, in _read_packet
[2023-06-06 15:58:36.276] [ERROR] packet.raise_for_error()
[2023-06-06 15:58:36.276] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/protocol.py”, line 221, in raise_for_error
[2023-06-06 15:58:36.276] [ERROR] err.raise_mysql_exception(self._data)
[2023-06-06 15:58:36.276] [ERROR] File “/root/.oceanbase-all-in-one/obd/usr/obd/lib/site-packages/pymysql/err.py”, line 143, in raise_mysql_exception
[2023-06-06 15:58:36.276] [ERROR] raise errorclass(errno, errval)
[2023-06-06 15:58:36.277] [ERROR] pymysql.err.OperationalError: (4012, ‘Timeout’)
[2023-06-06 15:58:36.277] [ERROR]
[2023-06-06 15:58:36.277] [DEBUG] – execute sql: select column_value from oceanbase.__all_core_table where table_name = ‘__all_global_stat’ and column_name = ‘baseline_schema_version’. args: None
[2023-06-06 16:15:16.334] [DEBUG] – OBD-5000: select column_value from oceanbase.__all_core_table where table_name = ‘__all_global_stat’ and column_name = ‘baseline_schema_version’ execute failed
[2023-06-06 16:15:16.335] [DEBUG] - sub bootstrap ref count to 0
[2023-06-06 16:15:16.335] [DEBUG] - export bootstrap
[2023-06-06 16:15:16.335] [ERROR] Cluster init failed
【附件】