[2024-07-24 09:11:19.110] [DEBUG] - cmd: [‘obtest’]
[2024-07-24 09:11:19.110] [DEBUG] - opts: {‘servers’: None, ‘components’: None, ‘force_delete’: None, ‘strict_check’: None, ‘without_parameter’: None}
[2024-07-24 09:11:19.110] [DEBUG] - mkdir /home/wsw/.obd/lock/
[2024-07-24 09:11:19.110] [DEBUG] - unknown lock mode
[2024-07-24 09:11:19.111] [DEBUG] - try to get share lock /home/wsw/.obd/lock/global
[2024-07-24 09:11:19.111] [DEBUG] - share lock /home/wsw/.obd/lock/global, count 1
[2024-07-24 09:11:19.111] [DEBUG] - Get Deploy by name
[2024-07-24 09:11:19.111] [DEBUG] - mkdir /home/wsw/.obd/cluster/
[2024-07-24 09:11:19.112] [DEBUG] - mkdir /home/wsw/.obd/config_parser/
[2024-07-24 09:11:19.112] [DEBUG] - try to get exclusive lock /home/wsw/.obd/lock/deploy_obtest
[2024-07-24 09:11:19.112] [DEBUG] - exclusive lock /home/wsw/.obd/lock/deploy_obtest, count 1
[2024-07-24 09:11:19.117] [DEBUG] - Deploy status judge
[2024-07-24 09:11:19.119] [INFO] Get local repositories
[2024-07-24 09:11:19.119] [DEBUG] - mkdir /home/wsw/.obd/repository
[2024-07-24 09:11:19.120] [DEBUG] - Get local repository oceanbase-ce-4.2.3.0-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.120] [DEBUG] - Search repository oceanbase-ce version: 4.2.3.0, tag: ef16f0db100e41625be18924797b9f2bc17967d5, release: None, package_hash: None
[2024-07-24 09:11:19.120] [DEBUG] - try to get share lock /home/wsw/.obd/lock/mirror_and_repo
[2024-07-24 09:11:19.120] [DEBUG] - share lock /home/wsw/.obd/lock/mirror_and_repo, count 1
[2024-07-24 09:11:19.120] [DEBUG] - mkdir /home/wsw/.obd/repository/oceanbase-ce
[2024-07-24 09:11:19.123] [DEBUG] - Found repository oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.251] [DEBUG] - Get deploy config
[2024-07-24 09:11:19.263] [INFO] Search plugins
[2024-07-24 09:11:19.264] [DEBUG] - Searching start_check plugin for components …
[2024-07-24 09:11:19.264] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.264] [DEBUG] - mkdir /home/wsw/.obd/plugins
[2024-07-24 09:11:19.268] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.3.0
[2024-07-24 09:11:19.268] [DEBUG] - Searching create_tenant plugin for components …
[2024-07-24 09:11:19.268] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.268] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.3.0
[2024-07-24 09:11:19.268] [DEBUG] - Searching tenant_optimize plugin for components …
[2024-07-24 09:11:19.268] [DEBUG] - Searching tenant_optimize plugin for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.269] [DEBUG] - No such tenant_optimize plugin for oceanbase-ce-4.2.3.0
[2024-07-24 09:11:19.269] [DEBUG] - Searching start plugin for components …
[2024-07-24 09:11:19.269] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.269] [DEBUG] - Found for oceanbase-ce-py_script_start-4.2.2.0 for oceanbase-ce-4.2.3.0
[2024-07-24 09:11:19.269] [DEBUG] - Searching connect plugin for components …
[2024-07-24 09:11:19.269] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.270] [DEBUG] - Found for oceanbase-ce-py_script_connect-4.2.2.0 for oceanbase-ce-4.2.3.0
[2024-07-24 09:11:19.270] [DEBUG] - Searching bootstrap plugin for components …
[2024-07-24 09:11:19.270] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.270] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.2.0 for oceanbase-ce-4.2.3.0
[2024-07-24 09:11:19.270] [DEBUG] - Searching display plugin for components …
[2024-07-24 09:11:19.270] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.271] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.3.0
[2024-07-24 09:11:19.397] [INFO] Load cluster param plugin
[2024-07-24 09:11:19.398] [DEBUG] - Get local repository oceanbase-ce-4.2.3.0-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.398] [DEBUG] - Searching param plugin for components …
[2024-07-24 09:11:19.398] [DEBUG] - Search param plugin for oceanbase-ce
[2024-07-24 09:11:19.398] [DEBUG] - Found for oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.3.0
[2024-07-24 09:11:19.398] [DEBUG] - Applying oceanbase-ce-param-4.2.2.0 for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.813] [INFO] Cluster status check
[2024-07-24 09:11:19.814] [DEBUG] - Searching status plugin for components …
[2024-07-24 09:11:19.814] [DEBUG] - Searching status plugin for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.815] [DEBUG] - Found for oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.3.0
[2024-07-24 09:11:19.815] [DEBUG] - host: 192.168.137.70, port: 22, user: wsw, password: root1234
[2024-07-24 09:11:19.947] [DEBUG] - Call oceanbase-ce-py_script_status-3.1.0 for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:19.947] [DEBUG] - import status
[2024-07-24 09:11:19.949] [DEBUG] - add status ref count to 1
[2024-07-24 09:11:19.949] [DEBUG] – wsw@192.168.137.70 execute: cat /home/admin/observer/run/observer.pid
[2024-07-24 09:11:19.955] [DEBUG] – exited code 0
[2024-07-24 09:11:19.955] [DEBUG] – wsw@192.168.137.70 execute: ls /proc/2278
[2024-07-24 09:11:20.001] [DEBUG] – exited code 2, error output:
[2024-07-24 09:11:20.001] [DEBUG] ls: 无法访问’/proc/2278’: 没有那个文件或目录
[2024-07-24 09:11:20.001] [DEBUG]
[2024-07-24 09:11:20.001] [DEBUG] - sub status ref count to 0
[2024-07-24 09:11:20.001] [DEBUG] - export status
[2024-07-24 09:11:20.001] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.2.0 for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:20.001] [DEBUG] - import start_check
[2024-07-24 09:11:20.013] [DEBUG] - add start_check ref count to 1
[2024-07-24 09:11:20.014] [INFO] Check before start observer
[2024-07-24 09:11:20.016] [DEBUG] – wsw@192.168.137.70 execute: ls /opt/software/data/clog/tenant_1/
[2024-07-24 09:11:20.051] [DEBUG] – exited code 0
[2024-07-24 09:11:20.052] [DEBUG] – wsw@192.168.137.70 execute: cat /home/admin/observer/run/observer.pid
[2024-07-24 09:11:20.097] [DEBUG] – exited code 0
[2024-07-24 09:11:20.098] [DEBUG] – wsw@192.168.137.70 execute: ls /proc/2278
[2024-07-24 09:11:20.143] [DEBUG] – exited code 2, error output:
[2024-07-24 09:11:20.143] [DEBUG] ls: 无法访问’/proc/2278’: 没有那个文件或目录
[2024-07-24 09:11:20.143] [DEBUG]
[2024-07-24 09:11:20.143] [DEBUG] – 192.168.137.70 port check
[2024-07-24 09:11:20.144] [DEBUG] – wsw@192.168.137.70 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B41’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-07-24 09:11:20.200] [DEBUG] – exited code 0
[2024-07-24 09:11:20.200] [DEBUG] – wsw@192.168.137.70 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B42’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-07-24 09:11:20.249] [DEBUG] – exited code 0
[2024-07-24 09:11:20.249] [DEBUG] – wsw@192.168.137.70 execute: bash -c ‘cat /proc/net/{tcp*,udp*}’ | awk -F’ ’ ‘{if($4==“0A”) print $2,$4,$10}’ | grep ‘:0B46’ | awk -F’ ’ ‘{print $3}’ | uniq
[2024-07-24 09:11:20.296] [DEBUG] – exited code 0
[2024-07-24 09:11:20.296] [DEBUG] – wsw@192.168.137.70 execute: ls /opt/software/data/sstable/block_file
[2024-07-24 09:11:20.345] [DEBUG] – exited code 0
[2024-07-24 09:11:20.345] [DEBUG] – wsw@192.168.137.70 execute: [ -w /tmp/ ] || [ -w /tmp/obshell ]
[2024-07-24 09:11:20.388] [DEBUG] – exited code 0
[2024-07-24 09:11:20.388] [DEBUG] – wsw@192.168.137.70 execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr
[2024-07-24 09:11:20.432] [DEBUG] – exited code 0
[2024-07-24 09:11:20.432] [WARNING] OBD-1011: (192.168.137.70) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[2024-07-24 09:11:20.432] [DEBUG] – wsw@192.168.137.70 execute: ulimit -a
[2024-07-24 09:11:20.478] [DEBUG] – exited code 0
[2024-07-24 09:11:20.479] [WARNING] OBD-1007: (192.168.137.70) The recommended number of max user processes is 655350 (Current value: 120000)
[2024-07-24 09:11:20.479] [WARNING] OBD-1007: (192.168.137.70) The recommended number of core file size is unlimited (Current value: 0)
[2024-07-24 09:11:20.479] [WARNING] OBD-1007: (192.168.137.70) The recommended number of stack size is unlimited (Current value: 8192)
[2024-07-24 09:11:20.479] [DEBUG] – wsw@192.168.137.70 execute: sysctl -a
[2024-07-24 09:11:20.553] [DEBUG] – exited code 0
[2024-07-24 09:11:20.555] [WARNING] OBD-1017: (192.168.137.70) The value of the “vm.max_map_count” must be within [327600, 1310720] (Current value: 65530, Recommended value: 655360)
[2024-07-24 09:11:20.555] [DEBUG] – wsw@192.168.137.70 execute: cat /proc/meminfo
[2024-07-24 09:11:20.597] [DEBUG] – exited code 0
[2024-07-24 09:11:20.598] [DEBUG] – wsw@192.168.137.70 execute: df --block-size=1024
[2024-07-24 09:11:20.643] [DEBUG] – exited code 0
[2024-07-24 09:11:20.643] [DEBUG] – get disk info for path /dev, total: 8322252800 avail: 8322252800
[2024-07-24 09:11:20.643] [DEBUG] – get disk info for path /run, total: 1675808768 avail: 1674784768
[2024-07-24 09:11:20.643] [DEBUG] – get disk info for path /, total: 46414041088 avail: 16178466816
[2024-07-24 09:11:20.643] [DEBUG] – get disk info for path /dev/shm, total: 8379023360 avail: 8379002880
[2024-07-24 09:11:20.643] [DEBUG] – get disk info for path /run/lock, total: 5242880 avail: 5238784
[2024-07-24 09:11:20.643] [DEBUG] – get disk info for path /boot, total: 1020702720 avail: 696090624
[2024-07-24 09:11:20.643] [DEBUG] – get disk info for path /data, total: 30979067904 avail: 11417616384
[2024-07-24 09:11:20.643] [DEBUG] – get disk info for path /run/user/1000, total: 1675804672 avail: 1675595776
[2024-07-24 09:11:20.643] [DEBUG] – get disk info for path /media/wsw/openKylin-1.0, total: 3977379840 avail: 0
[2024-07-24 09:11:20.643] [DEBUG] – disk: {’/dev’: {‘total’: 8322252800, ‘avail’: 8322252800, ‘need’: 0}, ‘/run’: {‘total’: 1675808768, ‘avail’: 1674784768, ‘need’: 0}, ‘/’: {‘total’: 46414041088, ‘avail’: 16178466816, ‘need’: 0}, ‘/dev/shm’: {‘total’: 8379023360, ‘avail’: 8379002880, ‘need’: 0}, ‘/run/lock’: {‘total’: 5242880, ‘avail’: 5238784, ‘need’: 0}, ‘/boot’: {‘total’: 1020702720, ‘avail’: 696090624, ‘need’: 0}, ‘/data’: {‘total’: 30979067904, ‘avail’: 11417616384, ‘need’: 0}, ‘/run/user/1000’: {‘total’: 1675804672, ‘avail’: 1675595776, ‘need’: 0}, ‘/media/wsw/openKylin-1.0’: {‘total’: 3977379840, ‘avail’: 0, ‘need’: 0}}
[2024-07-24 09:11:20.644] [DEBUG] – wsw@192.168.137.70 execute: date +%s%N
[2024-07-24 09:11:20.687] [DEBUG] – exited code 0
[2024-07-24 09:11:20.687] [DEBUG] – 192.168.137.70 time delta -0.983154296875
[2024-07-24 09:11:20.801] [INFO] [WARN] OBD-1011: (192.168.137.70) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536)
[2024-07-24 09:11:20.801] [INFO] [WARN] OBD-1007: (192.168.137.70) The recommended number of max user processes is 655350 (Current value: 120000)
[2024-07-24 09:11:20.801] [INFO] [WARN] OBD-1007: (192.168.137.70) The recommended number of core file size is unlimited (Current value: 0)
[2024-07-24 09:11:20.801] [INFO] [WARN] OBD-1007: (192.168.137.70) The recommended number of stack size is unlimited (Current value: 8192)
[2024-07-24 09:11:20.801] [INFO] [WARN] OBD-1017: (192.168.137.70) The value of the “vm.max_map_count” must be within [327600, 1310720] (Current value: 65530, Recommended value: 655360)
[2024-07-24 09:11:20.801] [INFO]
[2024-07-24 09:11:20.802] [DEBUG] - sub start_check ref count to 0
[2024-07-24 09:11:20.802] [DEBUG] - export start_check
[2024-07-24 09:11:20.802] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.2.0 for oceanbase-ce-4.2.3.0-100000112024042411.el7-ef16f0db100e41625be18924797b9f2bc17967d5
[2024-07-24 09:11:20.802] [DEBUG] - import start
[2024-07-24 09:11:20.808] [DEBUG] - add start ref count to 1
[2024-07-24 09:11:20.809] [INFO] Start observer
[2024-07-24 09:11:20.809] [DEBUG] – wsw@192.168.137.70 execute: ls /opt/software/data/clog/tenant_1/
[2024-07-24 09:11:20.813] [DEBUG] – exited code 0
[2024-07-24 09:11:20.813] [DEBUG] – wsw@192.168.137.70 execute: cat /home/admin/observer/run/observer.pid
[2024-07-24 09:11:20.857] [DEBUG] – exited code 0
[2024-07-24 09:11:20.858] [DEBUG] – wsw@192.168.137.70 execute: ls /proc/2278
[2024-07-24 09:11:20.902] [DEBUG] – exited code 2, error output:
[2024-07-24 09:11:20.902] [DEBUG] ls: 无法访问’/proc/2278’: 没有那个文件或目录
[2024-07-24 09:11:20.902] [DEBUG]
[2024-07-24 09:11:20.903] [DEBUG] – 192.168.137.70 start command construction
[2024-07-24 09:11:20.903] [DEBUG] – starting 192.168.137.70 observer
[2024-07-24 09:11:20.903] [DEBUG] – wsw@192.168.137.70 set env LD_LIBRARY_PATH to ‘/home/admin/observer/lib:’
[2024-07-24 09:11:20.903] [DEBUG] – wsw@192.168.137.70 execute: cd /home/admin/observer; /home/admin/observer/bin/observer -r ‘192.168.137.70:2882:2881’ -p 2881 -P 2882 -z ‘zone1’ -c 1 -d ‘/opt/software/data’ -I ‘192.168.137.70’ -o __min_full_resource_pool_memory=2147483648,memory_limit=‘6G’,system_memory=‘1G’,datafile_size=‘2G’,datafile_next=‘2G’,datafile_maxsize=‘20G’,log_disk_size=‘14G’,cpu_count=2,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4
[2024-07-24 09:11:21.705] [DEBUG] – exited code 0
[2024-07-24 09:11:21.706] [DEBUG] – wsw@192.168.137.70 delete env LD_LIBRARY_PATH
[2024-07-24 09:11:21.724] [DEBUG] – start_obshell: True
[2024-07-24 09:11:21.725] [DEBUG] – wsw@192.168.137.70 execute: cat /home/admin/observer/run/obshell.pid
[2024-07-24 09:11:21.756] [DEBUG] – exited code 0
[2024-07-24 09:11:21.757] [DEBUG] – wsw@192.168.137.70 execute: ls /proc/2325
[2024-07-24 09:11:21.802] [DEBUG] – exited code 2, error output:
[2024-07-24 09:11:21.802] [DEBUG] ls: 无法访问’/proc/2325’: 没有那个文件或目录
[2024-07-24 09:11:21.802] [DEBUG]
[2024-07-24 09:11:21.802] [DEBUG] – wsw@192.168.137.70 set env OB_ROOT_PASSWORD to ‘‘root1234’’
[2024-07-24 09:11:21.802] [DEBUG] – start obshell: cd /home/admin/observer; /home/admin/observer/bin/obshell admin start --ip 192.168.137.70 --port 2886
[2024-07-24 09:11:21.802] [DEBUG] – wsw@192.168.137.70 execute: cd /home/admin/observer; /home/admin/observer/bin/obshell admin start --ip 192.168.137.70 --port 2886
[2024-07-24 09:11:22.222] [DEBUG] – exited code 0
[2024-07-24 09:11:22.223] [INFO] observer program health check
[2024-07-24 09:11:25.226] [DEBUG] – 192.168.137.70 program health check
[2024-07-24 09:11:25.227] [DEBUG] – wsw@192.168.137.70 execute: cat /home/admin/observer/run/observer.pid
[2024-07-24 09:11:25.233] [DEBUG] – exited code 0
[2024-07-24 09:11:25.233] [DEBUG] – wsw@192.168.137.70 execute: ls /proc/2314
[2024-07-24 09:11:25.283] [DEBUG] – exited code 0
[2024-07-24 09:11:25.283] [DEBUG] – 192.168.137.70 observer[pid: 2314] started
[2024-07-24 09:11:25.356] [INFO] obshell program health check
[2024-07-24 09:11:25.357] [DEBUG] – wsw@192.168.137.70 execute: cat /home/admin/observer/run/obshell.pid
[2024-07-24 09:11:25.368] [DEBUG] – exited code 0
[2024-07-24 09:11:25.368] [DEBUG] – wsw@192.168.137.70 execute: ls /proc/2325
[2024-07-24 09:11:25.418] [DEBUG] – exited code 2, error output:
[2024-07-24 09:11:25.419] [DEBUG] ls: 无法访问’/proc/2325’: 没有那个文件或目录
[2024-07-24 09:11:25.419] [DEBUG]
[2024-07-24 09:11:25.489] [WARNING] OBD-2012: Failed to start 192.168.137.70 obshell
[2024-07-24 09:11:25.489] [DEBUG] - sub start ref count to 0
[2024-07-24 09:11:25.489] [DEBUG] - export start
[2024-07-24 09:11:25.489] [ERROR] oceanbase-ce start failed
[2024-07-24 09:11:25.494] [INFO] See OceanBase分布式数据库-海量数据 笔笔算数 .
[2024-07-24 09:11:25.494] [INFO] Trace ID: a2075130-4959-11ef-8b32-000c292c712f
[2024-07-24 09:11:25.494] [INFO] If you want to view detailed obd logs, please run: obd display-trace a2075130-4959-11ef-8b32-000c292c712f
[2024-07-24 09:11:25.494] [DEBUG] - share lock /home/wsw/.obd/lock/mirror_and_repo release, count 0
[2024-07-24 09:11:25.494] [DEBUG] - unlock /home/wsw/.obd/lock/mirror_and_repo
[2024-07-24 09:11:25.494] [DEBUG] - exclusive lock /home/wsw/.obd/lock/deploy_obtest release, count 0
[2024-07-24 09:11:25.494] [DEBUG] - unlock /home/wsw/.obd/lock/deploy_obtest
[2024-07-24 09:11:25.494] [DEBUG] - share lock /home/wsw/.obd/lock/global release, count 0
[2024-07-24 09:11:25.494] [DEBUG] - unlock /home/wsw/.obd/lock/global
你好，麻烦提供一下 192.168.137.70 节点上的 obshell 日志。observer 的日志目录里有个 log_obshell 目录，里面的就是 obshell 的日志。
目前ob集群正常么，看一下obshell进程是不是已经存在了