[root@e360bc204b87 ~]# obd display-trace 2dcd3536-4a59-11ef-b3f5-0242ac110002 [2024-07-25 07:40:35.274] [DEBUG] - cmd: ['obcluster'] [2024-07-25 07:40:35.275] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-07-25 07:40:35.275] [DEBUG] - mkdir /root/.obd/lock/ [2024-07-25 07:40:35.275] [DEBUG] - unknown lock mode [2024-07-25 07:40:35.275] [DEBUG] - try to get share lock /root/.obd/lock/global [2024-07-25 07:40:35.275] [DEBUG] - share lock `/root/.obd/lock/global`, count 1 [2024-07-25 07:40:35.275] [DEBUG] - Get Deploy by name [2024-07-25 07:40:35.275] [DEBUG] - mkdir /root/.obd/cluster/ [2024-07-25 07:40:35.276] [DEBUG] - mkdir /root/.obd/config_parser/ [2024-07-25 07:40:35.276] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_obcluster [2024-07-25 07:40:35.276] [DEBUG] - exclusive lock `/root/.obd/lock/deploy_obcluster`, count 1 [2024-07-25 07:40:35.280] [DEBUG] - Deploy status judge [2024-07-25 07:40:35.280] [INFO] Get local repositories [2024-07-25 07:40:35.281] [DEBUG] - mkdir /root/.obd/repository [2024-07-25 07:40:35.281] [DEBUG] - Get local repository oceanbase-ce-4.2.1.7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.281] [DEBUG] - Search repository oceanbase-ce version: 4.2.1.7, tag: f69097e5f6056a0f6949867a799eea3724a905aa, release: None, package_hash: None [2024-07-25 07:40:35.281] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo [2024-07-25 07:40:35.281] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 1 [2024-07-25 07:40:35.281] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce [2024-07-25 07:40:35.283] [DEBUG] - Found repository oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.283] [DEBUG] - Get local repository obagent-4.2.2-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.283] [DEBUG] - Search repository obagent version: 4.2.2, tag: 19739a07a12eab736aff86ecf357b1ae660b554e, release: None, package_hash: None [2024-07-25 07:40:35.283] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 2 [2024-07-25 07:40:35.283] [DEBUG] - mkdir /root/.obd/repository/obagent [2024-07-25 07:40:35.285] [DEBUG] - Found repository obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.285] [DEBUG] - Get deploy config [2024-07-25 07:40:35.304] [INFO] Search plugins [2024-07-25 07:40:35.304] [DEBUG] - Searching start_check plugin for components ... [2024-07-25 07:40:35.305] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.305] [DEBUG] - mkdir /root/.obd/plugins [2024-07-25 07:40:35.306] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.1.4 for oceanbase-ce-4.2.1.7 [2024-07-25 07:40:35.306] [DEBUG] - Searching start_check plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.306] [DEBUG] - Found for obagent-py_script_start_check-1.3.0 for obagent-4.2.2 [2024-07-25 07:40:35.306] [DEBUG] - Searching create_tenant plugin for components ... [2024-07-25 07:40:35.306] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.306] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.1.7 [2024-07-25 07:40:35.306] [DEBUG] - Searching create_tenant plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.307] [DEBUG] - No such create_tenant plugin for obagent-4.2.2 [2024-07-25 07:40:35.307] [DEBUG] - Searching tenant_optimize plugin for components ... [2024-07-25 07:40:35.307] [DEBUG] - Searching tenant_optimize plugin for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.307] [DEBUG] - No such tenant_optimize plugin for oceanbase-ce-4.2.1.7 [2024-07-25 07:40:35.307] [DEBUG] - Searching tenant_optimize plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.307] [DEBUG] - No such tenant_optimize plugin for obagent-4.2.2 [2024-07-25 07:40:35.307] [DEBUG] - Searching start plugin for components ... [2024-07-25 07:40:35.307] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.308] [DEBUG] - Found for oceanbase-ce-py_script_start-4.2.1.4 for oceanbase-ce-4.2.1.7 [2024-07-25 07:40:35.308] [DEBUG] - Searching start plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.308] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-4.2.2 [2024-07-25 07:40:35.308] [DEBUG] - Searching connect plugin for components ... [2024-07-25 07:40:35.308] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.309] [DEBUG] - Found for oceanbase-ce-py_script_connect-4.2.1.4 for oceanbase-ce-4.2.1.7 [2024-07-25 07:40:35.309] [DEBUG] - Searching connect plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.309] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-4.2.2 [2024-07-25 07:40:35.309] [DEBUG] - Searching bootstrap plugin for components ... [2024-07-25 07:40:35.309] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.310] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.1.4 for oceanbase-ce-4.2.1.7 [2024-07-25 07:40:35.310] [DEBUG] - Searching bootstrap plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.310] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-4.2.2 [2024-07-25 07:40:35.310] [DEBUG] - Searching display plugin for components ... [2024-07-25 07:40:35.310] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.310] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.1.7 [2024-07-25 07:40:35.310] [DEBUG] - Searching display plugin for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.310] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-4.2.2 [2024-07-25 07:40:35.311] [INFO] Load cluster param plugin [2024-07-25 07:40:35.311] [DEBUG] - Get local repository oceanbase-ce-4.2.1.7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.311] [DEBUG] - Get local repository obagent-4.2.2-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.311] [DEBUG] - Searching param plugin for components ... [2024-07-25 07:40:35.311] [DEBUG] - Search param plugin for oceanbase-ce [2024-07-25 07:40:35.312] [DEBUG] - Found for oceanbase-ce-param-4.2.1.4 for oceanbase-ce-4.2.1.7 [2024-07-25 07:40:35.312] [DEBUG] - Applying oceanbase-ce-param-4.2.1.4 for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.794] [DEBUG] - Search param plugin for obagent [2024-07-25 07:40:35.795] [DEBUG] - Found for obagent-param-1.3.0 for obagent-4.2.2 [2024-07-25 07:40:35.795] [DEBUG] - Applying obagent-param-1.3.0 for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.850] [INFO] Open ssh connection [2024-07-25 07:40:35.854] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.1.4 for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.854] [DEBUG] - import start_check [2024-07-25 07:40:35.859] [DEBUG] - add start_check ref count to 1 [2024-07-25 07:40:35.860] [INFO] Check before start observer [2024-07-25 07:40:35.862] [DEBUG] -- local execute: ls /root/ob/store/clog/tenant_1/ [2024-07-25 07:40:35.866] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.866] [DEBUG] -- local execute: cat /root/ob/run/observer.pid [2024-07-25 07:40:35.870] [DEBUG] -- exited code 1, error output: [2024-07-25 07:40:35.870] [DEBUG] cat: /root/ob/run/observer.pid: No such file or directory [2024-07-25 07:40:35.870] [DEBUG] [2024-07-25 07:40:35.870] [DEBUG] -- 127.0.0.1 port check [2024-07-25 07:40:35.870] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B41' | awk -F' ' '{print $3}' | uniq [2024-07-25 07:40:35.877] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.877] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B42' | awk -F' ' '{print $3}' | uniq [2024-07-25 07:40:35.884] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.884] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B46' | awk -F' ' '{print $3}' | uniq [2024-07-25 07:40:35.891] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.892] [DEBUG] -- local execute: ls /root/ob/store/sstable/block_file [2024-07-25 07:40:35.896] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.896] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-07-25 07:40:35.901] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.901] [WARNING] OBD-1011: (127.0.0.1) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536) [2024-07-25 07:40:35.902] [DEBUG] -- local execute: ulimit -a [2024-07-25 07:40:35.904] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.905] [WARNING] OBD-1007: (127.0.0.1) The recommended number of stack size is unlimited (Current value: 8192) [2024-07-25 07:40:35.905] [DEBUG] -- local execute: sysctl -a [2024-07-25 07:40:35.923] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.925] [WARNING] OBD-1017: (127.0.0.1) The value of the "vm.max_map_count" must be within [327600, 1310720] (Current value: 65530, Recommended value: 655360) [2024-07-25 07:40:35.925] [DEBUG] -- local execute: cat /proc/meminfo [2024-07-25 07:40:35.930] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.931] [WARNING] OBD-2000: (127.0.0.1) not enough memory. (Free: 4553M, Need: 6144M) [2024-07-25 07:40:35.931] [DEBUG] -- local execute: df --block-size=1024 [2024-07-25 07:40:35.935] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /, total: 983349346304 avail: 677995307008 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /dev, total: 67108864 avail: 67108864 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /sys/fs/cgroup, total: 8326750208 avail: 8326750208 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /dev/shm, total: 67108864 avail: 67108864 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /etc/hosts, total: 983349346304 avail: 677995307008 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /proc/asound, total: 8326750208 avail: 8326750208 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /proc/acpi, total: 8326750208 avail: 8326750208 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /proc/scsi, total: 8326750208 avail: 8326750208 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /sys/firmware, total: 8326750208 avail: 8326750208 [2024-07-25 07:40:35.936] [DEBUG] -- get disk info for path /sys/devices/virtual/powercap, total: 8326750208 avail: 8326750208 [2024-07-25 07:40:35.936] [DEBUG] -- disk: {'/': {'total': 983349346304, 'avail': 677995307008, 'need': 0}, '/dev': {'total': 67108864, 'avail': 67108864, 'need': 0}, '/sys/fs/cgroup': {'total': 8326750208, 'avail': 8326750208, 'need': 0}, '/dev/shm': {'total': 67108864, 'avail': 67108864, 'need': 0}, '/etc/hosts': {'total': 983349346304, 'avail': 677995307008, 'need': 0}, '/proc/asound': {'total': 8326750208, 'avail': 8326750208, 'need': 0}, '/proc/acpi': {'total': 8326750208, 'avail': 8326750208, 'need': 0}, '/proc/scsi': {'total': 8326750208, 'avail': 8326750208, 'need': 0}, '/sys/firmware': {'total': 8326750208, 'avail': 8326750208, 'need': 0}, '/sys/devices/virtual/powercap': {'total': 8326750208, 'avail': 8326750208, 'need': 0}} [2024-07-25 07:40:35.936] [DEBUG] -- local execute: date +%s%N [2024-07-25 07:40:35.941] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.941] [DEBUG] -- 127.0.0.1 time delta -1.551025390625 [2024-07-25 07:40:35.942] [INFO] [WARN] OBD-1011: (127.0.0.1) The recommended value of fs.aio-max-nr is 1048576 (Current value: 65536) [2024-07-25 07:40:35.942] [INFO] [WARN] OBD-1007: (127.0.0.1) The recommended number of stack size is unlimited (Current value: 8192) [2024-07-25 07:40:35.942] [INFO] [WARN] OBD-1017: (127.0.0.1) The value of the "vm.max_map_count" must be within [327600, 1310720] (Current value: 65530, Recommended value: 655360) [2024-07-25 07:40:35.942] [INFO] [WARN] OBD-2000: (127.0.0.1) not enough memory. (Free: 4553M, Need: 6144M) [2024-07-25 07:40:35.942] [INFO] [2024-07-25 07:40:35.942] [DEBUG] - sub start_check ref count to 0 [2024-07-25 07:40:35.942] [DEBUG] - export start_check [2024-07-25 07:40:35.942] [DEBUG] - plugin oceanbase-ce-py_script_start_check-4.2.1.4 result: True [2024-07-25 07:40:35.943] [DEBUG] - Call obagent-py_script_start_check-1.3.0 for obagent-4.2.2-100000042024011120.el7-19739a07a12eab736aff86ecf357b1ae660b554e [2024-07-25 07:40:35.943] [DEBUG] - import start_check [2024-07-25 07:40:35.945] [DEBUG] - add start_check ref count to 1 [2024-07-25 07:40:35.945] [INFO] Check before start obagent [2024-07-25 07:40:35.947] [DEBUG] -- local execute: cat /root/obagent/run/ob_agentd.pid [2024-07-25 07:40:35.952] [DEBUG] -- exited code 1, error output: [2024-07-25 07:40:35.952] [DEBUG] cat: /root/obagent/run/ob_agentd.pid: No such file or directory [2024-07-25 07:40:35.952] [DEBUG] [2024-07-25 07:40:35.952] [DEBUG] -- 127.0.0.1 port check [2024-07-25 07:40:35.952] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1F99' | awk -F' ' '{print $3}' | uniq [2024-07-25 07:40:35.959] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.960] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':1F98' | awk -F' ' '{print $3}' | uniq [2024-07-25 07:40:35.967] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.968] [DEBUG] - sub start_check ref count to 0 [2024-07-25 07:40:35.968] [DEBUG] - export start_check [2024-07-25 07:40:35.968] [DEBUG] - plugin obagent-py_script_start_check-1.3.0 result: True [2024-07-25 07:40:35.968] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.1.4 for oceanbase-ce-4.2.1.7-107000162024060611.el7-f69097e5f6056a0f6949867a799eea3724a905aa [2024-07-25 07:40:35.968] [DEBUG] - import start [2024-07-25 07:40:35.970] [DEBUG] - add start ref count to 1 [2024-07-25 07:40:35.971] [INFO] Start observer [2024-07-25 07:40:35.971] [DEBUG] -- local execute: ls /root/ob/store/clog/tenant_1/ [2024-07-25 07:40:35.975] [DEBUG] -- exited code 0 [2024-07-25 07:40:35.975] [DEBUG] -- local execute: cat /root/ob/run/observer.pid [2024-07-25 07:40:35.979] [DEBUG] -- exited code 1, error output: [2024-07-25 07:40:35.979] [DEBUG] cat: /root/ob/run/observer.pid: No such file or directory [2024-07-25 07:40:35.979] [DEBUG] [2024-07-25 07:40:35.980] [DEBUG] -- 127.0.0.1 start command construction [2024-07-25 07:40:35.980] [DEBUG] -- starting 127.0.0.1 observer [2024-07-25 07:40:35.980] [DEBUG] -- root@127.0.0.1 set env LD_LIBRARY_PATH to '/root/ob/lib:' [2024-07-25 07:40:35.980] [DEBUG] -- local execute: cd /root/ob; /root/ob/bin/observer -r '127.0.0.1:2882:2881' -p 2881 -P 2882 -z 'zone1' -n 'obcluster' -c 1 -d '/root/ob/store' -l 'INFO' -I '127.0.0.1' -o __min_full_resource_pool_memory=2147483648,memory_limit='6G',system_memory='1G',datafile_size='5G',log_disk_size='5G',cpu_count=16,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4,enable_rich_error_msg=True [2024-07-25 07:40:36.039] [DEBUG] -- exited code 0 [2024-07-25 07:40:36.039] [DEBUG] -- root@127.0.0.1 delete env LD_LIBRARY_PATH [2024-07-25 07:40:36.039] [DEBUG] -- start_obshell: True [2024-07-25 07:40:36.040] [DEBUG] -- local execute: cat /root/ob/run/obshell.pid [2024-07-25 07:40:36.044] [DEBUG] -- exited code 1, error output: [2024-07-25 07:40:36.044] [DEBUG] cat: /root/ob/run/obshell.pid: No such file or directory [2024-07-25 07:40:36.044] [DEBUG] [2024-07-25 07:40:36.044] [DEBUG] -- root@127.0.0.1 set env OB_ROOT_PASSWORD to '' [2024-07-25 07:40:36.045] [DEBUG] -- start obshell: cd /root/ob; /root/ob/bin/obshell admin start --ip 127.0.0.1 --port 2886 [2024-07-25 07:40:36.045] [DEBUG] -- local execute: cd /root/ob; /root/ob/bin/obshell admin start --ip 127.0.0.1 --port 2886 [2024-07-25 07:40:37.083] [DEBUG] -- exited code 30, error output: [2024-07-25 07:40:37.083] [DEBUG] open /root/ob/run/daemon.pid: file exists [2024-07-25 07:40:37.083] [DEBUG] [2024-07-25 07:40:37.083] [ERROR] 127.0.0.1 obshell failed [2024-07-25 07:40:37.083] [DEBUG] - sub start ref count to 0 [2024-07-25 07:40:37.083] [DEBUG] - export start [2024-07-25 07:40:37.084] [DEBUG] - plugin oceanbase-ce-py_script_start-4.2.1.4 result: False [2024-07-25 07:40:37.084] [ERROR] oceanbase-ce start failed [2024-07-25 07:40:37.086] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2024-07-25 07:40:37.086] [INFO] Trace ID: 2dcd3536-4a59-11ef-b3f5-0242ac110002 [2024-07-25 07:40:37.087] [INFO] If you want to view detailed obd logs, please run: obd display-trace 2dcd3536-4a59-11ef-b3f5-0242ac110002 [2024-07-25 07:40:37.087] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1 [2024-07-25 07:40:37.087] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0 [2024-07-25 07:40:37.087] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo [2024-07-25 07:40:37.087] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obcluster release, count 0 [2024-07-25 07:40:37.087] [DEBUG] - unlock /root/.obd/lock/deploy_obcluster [2024-07-25 07:40:37.087] [DEBUG] - share lock /root/.obd/lock/global release, count 0 [2024-07-25 07:40:37.087] [DEBUG] - unlock /root/.obd/lock/global