jks@jks2:~$ sudo sysctl fs.aio-max-nr=1048576 [sudo] jks 的密码: fs.aio-max-nr = 1048576 jks@jks2:~$ obd cluster start obytt100 Get local repositories ok Search plugins ok Load cluster param plugin ok Open ssh connection ok Check before start observer ok [WARN] (127.0.0.1) failed to get open files [WARN] (127.0.0.1) failed to get max user processes [WARN] (127.0.0.1) failed to get core file size [WARN] (127.0.0.1) failed to get stack size Start observer ok [ERROR] 127.0.0.1 obshell failed [ERROR] oceanbase-ce start failed See https://www.oceanbase.com/product/ob-deployer/error-codes . Trace ID: bb9980f2-c1fc-11ef-8a6a-f707f2a9521f If you want to view detailed obd logs, please run: obd display-trace bb9980f2-c1fc-11ef-8a6a-f707f2a9521f jks@jks2:~$ obd display-trace bb9980f2-c1fc-11ef-8a6a-f707f2a9521f [2024-12-24 13:41:09.531] [DEBUG] - cmd: ['obytt100'] [2024-12-24 13:41:09.547] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None} [2024-12-24 13:41:09.547] [DEBUG] - mkdir /home/jks/.obd/lock/ [2024-12-24 13:41:09.547] [DEBUG] - unknown lock mode [2024-12-24 13:41:09.548] [DEBUG] - try to get share lock /home/jks/.obd/lock/global [2024-12-24 13:41:09.548] [DEBUG] - share lock `/home/jks/.obd/lock/global`, count 1 [2024-12-24 13:41:09.548] [DEBUG] - Get Deploy by name [2024-12-24 13:41:09.548] [DEBUG] - mkdir /home/jks/.obd/cluster/ [2024-12-24 13:41:09.548] [DEBUG] - mkdir /home/jks/.obd/config_parser/ [2024-12-24 13:41:09.548] [DEBUG] - try to get exclusive lock /home/jks/.obd/lock/deploy_obytt100 [2024-12-24 13:41:09.549] [DEBUG] - exclusive lock `/home/jks/.obd/lock/deploy_obytt100`, count 1 [2024-12-24 13:41:09.565] [DEBUG] - Deploy status judge [2024-12-24 13:41:09.590] [INFO] Get local repositories [2024-12-24 13:41:09.590] [DEBUG] - mkdir /home/jks/.obd/repository [2024-12-24 13:41:09.591] [DEBUG] - Get local repository oceanbase-ce-4.2.1.10-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.591] [DEBUG] - Search repository oceanbase-ce version: 4.2.1.10, tag: 19e05fb07f899bc2640bbcc4edb3f125b642810a, release: None, package_hash: None [2024-12-24 13:41:09.591] [DEBUG] - try to get share lock /home/jks/.obd/lock/mirror_and_repo [2024-12-24 13:41:09.591] [DEBUG] - share lock `/home/jks/.obd/lock/mirror_and_repo`, count 1 [2024-12-24 13:41:09.591] [DEBUG] - mkdir /home/jks/.obd/repository/oceanbase-ce [2024-12-24 13:41:09.594] [DEBUG] - Found repository oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.721] [DEBUG] - Get deploy config [2024-12-24 13:41:09.745] [INFO] Search plugins [2024-12-24 13:41:09.746] [DEBUG] - Searching start_check plugin for components ... [2024-12-24 13:41:09.746] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.746] [DEBUG] - mkdir /home/jks/.obd/plugins [2024-12-24 13:41:09.764] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.2.1.4 for oceanbase-ce-4.2.1.10 [2024-12-24 13:41:09.764] [DEBUG] - Searching create_tenant plugin for components ... [2024-12-24 13:41:09.764] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.764] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.2.0.0 for oceanbase-ce-4.2.1.10 [2024-12-24 13:41:09.765] [DEBUG] - Searching tenant_optimize plugin for components ... [2024-12-24 13:41:09.765] [DEBUG] - Searching tenant_optimize plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.765] [DEBUG] - No such tenant_optimize plugin for oceanbase-ce-4.2.1.10 [2024-12-24 13:41:09.765] [DEBUG] - Searching start plugin for components ... [2024-12-24 13:41:09.765] [DEBUG] - Searching start plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.766] [DEBUG] - Found for oceanbase-ce-py_script_start-4.2.1.4 for oceanbase-ce-4.2.1.10 [2024-12-24 13:41:09.766] [DEBUG] - Searching connect plugin for components ... [2024-12-24 13:41:09.766] [DEBUG] - Searching connect plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.766] [DEBUG] - Found for oceanbase-ce-py_script_connect-4.2.1.4 for oceanbase-ce-4.2.1.10 [2024-12-24 13:41:09.766] [DEBUG] - Searching bootstrap plugin for components ... [2024-12-24 13:41:09.766] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.767] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.2.1.4 for oceanbase-ce-4.2.1.10 [2024-12-24 13:41:09.767] [DEBUG] - Searching display plugin for components ... [2024-12-24 13:41:09.767] [DEBUG] - Searching display plugin for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.767] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.2.1.10 [2024-12-24 13:41:09.876] [INFO] Load cluster param plugin [2024-12-24 13:41:09.877] [DEBUG] - Get local repository oceanbase-ce-4.2.1.10-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:09.877] [DEBUG] - Searching param plugin for components ... [2024-12-24 13:41:09.877] [DEBUG] - Search param plugin for oceanbase-ce [2024-12-24 13:41:09.877] [DEBUG] - Found for oceanbase-ce-param-4.2.1.4 for oceanbase-ce-4.2.1.10 [2024-12-24 13:41:09.877] [DEBUG] - Applying oceanbase-ce-param-4.2.1.4 for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:10.432] [INFO] Open ssh connection [2024-12-24 13:41:10.563] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.2.1.4 for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:10.563] [DEBUG] - import start_check [2024-12-24 13:41:10.570] [DEBUG] - add start_check ref count to 1 [2024-12-24 13:41:10.570] [INFO] Check before start observer [2024-12-24 13:41:10.572] [DEBUG] -- local execute: ls /home/jks/observer/store/clog/tenant_1/ [2024-12-24 13:41:10.576] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.577] [DEBUG] -- local execute: cat /home/jks/observer/run/observer.pid [2024-12-24 13:41:10.597] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.598] [DEBUG] -- local execute: ls /proc/6496 [2024-12-24 13:41:10.602] [DEBUG] -- exited code 2, error output: [2024-12-24 13:41:10.602] [DEBUG] ls: 无法访问 '/proc/6496': 没有那个文件或目录 [2024-12-24 13:41:10.602] [DEBUG] [2024-12-24 13:41:10.602] [DEBUG] -- 127.0.0.1 port check [2024-12-24 13:41:10.602] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B41' | awk -F' ' '{print $3}' | uniq [2024-12-24 13:41:10.615] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.615] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B42' | awk -F' ' '{print $3}' | uniq [2024-12-24 13:41:10.622] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.623] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{if($4=="0A") print $2,$4,$10}' | grep ':0B46' | awk -F' ' '{print $3}' | uniq [2024-12-24 13:41:10.630] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.631] [DEBUG] -- local execute: ls /home/jks/observer/store/sstable/block_file [2024-12-24 13:41:10.634] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.634] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr [2024-12-24 13:41:10.637] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.638] [DEBUG] -- local execute: ulimit -a [2024-12-24 13:41:10.640] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.641] [WARNING] (127.0.0.1) failed to get open files [2024-12-24 13:41:10.641] [WARNING] (127.0.0.1) failed to get max user processes [2024-12-24 13:41:10.641] [WARNING] (127.0.0.1) failed to get core file size [2024-12-24 13:41:10.641] [WARNING] (127.0.0.1) failed to get stack size [2024-12-24 13:41:10.641] [DEBUG] -- local execute: sysctl -a [2024-12-24 13:41:10.683] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.686] [DEBUG] -- local execute: cat /proc/meminfo [2024-12-24 13:41:10.689] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.689] [DEBUG] -- local execute: df --block-size=1024 [2024-12-24 13:41:10.693] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.693] [DEBUG] -- get disk info for path /run, total: 1668837376 avail: 1664049152 [2024-12-24 13:41:10.693] [DEBUG] -- get disk info for path /, total: 491104411648 avail: 434735550464 [2024-12-24 13:41:10.693] [DEBUG] -- get disk info for path /dev/shm, total: 8344186880 avail: 8344182784 [2024-12-24 13:41:10.693] [DEBUG] -- get disk info for path /run/lock, total: 5242880 avail: 5234688 [2024-12-24 13:41:10.694] [DEBUG] -- get disk info for path /run/user/1000, total: 1668837376 avail: 1668677632 [2024-12-24 13:41:10.694] [DEBUG] -- disk: {'/run': {'total': 1668837376, 'avail': 1664049152, 'need': 0}, '/': {'total': 491104411648, 'avail': 434735550464, 'need': 0}, '/dev/shm': {'total': 8344186880, 'avail': 8344182784, 'need': 0}, '/run/lock': {'total': 5242880, 'avail': 5234688, 'need': 0}, '/run/user/1000': {'total': 1668837376, 'avail': 1668677632, 'need': 0}} [2024-12-24 13:41:10.694] [DEBUG] -- local execute: date +%s%N [2024-12-24 13:41:10.697] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.697] [DEBUG] -- 127.0.0.1 time delta 0.930419921875 [2024-12-24 13:41:10.701] [INFO] [WARN] (127.0.0.1) failed to get open files [2024-12-24 13:41:10.701] [INFO] [WARN] (127.0.0.1) failed to get max user processes [2024-12-24 13:41:10.701] [INFO] [WARN] (127.0.0.1) failed to get core file size [2024-12-24 13:41:10.701] [INFO] [WARN] (127.0.0.1) failed to get stack size [2024-12-24 13:41:10.701] [INFO] [2024-12-24 13:41:10.701] [DEBUG] - sub start_check ref count to 0 [2024-12-24 13:41:10.702] [DEBUG] - export start_check [2024-12-24 13:41:10.702] [DEBUG] - plugin oceanbase-ce-py_script_start_check-4.2.1.4 result: True [2024-12-24 13:41:10.702] [DEBUG] - Call oceanbase-ce-py_script_start-4.2.1.4 for oceanbase-ce-4.2.1.10-110000072024112010.el7-19e05fb07f899bc2640bbcc4edb3f125b642810a [2024-12-24 13:41:10.702] [DEBUG] - import start [2024-12-24 13:41:10.705] [DEBUG] - add start ref count to 1 [2024-12-24 13:41:10.705] [INFO] Start observer [2024-12-24 13:41:10.705] [DEBUG] -- local execute: ls /home/jks/observer/store/clog/tenant_1/ [2024-12-24 13:41:10.709] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.709] [DEBUG] -- local execute: cat /home/jks/observer/run/observer.pid [2024-12-24 13:41:10.712] [DEBUG] -- exited code 0 [2024-12-24 13:41:10.712] [DEBUG] -- local execute: ls /proc/6496 [2024-12-24 13:41:10.716] [DEBUG] -- exited code 2, error output: [2024-12-24 13:41:10.716] [DEBUG] ls: 无法访问 '/proc/6496': 没有那个文件或目录 [2024-12-24 13:41:10.716] [DEBUG] [2024-12-24 13:41:10.716] [DEBUG] -- 127.0.0.1 start command construction [2024-12-24 13:41:10.717] [DEBUG] -- starting 127.0.0.1 observer [2024-12-24 13:41:10.717] [DEBUG] -- jks@127.0.0.1 export LD_LIBRARY_PATH='/home/jks/observer/lib:' [2024-12-24 13:41:10.717] [DEBUG] -- local execute: cd /home/jks/observer; /home/jks/observer/bin/observer -p 2881 -P 2882 -z 'zone1' -n 'obytt100' -c 100 -d '/home/jks/observer/store' -I '127.0.0.1' -o __min_full_resource_pool_memory=2147483648,memory_limit='4G',system_memory='1G',datafile_size='1G',datafile_next='1G',datafile_maxsize='20G',log_disk_size='10G',cpu_count=8,enable_syslog_wf=False,max_syslog_file_count=4 [2024-12-24 13:41:13.941] [DEBUG] -- exited code 0 [2024-12-24 13:41:13.942] [DEBUG] -- jks@127.0.0.1 delete env LD_LIBRARY_PATH [2024-12-24 13:41:13.964] [DEBUG] -- start_obshell: True [2024-12-24 13:41:13.964] [DEBUG] -- local execute: cat /home/jks/observer/run/obshell.pid [2024-12-24 13:41:13.968] [DEBUG] -- exited code 1, error output: [2024-12-24 13:41:13.968] [DEBUG] cat: /home/jks/observer/run/obshell.pid: 没有那个文件或目录 [2024-12-24 13:41:13.968] [DEBUG] [2024-12-24 13:41:13.968] [DEBUG] -- jks@127.0.0.1 export OB_ROOT_PASSWORD='******' [2024-12-24 13:41:13.968] [DEBUG] -- start obshell: cd /home/jks/observer; /home/jks/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886 [2024-12-24 13:41:13.968] [DEBUG] -- local execute: cd /home/jks/observer; /home/jks/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886 [2024-12-24 13:46:14.070] [DEBUG] -- exited code 255, error output: [2024-12-24 13:46:14.070] [DEBUG] Command 'cd /home/jks/observer; /home/jks/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886' timed out after 300 seconds [2024-12-24 13:46:14.072] [ERROR] Traceback (most recent call last): [2024-12-24 13:46:14.072] [ERROR] File "core.py", line 2104, in start_cluster [2024-12-24 13:46:14.072] [ERROR] File "core.py", line 2198, in _start_cluster [2024-12-24 13:46:14.072] [ERROR] File "core.py", line 198, in call_plugin [2024-12-24 13:46:14.072] [ERROR] File "_plugin.py", line 348, in __call__ [2024-12-24 13:46:14.072] [ERROR] File "_plugin.py", line 305, in _new_func [2024-12-24 13:46:14.072] [ERROR] File "/home/jks/.obd/plugins/oceanbase-ce/4.2.1.4/start.py", line 286, in start [2024-12-24 13:46:14.072] [ERROR] if not client.execute_command(cmd): [2024-12-24 13:46:14.072] [ERROR] File "_plugin.py", line 225, in new_method [2024-12-24 13:46:14.072] [ERROR] File "_stdio.py", line 956, in func_wrapper [2024-12-24 13:46:14.072] [ERROR] File "ssh.py", line 499, in execute_command [2024-12-24 13:46:14.072] [ERROR] File "_stdio.py", line 956, in func_wrapper [2024-12-24 13:46:14.073] [ERROR] File "ssh.py", line 210, in execute_command [2024-12-24 13:46:14.073] [ERROR] File "subprocess.py", line 1028, in communicate [2024-12-24 13:46:14.073] [ERROR] File "subprocess.py", line 1869, in _communicate [2024-12-24 13:46:14.073] [ERROR] File "subprocess.py", line 1072, in _check_timeout [2024-12-24 13:46:14.073] [ERROR] subprocess.TimeoutExpired: Command 'cd /home/jks/observer; /home/jks/observer/bin/obshell admin start --ip 127.0.0.1 --port 2886' timed out after 300 seconds [2024-12-24 13:46:14.073] [ERROR] [2024-12-24 13:46:14.073] [ERROR] 127.0.0.1 obshell failed [2024-12-24 13:46:14.073] [DEBUG] - sub start ref count to 0 [2024-12-24 13:46:14.073] [DEBUG] - export start [2024-12-24 13:46:14.073] [DEBUG] - plugin oceanbase-ce-py_script_start-4.2.1.4 result: None [2024-12-24 13:46:14.074] [ERROR] oceanbase-ce start failed [2024-12-24 13:46:14.077] [DEBUG] - share lock /home/jks/.obd/lock/mirror_and_repo release, count 0 [2024-12-24 13:46:14.077] [DEBUG] - unlock /home/jks/.obd/lock/mirror_and_repo [2024-12-24 13:46:14.077] [DEBUG] - exclusive lock /home/jks/.obd/lock/deploy_obytt100 release, count 0 [2024-12-24 13:46:14.077] [DEBUG] - unlock /home/jks/.obd/lock/deploy_obytt100 [2024-12-24 13:46:14.077] [DEBUG] - share lock /home/jks/.obd/lock/global release, count 0 [2024-12-24 13:46:14.077] [DEBUG] - unlock /home/jks/.obd/lock/global [2024-12-24 13:46:14.078] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes . [2024-12-24 13:46:14.078] [INFO] Trace ID: bb9980f2-c1fc-11ef-8a6a-f707f2a9521f [2024-12-24 13:46:14.078] [INFO] If you want to view detailed obd logs, please run: obd display-trace bb9980f2-c1fc-11ef-8a6a-f707f2a9521f jks@jks2:~$